# monitoring Stack

## Overview
The monitoring stack in NetGrimoire is a collection of services that provide metrics collection, dashboards, alert routing, and container metrics.

---

## Architecture

| Service | Image | Port | Role |
|---------|-------|------|------|
- **Prometheus** | prom/prometheus:latest | 9090 | Metrics Collection |
- **Grafana** | grafana/grafana:latest | 3000 | Dashboards |
- **Alertmanager** | prom/alertmanager:latest | 9093 | Alert Routing |
- **Cadvisor** | gcr.io/cadvisor/cadvisor:latest | Internal only | Container Metrics |
- **Node Exporter** | prom/node-exporter:latest | Internal only | Host Metrics |

Exposed via:
- `prometheus.netgrimoire.com`
- `grafana.netgrimoire.com`
- `alertmanager.netgrimoire.com`

Homepage group: Monitoring

---

## Build & Configuration

### Prerequisites
No specific prerequisites for this stack.

### Volume Setup
```bash
mkdir -p /DockerVol/prometheus/data
mkdir -p /DockerVol/grafana/data
mkdir -p /DockerVol/alertmanager/data
```

### Environment Variables
```bash
# generate: openssl rand -hex 32
GF_SECURITY_ADMIN_PASSWORD=F@lcon13
GF_USERS_DEFAULT_THEME=dark
GF_FEATURE_TOGGLES_ENABLE.publicDashboards=true
```

### Deploy
```bash
cd services/swarm/stack/monitoring
set -a && source .env && set +a
docker stack config --compose-file monitoring-stack.yml > resolved.yml
docker stack deploy --compose-file resolved.yml monitoring
rm resolved.yml
docker stack services monitoring
```

### First Run
Run the following command after deployment: `./deploy.sh`

---

## User Guide

### Accessing Monitoring
| Service | URL | Purpose |
- **Prometheus** | http://prometheus.netgrimoire.com | Metrics Collection |
- **Grafana** | http://grafana.netgrimoire.com | Dashboards |

### Primary Use Cases
To access the monitoring dashboard, navigate to `http://grafana.netgrimoire.com` and log in with the admin credentials.

### NetGrimoire Integrations
This stack connects to other services via environment variables and labels. Specifically, it integrates with `crowdsec` via Caddy reverse proxy labels.

---

## Operations

### Monitoring
```bash
docker stack services monitoring
docker service logs -f monitoring/prometheus
```

### Backups
Critical data volumes are stored in `/DockerVol/prometheus/data`, `/DockerVol/grafana/data`, and `/DockerVol/alertmanager/data`. These volumes can be backed up using `docker volume backup`.

### Restore
Restore the stack by running: `./deploy.sh`

---

## Common Failures

| Failure Mode | Symptom | Cause | Fix |
|-------------|---------|------|-----|
| Prometheus | No data in Grafana | No connections between services | Check Caddy reverse proxy labels and ensure proper connections |
| Grafana | Blank dashboard | Missing configuration file | Check for missing `GF_SERVER_ROOT_URL` environment variable |

---

## Changelog

| Date | Commit | Summary |
|------|--------|---------|
| 2026-04-07 | 04863ab6 | Initial documentation creation |
| 2026-04-07 | 0af60dbe | Updated monitoring services to use latest images and fixed a minor bug |

<Write a paragraph summarizing the evolution of this service based on the diffs above. This is the initial documentation for the monitoring stack in NetGrimoire, created on April 8th, 2026, with two commits: one for creating the initial documentation and another for updating the services to use latest images.>