---
title: monitoring Stack
description: NetGrimoire Monitoring Service
published: true
date: 2026-04-10T03:17:27.514Z
tags: docker,swarm,monitoring,netgrimoire
editor: markdown
dateCreated: 2026-04-10T03:17:27.514Z

---

# monitoring

## Overview
The monitoring stack in NetGrimoire provides a comprehensive suite of services for collecting, processing, and visualizing system metrics. The stack consists of Prometheus, Grafana, Alertmanager, Blackbox Exporter, Cadvisor, and Node Exporter. These services work together to provide real-time insights into the health and performance of the NetGrimoire infrastructure.

---

## Architecture
| Service | Image | Port | Role |
|- **Prometheus** | prom/prometheus:latest | 9090 | Metrics Collection |
|- **Grafana** | grafana/grafana:latest | 3000 | Dashboards |
|- **Alertmanager** | prom/alertmanager:latest | 9093 | Alert Routing |
|- **Blackbox Exporter** | prom/blackbox-exporter:latest | 9115 | HTTP/TCP/ICMP Probing |
|- **Cadvisor** | gcr.io/cadvisor/cadvisor:latest | / | Multi-arch image (global) |
|- **Node Exporter** | prom/node-exporter:latest | / | Host metrics (all nodes) |

Exposed via:
- `prometheus.netgrimoire.com`
- `grafana.netgrimoire.com`
- `alertmanager.netgrimoire.com`
- `blackbox.netgrimoire.com`

Exposed to internal services via Caddy reverse proxy.

Homepage group: Monitoring

---

## Build & Configuration

### Prerequisites
Generate environment variables using `openssl rand -hex 32`.

### Volume Setup
```bash
mkdir -p /DockerVol/prometheus/data
mkdir -p /DockerVol/grafana/data
mkdir -p /DockerVol/alertmanager/data
mkdir -p /DockerVol/blackbox/config
```

### Environment Variables
```bash
# generate: openssl rand -hex 32
```

### Deploy
```bash
cd services/swarm/stack/monitoring
set -a && source .env && set +a
docker stack config --compose-file monitoring-stack.yml > resolved.yml
docker stack deploy --compose-file resolved.yml monitoring
rm resolved.yml
docker stack services monitoring
```

### First Run
Post-deploy steps specific to these services:

- Start Cadvisor and Node Exporter.
- Configure Grafana with default settings.

---

## User Guide

### Accessing monitoring
| Service | URL | Purpose |
|- **Prometheus** | `http://prometheus.netgrimoire.com` | Metrics Collection |
|- **Grafana** | `https://grafana.netgrimoire.com` | Dashboards |
|- **Alertmanager** | `https://alertmanager.netgrimoire.com` | Alert Routing |
|- **Blackbox Exporter** | `http://blackbox.netgrimoire.com` | HTTP/TCP/ICMP Probing |

### Primary Use Cases
Use these services to monitor the health and performance of NetGrimoire infrastructure components.

### NetGrimoire Integrations
These services integrate with other NetGrimoire services, including Caddy, Uptime Kuma, and DIUN.

---

## Operations

### Monitoring
Use `docker stack services monitoring` to view service logs. Use `docker logs -f <service-name>` to view live logs.

### Backups
Critical vs reconstructable `/DockerVol/` paths:

- Critical: `/prometheus/data`
- Reconstructable: `/grafana/data`, `/alertmanager/data`

### Restore
```bash
cd services/swarm/stack/monitoring
./deploy.sh
```

---

## Common Failures
| Symptom | Cause | Fix |
|- **Prometheus not collecting metrics** | Insufficient disk space | Increase Prometheus storage size |
|- **Grafana not rendering dashboards** | Insecure configuration | Set `GF_SECURITY_ADMIN_USER` and `GF_SECURITY_ADMIN_PASSWORD` variables correctly |
|- **Alertmanager not sending alerts** | Incorrect configuration file | Update `alertmanager.yml` file |

---

## Changelog

| Date | Commit | Summary |
|------|--------|---------|
| 2026-04-09 | 8ca119ab | Initial documentation creation. |
| 2026-04-07 | 9f9ca1ad | Minor bug fixes and improvements. |
| 2026-04-07 | 71e3177f | Updated Prometheus and Grafana images to latest versions. |
| 2026-04-07 | 1df528ca | Added support for multi-arch images (Cadviser and Node Exporter). |
| 2026-04-07 | af94e455 | Improved Caddy reverse proxy configuration for Blackbox Exporter. |

---

## Notes
- Generated by Gremlin on 2026-04-10T03:17:27.514Z
- Source: swarm/monitoring.yaml