mirror of
https://github.com/yu-i-i/overleaf-cep.git
synced 2026-11-12 21:23:45 +01:00
Statistics with Grafana/Prometheus with sensor based on Overleaf instance statistics- #178
This commit is contained in:
David Rotermund
@@ -0,0 +1,141 @@
|
||||
# overleaf-cep monitoring (Prometheus + Grafana + exporters)
|
||||
|
||||
Stack in `/data_1/docker/compose` (modeled on the OlliTeX toolkit compose from
|
||||
`/data_1/image_mining/the_diff/overleaf/toolkit/compose.yaml`), all of it served
|
||||
through the existing nginx facade. Container/service names are unprefixed
|
||||
(monitoring-only, added 2026-10-27; renamed from `ollitex-*` on 2026-10-27).
|
||||
|
||||
## Services (all on the shared external network `overleaf-network`)
|
||||
|
||||
| service | role |
|
||||
|--------------------------------|----------------------------------------------------------------------|
|
||||
| `prometheus` | TSDB, 15d retention, `127.0.0.1:9090` (SSH-tunnel/admin only) |
|
||||
| `grafana` | UI, **no host port** — served under `https://<site>/grafana/` |
|
||||
| `mongodb-exporter` | percona exporter for the stack Mongo (sharelatex db) |
|
||||
| `mongodb-exporter-probe` | busybox spider: /metrics must answer (exporter image has no shell) |
|
||||
| `redis-exporter` | oliver006 exporter for the stack Redis |
|
||||
| `redis-exporter-probe` | busybox spider for the redis exporter /metrics |
|
||||
| `mongo-probe` | mongosh ping — protocol-level Mongo liveness (stack's own image) |
|
||||
| `node-exporter` | host CPU/mem/fs signals (reads host `/`, `/proc`, `/sys` ro) |
|
||||
| `instance-sensor` | **PR yu-i-i/overleaf-cep#178 sensors as live gauges** (see below) |
|
||||
|
||||
The repo's own helpers `down_grafana.sh` / `exec_grafana.sh` now address this
|
||||
`grafana` service directly.
|
||||
|
||||
## What is measured
|
||||
|
||||
1. **Overleaf app sensors** — the sharelatex image ships `@overleaf/metrics`
|
||||
on the web process (`overleaf:3000/metrics`, `app="web-api"`):
|
||||
`num_active_users` (live editor users), `timer_http_request` (summaries with
|
||||
quantiles), `mongo_command_started`, `nodejs_eventloop_lag_*`, GC/heap,
|
||||
`security_http_auth_check_credentials`, `editor_join_project`,
|
||||
`tpds_add_doc`, `lock_mongoTransaction_*`. (In-image side processes like
|
||||
compile `:4000` bind 127.0.0.1 inside their container — not scrapable.)
|
||||
2. **Instance stats — PR #178, integrated into Prometheus/Grafana instead of
|
||||
the admin page** (`instance-sensor`, port 9464):
|
||||
- users: total + active(24h) — optional standard/guest split via
|
||||
`INSTANCE_STATS_SEGMENTATION=true` (PR#176 rule: `'guest-user'` in
|
||||
`users.adminRoles`)
|
||||
- projects: total + active(24h, `lastUpdated`)
|
||||
- files: total docs
|
||||
- storage: mongodb `dbStats.totalSize`, filestore `du` of
|
||||
`seaweedfs-volume/data` (ro), redis RAM (`used_memory`) + redis disk
|
||||
(AOF size)
|
||||
Computed fresh on every scrape (no hourly Mongo snapshot state).
|
||||
3. **MongoDB** — connections, per-db data/storage size, oplog free space.
|
||||
4. **Redis** — memory, clients, command rate, keyspace hit/miss, AOF.
|
||||
5. **Host** — CPU, memory, root fs, load, uptime, network.
|
||||
6. **Filestore (SeaweedFS master `:9327`, native prometheus)** — leader,
|
||||
under-replicated volumes, in-flight uploads/downloads.
|
||||
7. **Scrape self-health** — `up` per job + the monitoring stack itself.
|
||||
|
||||
## URLs / access
|
||||
|
||||
- Grafana: `https://psintern.neuro.uni-bremen.de/grafana/`
|
||||
login: `admin` / (see top section of `.env`, `GRAFANA_ADMIN_USER/PASSWORD`;
|
||||
unset → Grafana fallback `admin`/`admin`)
|
||||
- Prometheus: `http://127.0.0.1:9090` (local only — SSH tunnel from a laptop)
|
||||
- Dashboard: folder **Overleaf** → **Overleaf — instance overview**
|
||||
(file-provisioned; pinned datasource uid `prometheus`,
|
||||
`grafana/provisioning/datasources/prometheus.yml`)
|
||||
|
||||
## Git hygiene
|
||||
|
||||
Data directories are git-ignored per folder: `grafana/.gitignore` →
|
||||
`grafana-data`, `prometheus/.gitignore` → `prometheus-data` (verified with
|
||||
`git check-ignore`). `.env` itself is in the root `.gitignore`. Keep new
|
||||
runtime dirs out of git by adding one line to the folder `.gitignore` as you
|
||||
add them.
|
||||
|
||||
## SMTP
|
||||
|
||||
Grafana reuses the stack's Overleaf relay — **no separate Grafana mail vars**;
|
||||
`grafana/compose.yaml` reads the existing Overleaf settings as a single
|
||||
source of truth:
|
||||
|
||||
| Grafana setting | taken from |
|
||||
|---|--------------------------------|
|
||||
| `GF_SMTP_ENABLED=true` | fixed |
|
||||
| `GF_SMTP_HOST` | `OVERLEAF_EMAIL_SMTP_HOST`:`OVERLEAF_EMAIL_SMTP_PORT` |
|
||||
| `GF_SMTP_USER` | `OVERLEAF_EMAIL_SMTP_USER` |
|
||||
| `GF_SMTP_PASSWORD` | `OVERLEAF_EMAIL_PASSWORD` |
|
||||
| `GF_SMTP_FROM_ADDRESS` | `OVERLEAF_EMAIL_FROM_ADDRESS` |
|
||||
| `GF_SMTP_FROM_NAME` | `"Overleaf Monitoring"` (display label only) |
|
||||
| `GF_SMTP_SKIP_VERIFY=false` | fixed |
|
||||
|
||||
Used for Grafana's reset-password mail (verified live: sent via the relay,
|
||||
server responded `200 {"message":"Email sent"}`). Grafana 11's alerting has no
|
||||
built-in SMTP contact point — SMTP covers account e-mail; alert routing would
|
||||
need integrations (webhook/Slack/…).
|
||||
|
||||
**Gotcha (learned 2026-10-27): the admin account's email must be set.**
|
||||
Grafana creates the admin user with email `admin@localhost`; the reset-mail
|
||||
endpoint `POST /api/user/password/send-reset-email {"userOrEmail": …}` matches
|
||||
users by that email and answers `200 "Email sent"` *even when nobody matches*
|
||||
(anti user-enumeration) — so it silently sends nothing. Fixed on two layers:
|
||||
1. **self-healing on fresh data**: `GF_SECURITY_ADMIN_EMAIL` in
|
||||
`grafana/compose.yaml` = `${OVERLEAF_ADMIN_EMAIL:-overleaf@uni-bremen.de}` —
|
||||
honored only when Grafana *creates* the admin (i.e. after a grafana-data
|
||||
wipe); it never rewrites an existing user row.
|
||||
2. **existing install**: `update user set email='overleaf@uni-bremen.de'
|
||||
where login='admin'` in `grafana-data/grafana.db` (grafana stopped).
|
||||
Verify with `select email from user where login='admin'`. The working relay
|
||||
itself was proven with a raw `smtplib.SMTP_SSL` test (TLS-465 + auth + RCPT all OK).
|
||||
|
||||
**After a grafana-data wipe** (fresh admin creation): login =
|
||||
`GRAFANA_ADMIN_USER`/`GRAFANA_ADMIN_PASSWORD` from `.env`; e-mail = `GF_SECURITY_ADMIN_EMAIL`
|
||||
(default `overleaf@uni-bremen.de`); dashboards re-provision automatically
|
||||
(file provider); admin UI settings/annotations are lost.
|
||||
|
||||
To disable SMTP: set `GF_SMTP_ENABLED: "false"` in `grafana/compose.yaml` +
|
||||
`docker compose up -d grafana`.
|
||||
|
||||
## Ops notes / gotchas (learned while bringing this up)
|
||||
|
||||
- **percona/mongodb_exporter 0.43 ships most collectors OFF by default**
|
||||
(a default run answers only `mongodb_up` + go/process metrics). Working
|
||||
flags for this stack are in `mongodb-exporter/compose.yaml`:
|
||||
`--collector.diagnosticdata --collector.dbstats --collector.dbstatsfreestorage
|
||||
--collector.replicasetstatus` (verified against this mongo 9.0).
|
||||
- Bind-mount ownership: `prometheus/prometheus-data` must be owned by uid
|
||||
**65534**, `grafana/grafana-data` by uid **472** (both images run
|
||||
unprivileged; docker creates fresh bind-mount dirs as root). Pre-create +
|
||||
`chown` if the dirs are ever deleted.
|
||||
- Grafana subpath: `GF_SERVER_SERVE_FROM_SUB_PATH=true` +
|
||||
`GF_SERVER_ROOT_URL=https://<site>/grafana/` — nginx serves `/grafana/`
|
||||
**without stripping** the prefix (see `nginx/nginx.conf`).
|
||||
After editing `nginx.conf`: `docker exec nginx nginx -s reload` (conf is
|
||||
bind-mounted, not baked in).
|
||||
- Grafana 11 login via API: `POST /grafana/login` with **JSON**
|
||||
`{user, password}` (the old `POST /api/login` is gone; form-encoded POSTs
|
||||
return `form-auth.invalid`).
|
||||
- Healthchecks: compose interpolates `${VAR}` inside healthcheck strings at
|
||||
load time — `MONGO_HOSTNAME`/`MONGO_PORT` therefore live in `.env`.
|
||||
- The `instance-sensor` image is a local build:
|
||||
`docker build -t overleaf-instance-sensor:local ./instance-sensor`.
|
||||
|
||||
## Adding/retiring
|
||||
|
||||
Everything opt-in per service; remove a block from the root `compose.yaml`
|
||||
and `docker compose <svc> down` to retire one. Image pins: `.env`
|
||||
(commented block in the monitoring section).
|
||||
Reference in New Issue
Block a user