From 3cc6928e50bd1bce6c66b4ce41e85a5752189432 Mon Sep 17 00:00:00 2001 From: David Rotermund Date: Mon, 26 Oct 2026 10:10:31 +0000 Subject: [PATCH] languagetools compose setup --- tools/languagetools_server_compose/.gitignore | 1 + tools/languagetools_server_compose/README.md | 155 ++++++++++++++++++ .../autoupdate_languagetool.sh | 34 ++++ .../languagetools_server_compose/compose.yaml | 7 + tools/languagetools_server_compose/down.sh | 2 + .../down_languagetool.sh | 3 + .../languagetool/.gitignore | 1 + .../languagetool/compose.yaml | 21 +++ .../languagetool/download_ngrams.sh | 71 ++++++++ .../languagetool/download_ngrams_untested.sh | 75 +++++++++ .../logs_languagetool.sh | 2 + .../languagetools_server_compose/lt-test.tex | 102 ++++++++++++ tools/languagetools_server_compose/up.sh | 2 + .../up_languagetool.sh | 2 + 14 files changed, 478 insertions(+) create mode 100644 tools/languagetools_server_compose/.gitignore create mode 100644 tools/languagetools_server_compose/README.md create mode 100755 tools/languagetools_server_compose/autoupdate_languagetool.sh create mode 100644 tools/languagetools_server_compose/compose.yaml create mode 100644 tools/languagetools_server_compose/down.sh create mode 100644 tools/languagetools_server_compose/down_languagetool.sh create mode 100644 tools/languagetools_server_compose/languagetool/.gitignore create mode 100644 tools/languagetools_server_compose/languagetool/compose.yaml create mode 100755 tools/languagetools_server_compose/languagetool/download_ngrams.sh create mode 100755 tools/languagetools_server_compose/languagetool/download_ngrams_untested.sh create mode 100755 tools/languagetools_server_compose/logs_languagetool.sh create mode 100644 tools/languagetools_server_compose/lt-test.tex create mode 100644 tools/languagetools_server_compose/up.sh create mode 100644 tools/languagetools_server_compose/up_languagetool.sh diff --git a/tools/languagetools_server_compose/.gitignore b/tools/languagetools_server_compose/.gitignore new file mode 100644 index 0000000000..2eea525d88 --- /dev/null +++ b/tools/languagetools_server_compose/.gitignore @@ -0,0 +1 @@ +.env \ No newline at end of file diff --git a/tools/languagetools_server_compose/README.md b/tools/languagetools_server_compose/README.md new file mode 100644 index 0000000000..fbde1a71e2 --- /dev/null +++ b/tools/languagetools_server_compose/README.md @@ -0,0 +1,155 @@ +# LanguageTool server (Docker Compose) + +Standalone [LanguageTool](https://languagetool.org) grammar-check server used by +the Overleaf integration in `services/web/modules/languagetool` +(Overleaf CE fork repo, see the module's `README.md`). + +- **Image:** `erikvl87/languagetool:latest` (community server fork of LanguageTool; + supports `level=default` and `level=picky` — other levels return HTTP 400) +- **Container name:** `languagetool` (overleaf reaches it by this hostname on the shared network) +- **Host port:** `127.0.0.1:32768` → container `8010` (pinned, stable across restarts) +- **Compose project name:** `languagetools_server_compose` (directory name) + +## Layout + +``` +tools/languagetools_server_compose/ +├── compose.yaml # top-level: `extends` the service in languagetool/compose.yaml +├── up.sh # docker compose up -d +├── down.sh # docker compose down +├── up_languagetool.sh # docker compose up -d languagetool +├── down_languagetool.sh # docker compose down languagetool +├── logs_languagetool.sh # docker compose logs -f languagetool +├── autoupdate_languagetool.sh # pull latest image, recreate container only if changed +├── lt-test.tex # test document: sentences with verified expected detections +└── languagetool/ + ├── compose.yaml # actual service definition + ├── download_ngrams.sh # main n-gram set (de en es fr nl) → ngrams/ + ├── download_ngrams_untested.sh # untested set (he it ru zh) → ngrams/untested/ + └── ngrams/ # n-gram data, mounted read-only into the container (gitignored) +``` + +## Quick start + +```bash +cd tools/languagetools_server_compose + +# 1. Populate n-gram data (one-time, ~45 GB disk; resumable, idempotent) +./languagetool/download_ngrams.sh +./languagetool/download_ngrams_untested.sh # optional: he/it/ru/zh + +# 2. Start +./up.sh + +# 3. Verify (container must be healthy) +docker ps --filter name=languagetool +curl -s http://127.0.0.1:32768/v2/languages | head -c 200 +``` + +## N-gram data + +LanguageTool loads per-language n-gram models from the directory given in +`langtool_languageModel` (here: `/ngrams` ← `languagetool/ngrams/`). +Every subdirectory named after a language code (e.g. `en/`, `de/`) is picked +up — currently present: `de en es fr nl` (main) and `he it ru zh` (untested). + +| Set | Languages | Size (extracted) | Zip sources | +|---|---|---|---| +| main | de en es fr nl | ~24 GB | `ngrams--YYYYMMDD.zip` (~1.7–9 GB each) | +| untested | he it ru zh | ~14 GB | `ngram--YYYYMMDD.zip` (~0.2–2.3 GB each) | + +- Downloads come from `https://languagetool.org/download/ngram-data/` + (untested set: `.../ngram-data/untested/`). +- Both scripts are idempotent: existing zips are skip-validated (`unzip -t`), + downloads are resumable (`curl -C -`). Re-run safely after an interrupted transfer. +- The downloaded zip archives stay in `ngrams/` (~21 GB total). + Safe to `rm ngrams/*.zip` once extraction succeeded, to reclaim that space. +- `an` (Aragonese) is **not** available on the server; it is kept commented + out in `download_ngrams_untested.sh` in case it appears later. + +> **Pitfall (cost us a debugging session):** if `ngrams/` does not exist on +> the host, Docker creates an *empty* directory for the bind mount and the +> server runs **without any n-gram models** — spelling detection +> (`recieve`, `realy`, …) silently degrades while the API still answers. +> Verify with `ls languagetool/ngrams/en` and a check request that +> `MORFOLOGIK_RULE_EN_US` still fires. + +## Service configuration (`languagetool/compose.yaml`) + +| Setting | Value | Notes | +|---|---|---| +| `image` | `erikvl87/languagetool:latest` | community fork, HTTP REST API on 8010 | +| `Java_Xms` / `Java_Xmx` | `512m` / `2g` | JVM heap | +| `langtool_languageModel` | `/ngrams` | n-gram model directory (read-only bind mount) | +| `ports` | `127.0.0.1:32768:8010` | host port pinned — do not reintroduce the ephemeral `127.0.0.1::8010` form | +| `healthcheck` | `wget --spider http://localhost:8010/v2/languages` | 30s interval, 60s start period | +| `restart` | `unless-stopped` | | + +## Connecting Overleaf + +The matching Overleaf side (CE fork v6.3.1) exposes the feature through +`services/web/modules/languagetool/` — enabled via overleaf service env vars. +In this deployment they live in `/data_1/docker/compose/overleaf/compose.yaml`: + +```yaml +env: + LANGUAGE_TOOL_ENABLED: "true" + LANGUAGE_TOOL_HOST: languagetool # resolves on the shared docker network + LANGUAGE_TOOL_PORT: "8010" +``` + +- The overleaf container reaches LT through the shared network + (`languagetools_server_compose_default`); from inside overleaf the host is + `languagetool:8010`, from the host it is `127.0.0.1:32768`. +- Overleaf checks `/v2/check` (POST, form-encoded `text`, `language`, `level`, + plus `disabledRules`); client-side defaults: **picky level on**, + spellcheck language = Overleaf default (`en-US`), and the checks are + stored as **project settings** (shared with collaborators). +- If `LANGUAGE_TOOL_ENABLED` is false or the proxy can't reach LT, Overleaf + simply does not offer the feature (no client errors). + +## Verification + +```bash +# basic +curl -s -X POST http://127.0.0.1:32768/v2/check \ + -d 'language=en-US' -d 'text=You can recieve a bonus.' -d 'level=picky' + +# picky vs default (passive voice is picky-only on this build) +curl -s -X POST http://127.0.0.1:32768/v2/check \ + --data-urlencode 'language=en-US' --data-urlencode 'text=The error was detected by the test suite.' \ + -d 'level=default' # → no passive-voice match +curl -s -X POST http://127.0.0.1:32768/v2/check \ + --data-urlencode 'language=en-US' --data-urlencode 'text=The error was detected by the test suite.' \ + -d 'level=picky' # → PASSIVE_VOICE_SIMPLE +``` + +Editor test document: **`lt-test.tex`** in this directory. +Verified expectations (2026-10-26, with the exact `disabledRules` the proxy sends): + +| Picky mode | Underlines | +|---|---| +| **on** (Overleaf default) | **27** — section A (2 passive-voice) + section B (23) | +| **off** | **25** — section A disappears, section B stays | + +See the file header for the per-sentence rule annotations. + +## Update / rollback + +```bash +./autoupdate_languagetool.sh # pull erikvl87/languagetool:latest, recreate only if image changed +docker compose down && docker compose up -d # manual restart +docker compose logs -f languagetool # tail logs +``` + +Troubleshooting: + +- **Overleaf can't reach LT** — from the overleaf container: + `wget -O- http://languagetool:8010/v2/languages`. If DNS fails, the overleaf + service is not attached to the `languagetools_server_compose_default` network + (see the two `networks` blocks above; both must carry the same name). +- **Port changed after a recreate** — always use the pinned form + `"127.0.0.1:32768:8010"`; the old `"127.0.0.1::8010"` form picks a random + host port per container creation. +- **No spelling detections** — n-gram mount empty, see the pitfall above. + diff --git a/tools/languagetools_server_compose/autoupdate_languagetool.sh b/tools/languagetools_server_compose/autoupdate_languagetool.sh new file mode 100755 index 0000000000..68e28da928 --- /dev/null +++ b/tools/languagetools_server_compose/autoupdate_languagetool.sh @@ -0,0 +1,34 @@ +#!/usr/bin/env bash +set -euo pipefail + +SERVICE_NAME="languagetool" +IMAGE_NAME="erikvl87/languagetool:latest" +YAML_DIRECTORY_NAME="${COMPOSE_DIRECTORY:-$(pwd)}" + +# Get current image ID (short format) +CURRENT_IMAGE=$(docker inspect --format='{{.Image}}' "${SERVICE_NAME}" 2>/dev/null | cut -d: -f2 | cut -c1-12) || true + +# Pull the latest image +echo "Pulling latest image..." +docker pull "${IMAGE_NAME}" + +# Get new image ID (short format) +NEW_IMAGE=$(docker images "${IMAGE_NAME}" --format '{{.ID}}') + +echo "Current image: $CURRENT_IMAGE" +echo "Latest image: $NEW_IMAGE" + +# Compare and restart only if different +if [ -z "$CURRENT_IMAGE" ]; then + echo "Container not found, starting it up..." + cd "${YAML_DIRECTORY_NAME}" + docker compose up -d "${SERVICE_NAME}" +elif [ "$CURRENT_IMAGE" != "$NEW_IMAGE" ]; then + echo "New image detected, restarting container..." + cd "${YAML_DIRECTORY_NAME}" + docker compose down "${SERVICE_NAME}" + docker compose up -d "${SERVICE_NAME}" + echo "Update completed!" +else + echo "Already running the latest version, no restart needed." +fi diff --git a/tools/languagetools_server_compose/compose.yaml b/tools/languagetools_server_compose/compose.yaml new file mode 100644 index 0000000000..a92a689e5c --- /dev/null +++ b/tools/languagetools_server_compose/compose.yaml @@ -0,0 +1,7 @@ +services: + languagetool: + extends: + file: ./languagetool/compose.yaml + service: languagetool + + diff --git a/tools/languagetools_server_compose/down.sh b/tools/languagetools_server_compose/down.sh new file mode 100644 index 0000000000..c8642093b6 --- /dev/null +++ b/tools/languagetools_server_compose/down.sh @@ -0,0 +1,2 @@ +docker compose down + diff --git a/tools/languagetools_server_compose/down_languagetool.sh b/tools/languagetools_server_compose/down_languagetool.sh new file mode 100644 index 0000000000..46f3c82a63 --- /dev/null +++ b/tools/languagetools_server_compose/down_languagetool.sh @@ -0,0 +1,3 @@ +docker compose down languagetool + + diff --git a/tools/languagetools_server_compose/languagetool/.gitignore b/tools/languagetools_server_compose/languagetool/.gitignore new file mode 100644 index 0000000000..32b1038359 --- /dev/null +++ b/tools/languagetools_server_compose/languagetool/.gitignore @@ -0,0 +1 @@ +ngrams diff --git a/tools/languagetools_server_compose/languagetool/compose.yaml b/tools/languagetools_server_compose/languagetool/compose.yaml new file mode 100644 index 0000000000..c1e878d2d3 --- /dev/null +++ b/tools/languagetools_server_compose/languagetool/compose.yaml @@ -0,0 +1,21 @@ +services: + languagetool: + image: erikvl87/languagetool:latest + container_name: languagetool + hostname: languagetool + restart: unless-stopped + environment: + Java_Xms: 512m + Java_Xmx: 2g + langtool_languageModel: /ngrams + ports: + - "127.0.0.1:32768:8010" + volumes: + - ./ngrams:/ngrams:ro + healthcheck: + test: ["CMD-SHELL", "wget --quiet --spider --timeout=5 http://localhost:8010/v2/languages 2>&1 || exit 1"] + interval: 30s + timeout: 10s + retries: 5 + start_period: 60s + diff --git a/tools/languagetools_server_compose/languagetool/download_ngrams.sh b/tools/languagetools_server_compose/languagetool/download_ngrams.sh new file mode 100755 index 0000000000..00cfce3a5b --- /dev/null +++ b/tools/languagetools_server_compose/languagetool/download_ngrams.sh @@ -0,0 +1,71 @@ +#!/usr/bin/env bash +# Download & extract LanguageTool n-gram data (main set: de en es fr nl) +# into ngrams/ next to this script. +# +# Sources: https://languagetool.org/download/ngram-data/ +# ngrams-de-20150819.zip (~1.7 GB) +# ngrams-en-20150817.zip (~9.0 GB) +# ngrams-es-20150915.zip (~1.7 GB) +# ngrams-fr-20150913.zip (~1.8 GB) +# ngrams-nl-20181229.zip (~1.2 GB) +# +# Usage: ./download_ngrams.sh +# Re-running is safe: already-downloaded & valid zips are skipped (download is +# resumable via curl -C). + +set -euo pipefail + +BASE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +NGRAMS_DIR="$BASE/ngrams" +BASE_URL="https://languagetool.org/download/ngram-data" + +# lang -> zip filename (verified against server directory listing) +declare -a FILES=( + "ngrams-de-20150819.zip" + "ngrams-en-20150817.zip" + "ngrams-es-20150915.zip" + "ngrams-fr-20150913.zip" + "ngrams-nl-20181229.zip" +) + +command -v curl >/dev/null || { echo "ERROR: curl not found" >&2; exit 1; } +command -v unzip >/dev/null || { echo "ERROR: unzip not found" >&2; exit 1; } + +mkdir -p "$NGRAMS_DIR" +echo "Target dir: $NGRAMS_DIR" + +failed=0 +for zip in "${FILES[@]}"; do + url="$BASE_URL/$zip" + dest="$NGRAMS_DIR/$zip" + + echo "==================================================================" + echo "$zip" + + if [ -f "$dest" ] && unzip -t "$dest" >/dev/null 2>&1; then + echo " [skip] already downloaded and valid: $zip" + else + echo " [dl] $url" + if ! curl --fail --location --retry 3 --retry-delay 5 -C - \ + --connect-timeout 30 --progress-bar -o "$dest" "$url"; then + echo " [FAIL] download failed for $url (partial file kept for resume: $dest)" >&2 + rm -f "$dest" # 404/error body would corrupt the name for future runs + failed=1 + continue + fi + fi + + echo " [extract] -> $NGRAMS_DIR" + if ! unzip -o -q "$dest" -d "$NGRAMS_DIR"; then + echo " [FAIL] extraction failed for $zip" >&2 + failed=1 + fi +done + +echo "==================================================================" +if [ "$failed" -ne 0 ]; then + echo "Finished with ERRORS — re-run to resume failed downloads." >&2 + exit 1 +fi +echo "All done. Contents:" +ls -lh "$NGRAMS_DIR" diff --git a/tools/languagetools_server_compose/languagetool/download_ngrams_untested.sh b/tools/languagetools_server_compose/languagetool/download_ngrams_untested.sh new file mode 100755 index 0000000000..9aef2271db --- /dev/null +++ b/tools/languagetools_server_compose/languagetool/download_ngrams_untested.sh @@ -0,0 +1,75 @@ +#!/usr/bin/env bash +# Download & extract LanguageTool n-gram data (UNTESTED set) into ngrams/untested/ +# next to this script. +# +# Sources: https://languagetool.org/download/ngram-data/untested/ +# ngram-he-20150916.zip (~225 MB) +# ngram-it-20150915.zip (~1.1 GB) +# ngram-ru-20150914.zip (~2.3 GB) +# ngram-zh-20150916.zip (~1.7 GB) +# +# NOTE: "an" (Aragonese) was requested but does NOT exist on the server +# (checked ngram-data/, untested/, temp/, archive/ — only he/it/ru/zh are +# offered). Kept commented out so it can be enabled if it appears later. +# +# Usage: ./download_ngrams_untested.sh +# Re-running is safe: already-downloaded & valid zips are skipped (download is +# resumable via curl -C). + +set -euo pipefail + +BASE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +NGRAMS_DIR="$BASE/ngrams" +UNTESTED_DIR="$NGRAMS_DIR" +BASE_URL="https://languagetool.org/download/ngram-data/untested" + +# zip filenames (verified against server directory listing) +declare -a FILES=( + "ngram-he-20150916.zip" + "ngram-it-20150915.zip" + "ngram-ru-20150914.zip" + "ngram-zh-20150916.zip" + # "ngram-an-*.zip (404 — not available on the server as of 2025/2026)" +) + +command -v curl >/dev/null || { echo "ERROR: curl not found" >&2; exit 1; } +command -v unzip >/dev/null || { echo "ERROR: unzip not found" >&2; exit 1; } + +mkdir -p "$UNTESTED_DIR" +echo "Target dir: $UNTESTED_DIR" + +failed=0 +for zip in "${FILES[@]}"; do + url="$BASE_URL/$zip" + dest="$UNTESTED_DIR/$zip" + + echo "==================================================================" + echo "$zip" + + if [ -f "$dest" ] && unzip -t "$dest" >/dev/null 2>&1; then + echo " [skip] already downloaded and valid: $zip" + else + echo " [dl] $url" + if ! curl --fail --location --retry 3 --retry-delay 5 -C - \ + --connect-timeout 30 --progress-bar -o "$dest" "$url"; then + echo " [FAIL] download failed for $url (partial file kept for resume: $dest)" >&2 + rm -f "$dest" # 404/error body would corrupt the name for future runs + failed=1 + continue + fi + fi + + echo " [extract] -> $UNTESTED_DIR" + if ! unzip -o -q "$dest" -d "$UNTESTED_DIR"; then + echo " [FAIL] extraction failed for $zip" >&2 + failed=1 + fi +done + +echo "==================================================================" +if [ "$failed" -ne 0 ]; then + echo "Finished with ERRORS — re-run to resume failed downloads." >&2 + exit 1 +fi +echo "All done. Contents:" +ls -lh "$UNTESTED_DIR" diff --git a/tools/languagetools_server_compose/logs_languagetool.sh b/tools/languagetools_server_compose/logs_languagetool.sh new file mode 100755 index 0000000000..f1b0670c34 --- /dev/null +++ b/tools/languagetools_server_compose/logs_languagetool.sh @@ -0,0 +1,2 @@ +docker compose logs -f languagetool + diff --git a/tools/languagetools_server_compose/lt-test.tex b/tools/languagetools_server_compose/lt-test.tex new file mode 100644 index 0000000000..a63eabb256 --- /dev/null +++ b/tools/languagetools_server_compose/lt-test.tex @@ -0,0 +1,102 @@ +% LanguageTool level test — ext-ce-v6.3.1 + erikvl87/languagetool +% +% Section A: expected to be flagged ONLY in Picky mode (level=picky). +% Section B: expected to be flagged in BOTH modes (default and picky). +% +% All sentences below were verified live against the running server +% (2026-10-26), with the exact disabledRules the Overleaf proxy sends. +% +% Usage: open in Overleaf, wait ~2 s after the last edit, +% hover a wavy underline for the message + one-click replacements. +% Toggle Settings → Spelling and Language → LanguageTool → "Picky mode" +% off/on and compare: section A marks should appear/disappear. + +\documentclass{article} +\begin{document} + +\section*{A: Picky-only (missed at default level)} + +% PASSIVE_VOICE_SIMPLE — passive voice +The experiment was conducted by our group. + +% PASSIVE_VOICE_SIMPLE — passive voice +The error was detected by the test suite. + +\section*{B: Both modes (default and picky)} + +% EN_A_VS_AN — wrong article +An sequence off independent variables. + +% MORFOLOGIK_RULE_EN_US — spelling mistake +You can recieve a bonus for this. + +% MORFOLOGIK_RULE_EN_US — spelling mistake +This is a realy interesting result. + +% MORFOLOGIK_RULE_EN_US — spelling mistake +He shoulda told us about the crash. + +% AGREEMENT_SENT_START — agreement error +The results shows a significant difference. + +% BEEN_PART_AGREEMENT — possible agreement error +A new paper was publish last week. + +% THERE_S_MANY — possible agreement error +There is three reasons for this choice. + +% IT_IS — possible typo (missing apostrophe) +Its time to finish the manuscript. + +% AFFECT_EFFECT — commonly confused word +This should have an affect on the outcome. + +% LOOSE_LOSE — possible typo +You will loose the key if you leave it here. + +% MODAL_OF — possible typo +You should of checked the configuration. + +% MODAL_OF — possible typo +It might of happened last night. + +% MODAL_OF — possible typo +We could of fixed the bug yesterday. + +% INFORMATIONS — nonstandard word +We need more informations about this topic. + +% EN_A_VS_AN — wrong article (uncountable) +He gave me a advice yesterday morning. + +% ENGLISH_WORD_REPEAT_RULE — word repetition +The the method was tested twice. + +% ADMIT_ENJOY_VB — use 'ing' +I look forward to hear from you again. + +% TOO_TO — possible typo +I want to too participate in this. + +% AGREEMENT_SENT_START — agreement error +Each of the students have passed the exam. + +% AGREEMENT_SENT_START — agreement error +The dog which barks is friendly. + +% RB_RB_COMMA — intensifier comma +We have a very very good result here. + +% FREE_GIFT — redundant phrase +This is a free gift for all of you. + +% ADMIT_ENJOY_VB — use 'ing' +The manual recommends to restart the server. + +% ALLOW_TO — use a gerund +The tool allows to configure the output. + +% ALLOW_TO — use a gerund +The server requires to restart after the change. + +\end{document} diff --git a/tools/languagetools_server_compose/up.sh b/tools/languagetools_server_compose/up.sh new file mode 100644 index 0000000000..a4a5dbb01a --- /dev/null +++ b/tools/languagetools_server_compose/up.sh @@ -0,0 +1,2 @@ +docker compose up -d + diff --git a/tools/languagetools_server_compose/up_languagetool.sh b/tools/languagetools_server_compose/up_languagetool.sh new file mode 100644 index 0000000000..9fbf48ed78 --- /dev/null +++ b/tools/languagetools_server_compose/up_languagetool.sh @@ -0,0 +1,2 @@ +docker compose up -d languagetool +