languagetools compose setup

2026-11-12 21:23:45 +01:00 · 2026-10-26 10:10:31 +00:00
parent 9d4ec4ff96
commit 3cc6928e50
14 changed files with 478 additions and 0 deletions
@@ -0,0 +1 @@
+.env
@@ -0,0 +1,155 @@
+# LanguageTool server (Docker Compose)
+
+Standalone [LanguageTool](https://languagetool.org) grammar-check server used by
+the Overleaf integration in `services/web/modules/languagetool`
+(Overleaf CE fork repo, see the module's `README.md`).
+
+- **Image:** `erikvl87/languagetool:latest` (community server fork of LanguageTool;
+  supports `level=default` and `level=picky` — other levels return HTTP 400)
+- **Container name:** `languagetool` (overleaf reaches it by this hostname on the shared network)
+- **Host port:** `127.0.0.1:32768` → container `8010` (pinned, stable across restarts)
+- **Compose project name:** `languagetools_server_compose` (directory name)
+
+## Layout
+
+```
+tools/languagetools_server_compose/
+├── compose.yaml            # top-level: `extends` the service in languagetool/compose.yaml
+├── up.sh                   # docker compose up -d
+├── down.sh                 # docker compose down
+├── up_languagetool.sh      # docker compose up -d languagetool
+├── down_languagetool.sh    # docker compose down languagetool
+├── logs_languagetool.sh    # docker compose logs -f languagetool
+├── autoupdate_languagetool.sh   # pull latest image, recreate container only if changed
+├── lt-test.tex             # test document: sentences with verified expected detections
+└── languagetool/
+    ├── compose.yaml        # actual service definition
+    ├── download_ngrams.sh          # main n-gram set  (de en es fr nl)  → ngrams/
+    ├── download_ngrams_untested.sh # untested set      (he it ru zh)    → ngrams/untested/
+    └── ngrams/             # n-gram data, mounted read-only into the container (gitignored)
+```
+
+## Quick start
+
+```bash
+cd tools/languagetools_server_compose
+
+# 1. Populate n-gram data (one-time, ~45 GB disk; resumable, idempotent)
+./languagetool/download_ngrams.sh
+./languagetool/download_ngrams_untested.sh   # optional: he/it/ru/zh
+
+# 2. Start
+./up.sh
+
+# 3. Verify (container must be healthy)
+docker ps --filter name=languagetool
+curl -s http://127.0.0.1:32768/v2/languages | head -c 200
+```
+
+## N-gram data
+
+LanguageTool loads per-language n-gram models from the directory given in
+`langtool_languageModel` (here: `/ngrams` ← `languagetool/ngrams/`).
+Every subdirectory named after a language code (e.g. `en/`, `de/`) is picked
+up — currently present: `de en es fr nl` (main) and `he it ru zh` (untested).
+
+| Set | Languages | Size (extracted) | Zip sources |
+|---|---|---|---|
+| main | de en es fr nl | ~24 GB | `ngrams-<lang>-YYYYMMDD.zip` (~1.7–9 GB each) |
+| untested | he it ru zh | ~14 GB | `ngram-<lang>-YYYYMMDD.zip` (~0.2–2.3 GB each) |
+
+- Downloads come from `https://languagetool.org/download/ngram-data/`
+  (untested set: `.../ngram-data/untested/`).
+- Both scripts are idempotent: existing zips are skip-validated (`unzip -t`),
+  downloads are resumable (`curl -C -`). Re-run safely after an interrupted transfer.
+- The downloaded zip archives stay in `ngrams/` (~21 GB total).
+  Safe to `rm ngrams/*.zip` once extraction succeeded, to reclaim that space.
+- `an` (Aragonese) is **not** available on the server; it is kept commented
+  out in `download_ngrams_untested.sh` in case it appears later.
+
+> **Pitfall (cost us a debugging session):** if `ngrams/` does not exist on
+> the host, Docker creates an *empty* directory for the bind mount and the
+> server runs **without any n-gram models** — spelling detection
+> (`recieve`, `realy`, …) silently degrades while the API still answers.
+> Verify with `ls languagetool/ngrams/en` and a check request that
+> `MORFOLOGIK_RULE_EN_US` still fires.
+
+## Service configuration (`languagetool/compose.yaml`)
+
+| Setting | Value | Notes |
+|---|---|---|
+| `image` | `erikvl87/languagetool:latest` | community fork, HTTP REST API on 8010 |
+| `Java_Xms` / `Java_Xmx` | `512m` / `2g` | JVM heap |
+| `langtool_languageModel` | `/ngrams` | n-gram model directory (read-only bind mount) |
+| `ports` | `127.0.0.1:32768:8010` | host port pinned — do not reintroduce the ephemeral `127.0.0.1::8010` form |
+| `healthcheck` | `wget --spider http://localhost:8010/v2/languages` | 30s interval, 60s start period |
+| `restart` | `unless-stopped` | |
+
+## Connecting Overleaf
+
+The matching Overleaf side (CE fork v6.3.1) exposes the feature through
+`services/web/modules/languagetool/` — enabled via overleaf service env vars.
+In this deployment they live in `/data_1/docker/compose/overleaf/compose.yaml`:
+
+```yaml
+env:
+  LANGUAGE_TOOL_ENABLED: "true"
+  LANGUAGE_TOOL_HOST: languagetool   # resolves on the shared docker network
+  LANGUAGE_TOOL_PORT: "8010"
+```
+
+- The overleaf container reaches LT through the shared network
+  (`languagetools_server_compose_default`); from inside overleaf the host is
+  `languagetool:8010`, from the host it is `127.0.0.1:32768`.
+- Overleaf checks `/v2/check` (POST, form-encoded `text`, `language`, `level`,
+  plus `disabledRules`); client-side defaults: **picky level on**,
+  spellcheck language = Overleaf default (`en-US`), and the checks are
+  stored as **project settings** (shared with collaborators).
+- If `LANGUAGE_TOOL_ENABLED` is false or the proxy can't reach LT, Overleaf
+  simply does not offer the feature (no client errors).
+
+## Verification
+
+```bash
+# basic
+curl -s -X POST http://127.0.0.1:32768/v2/check \
+  -d 'language=en-US' -d 'text=You can recieve a bonus.' -d 'level=picky'
+
+# picky vs default (passive voice is picky-only on this build)
+curl -s -X POST http://127.0.0.1:32768/v2/check \
+  --data-urlencode 'language=en-US' --data-urlencode 'text=The error was detected by the test suite.' \
+  -d 'level=default'   # → no passive-voice match
+curl -s -X POST http://127.0.0.1:32768/v2/check \
+  --data-urlencode 'language=en-US' --data-urlencode 'text=The error was detected by the test suite.' \
+  -d 'level=picky'     # → PASSIVE_VOICE_SIMPLE
+```
+
+Editor test document: **`lt-test.tex`** in this directory.
+Verified expectations (2026-10-26, with the exact `disabledRules` the proxy sends):
+
+| Picky mode | Underlines |
+|---|---|
+| **on** (Overleaf default) | **27** — section A (2 passive-voice) + section B (23) |
+| **off** | **25** — section A disappears, section B stays |
+
+See the file header for the per-sentence rule annotations.
+
+## Update / rollback
+
+```bash
+./autoupdate_languagetool.sh   # pull erikvl87/languagetool:latest, recreate only if image changed
+docker compose down && docker compose up -d        # manual restart
+docker compose logs -f languagetool                # tail logs
+```
+
+Troubleshooting:
+
+- **Overleaf can't reach LT** — from the overleaf container:
+  `wget -O- http://languagetool:8010/v2/languages`. If DNS fails, the overleaf
+  service is not attached to the `languagetools_server_compose_default` network
+  (see the two `networks` blocks above; both must carry the same name).
+- **Port changed after a recreate** — always use the pinned form
+  `"127.0.0.1:32768:8010"`; the old `"127.0.0.1::8010"` form picks a random
+  host port per container creation.
+- **No spelling detections** — n-gram mount empty, see the pitfall above.
+
@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+SERVICE_NAME="languagetool"
+IMAGE_NAME="erikvl87/languagetool:latest"
+YAML_DIRECTORY_NAME="${COMPOSE_DIRECTORY:-$(pwd)}"
+
+# Get current image ID (short format)
+CURRENT_IMAGE=$(docker inspect --format='{{.Image}}' "${SERVICE_NAME}" 2>/dev/null | cut -d: -f2 | cut -c1-12) || true
+
+# Pull the latest image
+echo "Pulling latest image..."
+docker pull "${IMAGE_NAME}"
+
+# Get new image ID (short format)
+NEW_IMAGE=$(docker images "${IMAGE_NAME}" --format '{{.ID}}')
+
+echo "Current image: $CURRENT_IMAGE"
+echo "Latest image:  $NEW_IMAGE"
+
+# Compare and restart only if different
+if [ -z "$CURRENT_IMAGE" ]; then
+    echo "Container not found, starting it up..."
+    cd "${YAML_DIRECTORY_NAME}"
+    docker compose up -d "${SERVICE_NAME}"
+elif [ "$CURRENT_IMAGE" != "$NEW_IMAGE" ]; then
+    echo "New image detected, restarting container..."
+    cd "${YAML_DIRECTORY_NAME}"
+    docker compose down "${SERVICE_NAME}"
+    docker compose up -d "${SERVICE_NAME}"
+    echo "Update completed!"
+else
+    echo "Already running the latest version, no restart needed."
+fi
@@ -0,0 +1,7 @@
+services:
+  languagetool:
+    extends:
+      file: ./languagetool/compose.yaml
+      service: languagetool
+
+
@@ -0,0 +1,2 @@
+docker compose down
+
@@ -0,0 +1,3 @@
+docker compose down languagetool
+
+
@@ -0,0 +1 @@
+ngrams
@@ -0,0 +1,21 @@
+services:
+  languagetool:
+    image: erikvl87/languagetool:latest
+    container_name: languagetool
+    hostname: languagetool
+    restart: unless-stopped
+    environment:
+      Java_Xms: 512m
+      Java_Xmx: 2g
+      langtool_languageModel: /ngrams
+    ports:
+      - "127.0.0.1:32768:8010"
+    volumes:
+      - ./ngrams:/ngrams:ro
+    healthcheck:
+      test: ["CMD-SHELL", "wget --quiet --spider --timeout=5 http://localhost:8010/v2/languages 2>&1 || exit 1"]
+      interval: 30s
+      timeout: 10s
+      retries: 5
+      start_period: 60s
+
@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+# Download & extract LanguageTool n-gram data (main set: de en es fr nl)
+# into ngrams/ next to this script.
+#
+# Sources: https://languagetool.org/download/ngram-data/
+#   ngrams-de-20150819.zip  (~1.7 GB)
+#   ngrams-en-20150817.zip  (~9.0 GB)
+#   ngrams-es-20150915.zip  (~1.7 GB)
+#   ngrams-fr-20150913.zip  (~1.8 GB)
+#   ngrams-nl-20181229.zip  (~1.2 GB)
+#
+# Usage: ./download_ngrams.sh
+# Re-running is safe: already-downloaded & valid zips are skipped (download is
+# resumable via curl -C).
+
+set -euo pipefail
+
+BASE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+NGRAMS_DIR="$BASE/ngrams"
+BASE_URL="https://languagetool.org/download/ngram-data"
+
+# lang -> zip filename (verified against server directory listing)
+declare -a FILES=(
+  "ngrams-de-20150819.zip"
+  "ngrams-en-20150817.zip"
+  "ngrams-es-20150915.zip"
+  "ngrams-fr-20150913.zip"
+  "ngrams-nl-20181229.zip"
+)
+
+command -v curl  >/dev/null || { echo "ERROR: curl not found"  >&2; exit 1; }
+command -v unzip >/dev/null || { echo "ERROR: unzip not found" >&2; exit 1; }
+
+mkdir -p "$NGRAMS_DIR"
+echo "Target dir: $NGRAMS_DIR"
+
+failed=0
+for zip in "${FILES[@]}"; do
+  url="$BASE_URL/$zip"
+  dest="$NGRAMS_DIR/$zip"
+
+  echo "=================================================================="
+  echo "$zip"
+
+  if [ -f "$dest" ] && unzip -t "$dest" >/dev/null 2>&1; then
+    echo "  [skip] already downloaded and valid: $zip"
+  else
+    echo "  [dl] $url"
+    if ! curl --fail --location --retry 3 --retry-delay 5 -C - \
+             --connect-timeout 30 --progress-bar -o "$dest" "$url"; then
+      echo "  [FAIL] download failed for $url (partial file kept for resume: $dest)" >&2
+      rm -f "$dest"  # 404/error body would corrupt the name for future runs
+      failed=1
+      continue
+    fi
+  fi
+
+  echo "  [extract] -> $NGRAMS_DIR"
+  if ! unzip -o -q "$dest" -d "$NGRAMS_DIR"; then
+    echo "  [FAIL] extraction failed for $zip" >&2
+    failed=1
+  fi
+done
+
+echo "=================================================================="
+if [ "$failed" -ne 0 ]; then
+  echo "Finished with ERRORS — re-run to resume failed downloads." >&2
+  exit 1
+fi
+echo "All done. Contents:"
+ls -lh "$NGRAMS_DIR"
@@ -0,0 +1,75 @@
+#!/usr/bin/env bash
+# Download & extract LanguageTool n-gram data (UNTESTED set) into ngrams/untested/
+# next to this script.
+#
+# Sources: https://languagetool.org/download/ngram-data/untested/
+#   ngram-he-20150916.zip  (~225 MB)
+#   ngram-it-20150915.zip  (~1.1 GB)
+#   ngram-ru-20150914.zip  (~2.3 GB)
+#   ngram-zh-20150916.zip  (~1.7 GB)
+#
+# NOTE: "an" (Aragonese) was requested but does NOT exist on the server
+# (checked ngram-data/, untested/, temp/, archive/ — only he/it/ru/zh are
+# offered). Kept commented out so it can be enabled if it appears later.
+#
+# Usage: ./download_ngrams_untested.sh
+# Re-running is safe: already-downloaded & valid zips are skipped (download is
+# resumable via curl -C).
+
+set -euo pipefail
+
+BASE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+NGRAMS_DIR="$BASE/ngrams"
+UNTESTED_DIR="$NGRAMS_DIR"
+BASE_URL="https://languagetool.org/download/ngram-data/untested"
+
+# zip filenames (verified against server directory listing)
+declare -a FILES=(
+  "ngram-he-20150916.zip"
+  "ngram-it-20150915.zip"
+  "ngram-ru-20150914.zip"
+  "ngram-zh-20150916.zip"
+  # "ngram-an-*.zip  (404 — not available on the server as of 2025/2026)"
+)
+
+command -v curl  >/dev/null || { echo "ERROR: curl not found"  >&2; exit 1; }
+command -v unzip >/dev/null || { echo "ERROR: unzip not found" >&2; exit 1; }
+
+mkdir -p "$UNTESTED_DIR"
+echo "Target dir: $UNTESTED_DIR"
+
+failed=0
+for zip in "${FILES[@]}"; do
+  url="$BASE_URL/$zip"
+  dest="$UNTESTED_DIR/$zip"
+
+  echo "=================================================================="
+  echo "$zip"
+
+  if [ -f "$dest" ] && unzip -t "$dest" >/dev/null 2>&1; then
+    echo "  [skip] already downloaded and valid: $zip"
+  else
+    echo "  [dl] $url"
+    if ! curl --fail --location --retry 3 --retry-delay 5 -C - \
+             --connect-timeout 30 --progress-bar -o "$dest" "$url"; then
+      echo "  [FAIL] download failed for $url (partial file kept for resume: $dest)" >&2
+      rm -f "$dest"  # 404/error body would corrupt the name for future runs
+      failed=1
+      continue
+    fi
+  fi
+
+  echo "  [extract] -> $UNTESTED_DIR"
+  if ! unzip -o -q "$dest" -d "$UNTESTED_DIR"; then
+    echo "  [FAIL] extraction failed for $zip" >&2
+    failed=1
+  fi
+done
+
+echo "=================================================================="
+if [ "$failed" -ne 0 ]; then
+  echo "Finished with ERRORS — re-run to resume failed downloads." >&2
+  exit 1
+fi
+echo "All done. Contents:"
+ls -lh "$UNTESTED_DIR"
@@ -0,0 +1,2 @@
+docker compose logs -f languagetool
+
@@ -0,0 +1,102 @@
+% LanguageTool level test — ext-ce-v6.3.1 + erikvl87/languagetool
+%
+% Section A: expected to be flagged ONLY in Picky mode (level=picky).
+% Section B: expected to be flagged in BOTH modes (default and picky).
+%
+% All sentences below were verified live against the running server
+% (2026-10-26), with the exact disabledRules the Overleaf proxy sends.
+%
+% Usage: open in Overleaf, wait ~2 s after the last edit,
+% hover a wavy underline for the message + one-click replacements.
+% Toggle Settings → Spelling and Language → LanguageTool → "Picky mode"
+% off/on and compare: section A marks should appear/disappear.
+
+\documentclass{article}
+\begin{document}
+
+\section*{A: Picky-only (missed at default level)}
+
+% PASSIVE_VOICE_SIMPLE — passive voice
+The experiment was conducted by our group.
+
+% PASSIVE_VOICE_SIMPLE — passive voice
+The error was detected by the test suite.
+
+\section*{B: Both modes (default and picky)}
+
+% EN_A_VS_AN — wrong article
+An sequence off independent variables.
+
+% MORFOLOGIK_RULE_EN_US — spelling mistake
+You can recieve a bonus for this.
+
+% MORFOLOGIK_RULE_EN_US — spelling mistake
+This is a realy interesting result.
+
+% MORFOLOGIK_RULE_EN_US — spelling mistake
+He shoulda told us about the crash.
+
+% AGREEMENT_SENT_START — agreement error
+The results shows a significant difference.
+
+% BEEN_PART_AGREEMENT — possible agreement error
+A new paper was publish last week.
+
+% THERE_S_MANY — possible agreement error
+There is three reasons for this choice.
+
+% IT_IS — possible typo (missing apostrophe)
+Its time to finish the manuscript.
+
+% AFFECT_EFFECT — commonly confused word
+This should have an affect on the outcome.
+
+% LOOSE_LOSE — possible typo
+You will loose the key if you leave it here.
+
+% MODAL_OF — possible typo
+You should of checked the configuration.
+
+% MODAL_OF — possible typo
+It might of happened last night.
+
+% MODAL_OF — possible typo
+We could of fixed the bug yesterday.
+
+% INFORMATIONS — nonstandard word
+We need more informations about this topic.
+
+% EN_A_VS_AN — wrong article (uncountable)
+He gave me a advice yesterday morning.
+
+% ENGLISH_WORD_REPEAT_RULE — word repetition
+The the method was tested twice.
+
+% ADMIT_ENJOY_VB — use 'ing'
+I look forward to hear from you again.
+
+% TOO_TO — possible typo
+I want to too participate in this.
+
+% AGREEMENT_SENT_START — agreement error
+Each of the students have passed the exam.
+
+% AGREEMENT_SENT_START — agreement error
+The dog which barks is friendly.
+
+% RB_RB_COMMA — intensifier comma
+We have a very very good result here.
+
+% FREE_GIFT — redundant phrase
+This is a free gift for all of you.
+
+% ADMIT_ENJOY_VB — use 'ing'
+The manual recommends to restart the server.
+
+% ALLOW_TO — use a gerund
+The tool allows to configure the output.
+
+% ALLOW_TO — use a gerund
+The server requires to restart after the change.
+
+\end{document}
@@ -0,0 +1,2 @@
+docker compose up -d
+
@@ -0,0 +1,2 @@
+docker compose up -d languagetool
+