mirror of
https://github.com/yu-i-i/overleaf-cep.git
synced 2026-11-12 21:23:45 +01:00
Restoring the support images: \n png2pdf: \n Without this cache you need to convert png to pdf manualy or you risk a compile timeout (had this problem yesterday and did the conversion manualy. Compile time reduction was from time out to 3sec.)\n env settings required:\n (mount point INSIDE the container)\n CLSI_CACHE_PATH: /var/lib/overleaf/data/cache \n (Mount point outside of the container i.e. host, based on your compose volume rules)\n SANDBOXED_COMPILES_HOST_DIR_CACHE: points to the same folder outside of the container\n Test parameter:\n PNG2PDF_MIN_FILE_SIZE_BYTES: 1024 \n ENABLE_PNG2PDF_CONVERSIONS: true \n PNG2PDF_IMAGE: sharelatex/png2pdf:2026-06-24 \n\n It acts on slow png conversions and not all. Hence the texlive image needs this detection hook as the creator intended it. I added them to:\n https://github.com/davrot/texlive-full/tree/png2pdf-slow \n It is a modification of the LatexMK file. Without the hook no png files are cached as pdf.\n\n pdftocairo: \n The creater intended that pdf to thumbnail / preview are created by an external container for safety reasons. \n The only action we do a to-thumbnail-conversion is in the template generation. Since we don't want to disapoint the creater, I restored this safety feature.\n env settings required:\n (default: local imagemagick; other options are local pdftocairo and container pdftocairo via pdftocairo-sandbox)\n CONVERTER: pdftocairo-sandbox \n CONVERSION_SANDBOX_IMAGE: sharelatex/pdftocairo:24.02 \n (mount point INSIDE the container)\n CONVERSION_SANDBOX_DIR: /var/lib/overleaf/tmp/conversions \n (Mount point outside of the container i.e. host, based on your compose volume rules)\n CONVERSION_SANDBOX_HOST_DIR: points to the same folder outside of the container\n
This commit is contained in:
David Rotermund
@@ -0,0 +1,24 @@
|
||||
# replacement for quay.io/sharelatex/pdftocairo:24.02 (clsitex PDF→JPEG)
|
||||
#
|
||||
# ALIASES: pdftocairo-ol:25.12.0
|
||||
#
|
||||
# alpine:3.24 with poppler (25.12.0, >= 24.02 required by CLSI).
|
||||
#
|
||||
# Why: CLSI convertPDFToJPEG runs `pdftocairo -jpeg ...` against this image
|
||||
# for preview (794px, q90) and thumbnail (190px, q50) JPEGs of compiled PDFs.
|
||||
#
|
||||
# The SaaS image is Ubuntu-based; alpine 3.24's poppler 25.12.0 covers exactly
|
||||
# the flags CLSI uses (-jpeg -jpegopt -singlefile -scale-to-x -scale-to-y), and
|
||||
# the image is tiny.
|
||||
#
|
||||
# images/pandoc-amd64/Dockerfile is the sibling (pandoc/core + zip). Same pattern:
|
||||
# ENTRYPOINT [] because CLSI docker-runs with an explicit command (no
|
||||
# --entrypoint override).
|
||||
|
||||
FROM alpine:3.24
|
||||
|
||||
ENTRYPOINT []
|
||||
|
||||
RUN apk update \
|
||||
&& apk add --no-cache poppler-utils \
|
||||
&& rm -rf /var/cache/apk/*
|
||||
@@ -0,0 +1,51 @@
|
||||
# replacement for quay.io/sharelatex/png2pdf:2026-06-24 (clsitex PNG→PDF)
|
||||
#
|
||||
# Engine: img2pdf (josch) — lossless. For PNG input the IDAT streams are
|
||||
# embedded verbatim into the PDF (no re-encode); the alpha channel becomes
|
||||
# a PDF soft mask (/SMask), also lossless. That is the mechanism of the
|
||||
# whole feature: pdfTeX copies PDF image streams cheaply, while PNGs with
|
||||
# alpha/gamma/palette/interlace are re-decoded on every compile, which is
|
||||
# what made them "slow" and what fills CLSI's slow-PNG list.
|
||||
#
|
||||
# The SaaS tool is proprietary. The Go port (go/services/clsitex/png2pdf)
|
||||
# pins the exact container contract this image fulfils:
|
||||
# ENTRYPOINT + ["--in-place", "--", "<relpath>", ...] CWD=/compile
|
||||
# stdout: one "Converted <f> to PDF" line per file (counted by Go)
|
||||
# exit non-zero on first failure (CLSI falls back to the original PNGs)
|
||||
#
|
||||
# Why git@commit instead of `pip install img2pdf==0.6.3`:
|
||||
# 0.6.3 (2025-11-06) is the last PyPI release, and it PRE-DATES the
|
||||
# parse_png quadratic-rewrite fix (2025-11-16, "my 311MB ARCHD at 600dpi
|
||||
# with 17638 IDAT records went from many minutes to 320ms"). The slow-PNG
|
||||
# list is populated by BIG PNGs (min 1MB, no upper bound), so we pin main
|
||||
# at this commit: it is still reproducible (exact hash) and carries the
|
||||
# fix plus the JPEG2000/ICC and pikepdf-writer-close follow-ups.
|
||||
#
|
||||
# pikepdf + pillow have musllinux cp314 wheels for this python version, so
|
||||
# no compiler is needed; git is needed only for the pinned checkout.
|
||||
|
||||
FROM python:3.14-alpine
|
||||
|
||||
ARG IMG2PDF_REF=0fafa2c312be50c68a07e2331a89bf4adbd274ed
|
||||
|
||||
RUN apk update \
|
||||
&& apk add --no-cache git \
|
||||
&& pip install --no-cache-dir --disable-pip-version-check \
|
||||
"git+https://gitlab.mister-muffin.de/josch/img2pdf.git@${IMG2PDF_REF}#egg=img2pdf" \
|
||||
pikepdf==10.15.0 \
|
||||
pillow==12.3.0 \
|
||||
packaging \
|
||||
&& rm -rf /root/.cache/pip /usr/src/python/lib*
|
||||
|
||||
# shim: arg parsing, leading-dash path handling (paths derive from URL
|
||||
# paths and start with '-'), temp-file + atomic replace (input must survive
|
||||
# a failed conversion — CLSI renames the untouched original as fallback),
|
||||
# and the "Converted <f> to PDF" stdout line Go counts.
|
||||
COPY server-ce/png2pdf/png2pdf.sh /usr/local/bin/png2pdf
|
||||
RUN chmod 0755 /usr/local/bin/png2pdf
|
||||
|
||||
# The Go runner passes the arg list ["--in-place", "--", ...] as the container
|
||||
# command, never a binary name — see the pdftocairo image (ENTRYPOINT []) for
|
||||
# the contrast: that image provides the binary NAME in its command, this one
|
||||
# relies on the image's own ENTRYPOINT.
|
||||
ENTRYPOINT ["/usr/local/bin/png2pdf"]
|
||||
+36
-4
@@ -14,23 +14,30 @@ export OVERLEAF_TAG ?= sharelatex/sharelatex:$(BRANCH_NAME_TAG_SAFE)-$(MONOREPO_
|
||||
export OVERLEAF_PANDOC_BRANCH ?= sharelatex/sharelatex-pandoc:$(BRANCH_NAME_TAG_SAFE)
|
||||
export OVERLEAF_PANDOC_LATEST ?= sharelatex/sharelatex-pandoc
|
||||
export OVERLEAF_PANDOC_TAG ?= sharelatex/sharelatex-pandoc:$(BRANCH_NAME_TAG_SAFE)-$(MONOREPO_REVISION)
|
||||
export OVERLEAF_PDFTOCAIRO ?= sharelatex/pdftocairo:24.02
|
||||
export OVERLEAF_PNG2PDF ?= sharelatex/png2pdf:2026-06-24
|
||||
|
||||
all: build-base build-community build-pandoc
|
||||
|
||||
all: build-base build-community build-pandoc build-pdftocairo build-png2pdf
|
||||
|
||||
clean:
|
||||
-docker rmi --force $(OVERLEAF_BASE_TAG) $(OVERLEAF_TAG) $(OVERLEAF_PANDOC_TAG)
|
||||
-docker rmi --force $(OVERLEAF_BASE_TAG) $(OVERLEAF_TAG) $(OVERLEAF_PANDOC_TAG) $(OVERLEAF_PDFTOCAIRO) $(OVERLEAF_PNG2PDF)
|
||||
|
||||
refresh-cache: refresh-cache-branch
|
||||
refresh-cache-branch:
|
||||
docker inspect $(OVERLEAF_BASE_BRANCH) > /dev/null && docker pull $(OVERLEAF_BASE_BRANCH) || true
|
||||
docker inspect $(OVERLEAF_BRANCH) > /dev/null && docker pull $(OVERLEAF_BRANCH) || true
|
||||
docker inspect $(OVERLEAF_PANDOC_BRANCH) > /dev/null && docker pull $(OVERLEAF_PANDOC_BRANCH) || true
|
||||
docker inspect $(OVERLEAF_PDFTOCAIRO) > /dev/null && docker pull $(OVERLEAF_PDFTOCAIRO) || true
|
||||
docker inspect $(OVERLEAF_PNG2PDF) > /dev/null && docker pull $(OVERLEAF_PNG2PDF) || true
|
||||
|
||||
refresh-cache: refresh-cache-latest
|
||||
refresh-cache-latest:
|
||||
docker inspect $(OVERLEAF_BASE_LATEST) > /dev/null && docker pull $(OVERLEAF_BASE_LATEST) || true
|
||||
docker inspect $(OVERLEAF_LATEST) > /dev/null && docker pull $(OVERLEAF_LATEST) || true
|
||||
docker inspect $(OVERLEAF_PANDOC_LATEST) > /dev/null && docker pull $(OVERLEAF_PANDOC_LATEST) || true
|
||||
docker inspect $(OVERLEAF_PDFTOCAIRO) > /dev/null && docker pull $(OVERLEAF_PDFTOCAIRO) || true
|
||||
docker inspect $(OVERLEAF_PNG2PDF) > /dev/null && docker pull $(OVERLEAF_PNG2PDF) || true
|
||||
|
||||
build-base:
|
||||
cp .dockerignore $(MONOREPO_ROOT)
|
||||
@@ -45,7 +52,6 @@ build-base:
|
||||
--tag $(OVERLEAF_BASE_BRANCH) \
|
||||
$(MONOREPO_ROOT)
|
||||
|
||||
|
||||
build-community:
|
||||
cp .dockerignore $(MONOREPO_ROOT)
|
||||
docker build \
|
||||
@@ -74,6 +80,30 @@ build-pandoc:
|
||||
--tag $(OVERLEAF_PANDOC_BRANCH) \
|
||||
$(MONOREPO_ROOT)
|
||||
|
||||
build-pdftocairo:
|
||||
cp .dockerignore $(MONOREPO_ROOT)
|
||||
docker build \
|
||||
--build-arg BUILDKIT_INLINE_CACHE=1 \
|
||||
--progress=plain \
|
||||
--build-arg OVERLEAF_BASE_TAG \
|
||||
--label "com.overleaf.ce.revision=$(MONOREPO_REVISION)" \
|
||||
--cache-from $(OVERLEAF_PDFTOCAIRO) \
|
||||
--file Dockerfile-pdftocairo \
|
||||
--tag $(OVERLEAF_PDFTOCAIRO) \
|
||||
$(MONOREPO_ROOT)
|
||||
|
||||
build-png2pdf:
|
||||
cp .dockerignore $(MONOREPO_ROOT)
|
||||
docker build \
|
||||
--build-arg BUILDKIT_INLINE_CACHE=1 \
|
||||
--progress=plain \
|
||||
--build-arg OVERLEAF_BASE_TAG \
|
||||
--label "com.overleaf.ce.revision=$(MONOREPO_REVISION)" \
|
||||
--cache-from $(OVERLEAF_PNG2PDF) \
|
||||
--file Dockerfile-png2pdf \
|
||||
--tag $(OVERLEAF_PNG2PDF) \
|
||||
$(MONOREPO_ROOT)
|
||||
|
||||
push:
|
||||
docker push $(OVERLEAF_BASE_TAG)
|
||||
docker push $(OVERLEAF_BASE_BRANCH)
|
||||
@@ -81,6 +111,8 @@ push:
|
||||
docker push $(OVERLEAF_BRANCH)
|
||||
docker push $(OVERLEAF_PANDOC_TAG)
|
||||
docker push $(OVERLEAF_PANDOC_BRANCH)
|
||||
docker push $(OVERLEAF_PDFTOCAIRO)
|
||||
docker push $(OVERLEAF_PNG2PDF)
|
||||
|
||||
SHELLCHECK_OPTS = \
|
||||
--shell=bash \
|
||||
@@ -102,5 +134,5 @@ shellcheck_fix:
|
||||
done
|
||||
|
||||
.PHONY: all \
|
||||
build-base build-community build-pandoc\
|
||||
build-base build-community build-pandoc build-pdftocairo build-png2pdf\
|
||||
shellcheck shellcheck_fix
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
#!/bin/sh
|
||||
# CLSI "png2pdf" container contract shim (engine: img2pdf 0.6.3, josch).
|
||||
# Replaces the proprietary quay.io/sharelatex/png2pdf:2026-06-24.
|
||||
#
|
||||
# Invoked by the CLSI docker runner (go/services/clsitex/dockerrunner.go, go/services/clsitex/png2pdf/png2pdf.go):
|
||||
# args: ["--in-place", "--", "<relpath1>", ...] CWD=/compile
|
||||
# the docker runner does NOT supply a binary name in the command, so the
|
||||
# image ENTRYPOINT is this script.
|
||||
#
|
||||
# Paths: each arg is a conversion path CLSI downloaded, RELATIVE to the
|
||||
# mounted cache dir (CWD /compile). Real CLSI derives it from the file
|
||||
# URL path (u.Path: '/'->'-') + mtime + ".opt<10-hex cacheKey>",
|
||||
# e.g. ".-uploads-filestore-a1b2c3d4-0-1-1756600000000.opta1b2c3d4a1bc"
|
||||
# (leading dash, no directory separator).
|
||||
#
|
||||
# "IN-PLACE" means "rewrite each input PATH as a PDF". CLSI then renames
|
||||
# each path to its own ".opt" cache entry and copies it into the compile
|
||||
# dir (urlcache.CommitConversion) — PER FILE and UNCONDITIONALLY (sync.go:
|
||||
# a conversion error only logs a warning). So each converted file lands
|
||||
# in the cache even in a partially-failed run; an unconverted file is
|
||||
# committed as the original-PNG fallback.
|
||||
#
|
||||
# Contract (go/services/clsitex/png2pdf/png2pdf.go):
|
||||
# - stdout: one "Converted <f> to PDF" per converted file (Go counts
|
||||
# lines matching ^Converted .+ to PDF$ -> stats.png2pdf).
|
||||
# - exit non-zero if anything fails: CLSI logs a warning and commits
|
||||
# each file's current on-disk state (converted or original).
|
||||
#
|
||||
# Why img2pdf: LOSSLESS. PNG IDAT is embedded verbatim into the PDF (no
|
||||
# re-encode); the alpha channel becomes a PDF /SMask, also lossless.
|
||||
# That is the mechanism of the feature: pdfTeX copies PDF image streams
|
||||
# cheaply, whereas PNGs with alpha/gamma/palette/interlace are re-decoded
|
||||
# from the PNG on every compile — what made them "slow". Without
|
||||
# --imgsize the image keeps its pHYs DPI, falling back to 96dpi (the same
|
||||
# default pdfLaTeX itself uses).
|
||||
#
|
||||
# Failure handling: convert to a temp + atomic replace, and CONTINUE with
|
||||
# the remaining files after a failure (their valid conversions are
|
||||
# committed by CLSI even though the run is reported failed). A failed
|
||||
# file stays exactly the original PNG CLSI downloaded.
|
||||
|
||||
set -u
|
||||
|
||||
# argv is exactly [--in-place]... ["--"] <files...>. The loop terminates:
|
||||
# every iteration either shifts, breaks, or exits.
|
||||
while [ "$#" -gt 0 ]; do
|
||||
case "$1" in
|
||||
--) shift ; break ;;
|
||||
--in-place) shift ;;
|
||||
*) echo "png2pdf: unexpected argument: $1" >&2 ; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [ "$#" -lt 1 ]; then
|
||||
echo "png2pdf: no input files (usage: --in-place -- <png>...)" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
rc=0
|
||||
for file in "$@"; do
|
||||
# Resolve against CWD so a leading '-' (or '.') never lands in an argv
|
||||
# slot of img2pdf/mv, where it would be parsed as an option.
|
||||
case "$file" in
|
||||
/ | /*) abs="$file" ;;
|
||||
*) abs="$(pwd)/$file" ;;
|
||||
esac
|
||||
tmp="${abs}.tmp"
|
||||
|
||||
if ! img2pdf --output "$tmp" "$abs"; then
|
||||
echo "png2pdf: conversion failed for $file" >&2
|
||||
rm -f -- "$tmp"
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
if ! mv -- "$tmp" "$abs"; then
|
||||
echo "png2pdf: failed to replace $file" >&2
|
||||
rm -f -- "$tmp"
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
echo "Converted $file to PDF"
|
||||
done
|
||||
exit "$rc"
|
||||
@@ -1,5 +1,18 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Grant the service user access to the Docker socket group, same as the CLSI
|
||||
# run script. Needed for CONVERTER=pdftocairo-sandbox (filestore launches the
|
||||
# sibling conversion container through /var/run/docker.sock). Idempotent: a
|
||||
# fresh container may have no such group yet, and CLSI's run script does the
|
||||
# same thing, so just append to the membership if it exists.
|
||||
if [ -e '/var/run/docker.sock' ]; then
|
||||
DOCKER_GROUP=$(stat -c '%g' /var/run/docker.sock)
|
||||
if ! getent group dockeronhost >/dev/null; then
|
||||
groupadd --non-unique --gid "${DOCKER_GROUP}" dockeronhost
|
||||
fi
|
||||
usermod -aG dockeronhost www-data
|
||||
fi
|
||||
|
||||
source /etc/overleaf/env.sh
|
||||
export LISTEN_ADDRESS=127.0.0.1
|
||||
|
||||
|
||||
Reference in New Issue
Block a user