mirror of
https://github.com/yu-i-i/overleaf-cep.git
synced 2026-11-12 13:13:45 +01:00
Restoring the support images: \n png2pdf: \n Without this cache you need to convert png to pdf manualy or you risk a compile timeout (had this problem yesterday and did the conversion manualy. Compile time reduction was from time out to 3sec.)\n env settings required:\n (mount point INSIDE the container)\n CLSI_CACHE_PATH: /var/lib/overleaf/data/cache \n (Mount point outside of the container i.e. host, based on your compose volume rules)\n SANDBOXED_COMPILES_HOST_DIR_CACHE: points to the same folder outside of the container\n Test parameter:\n PNG2PDF_MIN_FILE_SIZE_BYTES: 1024 \n ENABLE_PNG2PDF_CONVERSIONS: true \n PNG2PDF_IMAGE: sharelatex/png2pdf:2026-06-24 \n\n It acts on slow png conversions and not all. Hence the texlive image needs this detection hook as the creator intended it. I added them to:\n https://github.com/davrot/texlive-full/tree/png2pdf-slow \n It is a modification of the LatexMK file. Without the hook no png files are cached as pdf.\n\n pdftocairo: \n The creater intended that pdf to thumbnail / preview are created by an external container for safety reasons. \n The only action we do a to-thumbnail-conversion is in the template generation. Since we don't want to disapoint the creater, I restored this safety feature.\n env settings required:\n (default: local imagemagick; other options are local pdftocairo and container pdftocairo via pdftocairo-sandbox)\n CONVERTER: pdftocairo-sandbox \n CONVERSION_SANDBOX_IMAGE: sharelatex/pdftocairo:24.02 \n (mount point INSIDE the container)\n CONVERSION_SANDBOX_DIR: /var/lib/overleaf/tmp/conversions \n (Mount point outside of the container i.e. host, based on your compose volume rules)\n CONVERSION_SANDBOX_HOST_DIR: points to the same folder outside of the container\n
This commit is contained in:
@@ -0,0 +1,24 @@
|
||||
# replacement for quay.io/sharelatex/pdftocairo:24.02 (clsitex PDF→JPEG)
|
||||
#
|
||||
# ALIASES: pdftocairo-ol:25.12.0
|
||||
#
|
||||
# alpine:3.24 with poppler (25.12.0, >= 24.02 required by CLSI).
|
||||
#
|
||||
# Why: CLSI convertPDFToJPEG runs `pdftocairo -jpeg ...` against this image
|
||||
# for preview (794px, q90) and thumbnail (190px, q50) JPEGs of compiled PDFs.
|
||||
#
|
||||
# The SaaS image is Ubuntu-based; alpine 3.24's poppler 25.12.0 covers exactly
|
||||
# the flags CLSI uses (-jpeg -jpegopt -singlefile -scale-to-x -scale-to-y), and
|
||||
# the image is tiny.
|
||||
#
|
||||
# images/pandoc-amd64/Dockerfile is the sibling (pandoc/core + zip). Same pattern:
|
||||
# ENTRYPOINT [] because CLSI docker-runs with an explicit command (no
|
||||
# --entrypoint override).
|
||||
|
||||
FROM alpine:3.24
|
||||
|
||||
ENTRYPOINT []
|
||||
|
||||
RUN apk update \
|
||||
&& apk add --no-cache poppler-utils \
|
||||
&& rm -rf /var/cache/apk/*
|
||||
@@ -0,0 +1,51 @@
|
||||
# replacement for quay.io/sharelatex/png2pdf:2026-06-24 (clsitex PNG→PDF)
|
||||
#
|
||||
# Engine: img2pdf (josch) — lossless. For PNG input the IDAT streams are
|
||||
# embedded verbatim into the PDF (no re-encode); the alpha channel becomes
|
||||
# a PDF soft mask (/SMask), also lossless. That is the mechanism of the
|
||||
# whole feature: pdfTeX copies PDF image streams cheaply, while PNGs with
|
||||
# alpha/gamma/palette/interlace are re-decoded on every compile, which is
|
||||
# what made them "slow" and what fills CLSI's slow-PNG list.
|
||||
#
|
||||
# The SaaS tool is proprietary. The Go port (go/services/clsitex/png2pdf)
|
||||
# pins the exact container contract this image fulfils:
|
||||
# ENTRYPOINT + ["--in-place", "--", "<relpath>", ...] CWD=/compile
|
||||
# stdout: one "Converted <f> to PDF" line per file (counted by Go)
|
||||
# exit non-zero on first failure (CLSI falls back to the original PNGs)
|
||||
#
|
||||
# Why git@commit instead of `pip install img2pdf==0.6.3`:
|
||||
# 0.6.3 (2025-11-06) is the last PyPI release, and it PRE-DATES the
|
||||
# parse_png quadratic-rewrite fix (2025-11-16, "my 311MB ARCHD at 600dpi
|
||||
# with 17638 IDAT records went from many minutes to 320ms"). The slow-PNG
|
||||
# list is populated by BIG PNGs (min 1MB, no upper bound), so we pin main
|
||||
# at this commit: it is still reproducible (exact hash) and carries the
|
||||
# fix plus the JPEG2000/ICC and pikepdf-writer-close follow-ups.
|
||||
#
|
||||
# pikepdf + pillow have musllinux cp314 wheels for this python version, so
|
||||
# no compiler is needed; git is needed only for the pinned checkout.
|
||||
|
||||
FROM python:3.14-alpine
|
||||
|
||||
ARG IMG2PDF_REF=0fafa2c312be50c68a07e2331a89bf4adbd274ed
|
||||
|
||||
RUN apk update \
|
||||
&& apk add --no-cache git \
|
||||
&& pip install --no-cache-dir --disable-pip-version-check \
|
||||
"git+https://gitlab.mister-muffin.de/josch/img2pdf.git@${IMG2PDF_REF}#egg=img2pdf" \
|
||||
pikepdf==10.15.0 \
|
||||
pillow==12.3.0 \
|
||||
packaging \
|
||||
&& rm -rf /root/.cache/pip /usr/src/python/lib*
|
||||
|
||||
# shim: arg parsing, leading-dash path handling (paths derive from URL
|
||||
# paths and start with '-'), temp-file + atomic replace (input must survive
|
||||
# a failed conversion — CLSI renames the untouched original as fallback),
|
||||
# and the "Converted <f> to PDF" stdout line Go counts.
|
||||
COPY server-ce/png2pdf/png2pdf.sh /usr/local/bin/png2pdf
|
||||
RUN chmod 0755 /usr/local/bin/png2pdf
|
||||
|
||||
# The Go runner passes the arg list ["--in-place", "--", ...] as the container
|
||||
# command, never a binary name — see the pdftocairo image (ENTRYPOINT []) for
|
||||
# the contrast: that image provides the binary NAME in its command, this one
|
||||
# relies on the image's own ENTRYPOINT.
|
||||
ENTRYPOINT ["/usr/local/bin/png2pdf"]
|
||||
+36
-4
@@ -14,23 +14,30 @@ export OVERLEAF_TAG ?= sharelatex/sharelatex:$(BRANCH_NAME_TAG_SAFE)-$(MONOREPO_
|
||||
export OVERLEAF_PANDOC_BRANCH ?= sharelatex/sharelatex-pandoc:$(BRANCH_NAME_TAG_SAFE)
|
||||
export OVERLEAF_PANDOC_LATEST ?= sharelatex/sharelatex-pandoc
|
||||
export OVERLEAF_PANDOC_TAG ?= sharelatex/sharelatex-pandoc:$(BRANCH_NAME_TAG_SAFE)-$(MONOREPO_REVISION)
|
||||
export OVERLEAF_PDFTOCAIRO ?= sharelatex/pdftocairo:24.02
|
||||
export OVERLEAF_PNG2PDF ?= sharelatex/png2pdf:2026-06-24
|
||||
|
||||
all: build-base build-community build-pandoc
|
||||
|
||||
all: build-base build-community build-pandoc build-pdftocairo build-png2pdf
|
||||
|
||||
clean:
|
||||
-docker rmi --force $(OVERLEAF_BASE_TAG) $(OVERLEAF_TAG) $(OVERLEAF_PANDOC_TAG)
|
||||
-docker rmi --force $(OVERLEAF_BASE_TAG) $(OVERLEAF_TAG) $(OVERLEAF_PANDOC_TAG) $(OVERLEAF_PDFTOCAIRO) $(OVERLEAF_PNG2PDF)
|
||||
|
||||
refresh-cache: refresh-cache-branch
|
||||
refresh-cache-branch:
|
||||
docker inspect $(OVERLEAF_BASE_BRANCH) > /dev/null && docker pull $(OVERLEAF_BASE_BRANCH) || true
|
||||
docker inspect $(OVERLEAF_BRANCH) > /dev/null && docker pull $(OVERLEAF_BRANCH) || true
|
||||
docker inspect $(OVERLEAF_PANDOC_BRANCH) > /dev/null && docker pull $(OVERLEAF_PANDOC_BRANCH) || true
|
||||
docker inspect $(OVERLEAF_PDFTOCAIRO) > /dev/null && docker pull $(OVERLEAF_PDFTOCAIRO) || true
|
||||
docker inspect $(OVERLEAF_PNG2PDF) > /dev/null && docker pull $(OVERLEAF_PNG2PDF) || true
|
||||
|
||||
refresh-cache: refresh-cache-latest
|
||||
refresh-cache-latest:
|
||||
docker inspect $(OVERLEAF_BASE_LATEST) > /dev/null && docker pull $(OVERLEAF_BASE_LATEST) || true
|
||||
docker inspect $(OVERLEAF_LATEST) > /dev/null && docker pull $(OVERLEAF_LATEST) || true
|
||||
docker inspect $(OVERLEAF_PANDOC_LATEST) > /dev/null && docker pull $(OVERLEAF_PANDOC_LATEST) || true
|
||||
docker inspect $(OVERLEAF_PDFTOCAIRO) > /dev/null && docker pull $(OVERLEAF_PDFTOCAIRO) || true
|
||||
docker inspect $(OVERLEAF_PNG2PDF) > /dev/null && docker pull $(OVERLEAF_PNG2PDF) || true
|
||||
|
||||
build-base:
|
||||
cp .dockerignore $(MONOREPO_ROOT)
|
||||
@@ -45,7 +52,6 @@ build-base:
|
||||
--tag $(OVERLEAF_BASE_BRANCH) \
|
||||
$(MONOREPO_ROOT)
|
||||
|
||||
|
||||
build-community:
|
||||
cp .dockerignore $(MONOREPO_ROOT)
|
||||
docker build \
|
||||
@@ -74,6 +80,30 @@ build-pandoc:
|
||||
--tag $(OVERLEAF_PANDOC_BRANCH) \
|
||||
$(MONOREPO_ROOT)
|
||||
|
||||
build-pdftocairo:
|
||||
cp .dockerignore $(MONOREPO_ROOT)
|
||||
docker build \
|
||||
--build-arg BUILDKIT_INLINE_CACHE=1 \
|
||||
--progress=plain \
|
||||
--build-arg OVERLEAF_BASE_TAG \
|
||||
--label "com.overleaf.ce.revision=$(MONOREPO_REVISION)" \
|
||||
--cache-from $(OVERLEAF_PDFTOCAIRO) \
|
||||
--file Dockerfile-pdftocairo \
|
||||
--tag $(OVERLEAF_PDFTOCAIRO) \
|
||||
$(MONOREPO_ROOT)
|
||||
|
||||
build-png2pdf:
|
||||
cp .dockerignore $(MONOREPO_ROOT)
|
||||
docker build \
|
||||
--build-arg BUILDKIT_INLINE_CACHE=1 \
|
||||
--progress=plain \
|
||||
--build-arg OVERLEAF_BASE_TAG \
|
||||
--label "com.overleaf.ce.revision=$(MONOREPO_REVISION)" \
|
||||
--cache-from $(OVERLEAF_PNG2PDF) \
|
||||
--file Dockerfile-png2pdf \
|
||||
--tag $(OVERLEAF_PNG2PDF) \
|
||||
$(MONOREPO_ROOT)
|
||||
|
||||
push:
|
||||
docker push $(OVERLEAF_BASE_TAG)
|
||||
docker push $(OVERLEAF_BASE_BRANCH)
|
||||
@@ -81,6 +111,8 @@ push:
|
||||
docker push $(OVERLEAF_BRANCH)
|
||||
docker push $(OVERLEAF_PANDOC_TAG)
|
||||
docker push $(OVERLEAF_PANDOC_BRANCH)
|
||||
docker push $(OVERLEAF_PDFTOCAIRO)
|
||||
docker push $(OVERLEAF_PNG2PDF)
|
||||
|
||||
SHELLCHECK_OPTS = \
|
||||
--shell=bash \
|
||||
@@ -102,5 +134,5 @@ shellcheck_fix:
|
||||
done
|
||||
|
||||
.PHONY: all \
|
||||
build-base build-community build-pandoc\
|
||||
build-base build-community build-pandoc build-pdftocairo build-png2pdf\
|
||||
shellcheck shellcheck_fix
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
#!/bin/sh
|
||||
# CLSI "png2pdf" container contract shim (engine: img2pdf 0.6.3, josch).
|
||||
# Replaces the proprietary quay.io/sharelatex/png2pdf:2026-06-24.
|
||||
#
|
||||
# Invoked by the CLSI docker runner (go/services/clsitex/dockerrunner.go, go/services/clsitex/png2pdf/png2pdf.go):
|
||||
# args: ["--in-place", "--", "<relpath1>", ...] CWD=/compile
|
||||
# the docker runner does NOT supply a binary name in the command, so the
|
||||
# image ENTRYPOINT is this script.
|
||||
#
|
||||
# Paths: each arg is a conversion path CLSI downloaded, RELATIVE to the
|
||||
# mounted cache dir (CWD /compile). Real CLSI derives it from the file
|
||||
# URL path (u.Path: '/'->'-') + mtime + ".opt<10-hex cacheKey>",
|
||||
# e.g. ".-uploads-filestore-a1b2c3d4-0-1-1756600000000.opta1b2c3d4a1bc"
|
||||
# (leading dash, no directory separator).
|
||||
#
|
||||
# "IN-PLACE" means "rewrite each input PATH as a PDF". CLSI then renames
|
||||
# each path to its own ".opt" cache entry and copies it into the compile
|
||||
# dir (urlcache.CommitConversion) — PER FILE and UNCONDITIONALLY (sync.go:
|
||||
# a conversion error only logs a warning). So each converted file lands
|
||||
# in the cache even in a partially-failed run; an unconverted file is
|
||||
# committed as the original-PNG fallback.
|
||||
#
|
||||
# Contract (go/services/clsitex/png2pdf/png2pdf.go):
|
||||
# - stdout: one "Converted <f> to PDF" per converted file (Go counts
|
||||
# lines matching ^Converted .+ to PDF$ -> stats.png2pdf).
|
||||
# - exit non-zero if anything fails: CLSI logs a warning and commits
|
||||
# each file's current on-disk state (converted or original).
|
||||
#
|
||||
# Why img2pdf: LOSSLESS. PNG IDAT is embedded verbatim into the PDF (no
|
||||
# re-encode); the alpha channel becomes a PDF /SMask, also lossless.
|
||||
# That is the mechanism of the feature: pdfTeX copies PDF image streams
|
||||
# cheaply, whereas PNGs with alpha/gamma/palette/interlace are re-decoded
|
||||
# from the PNG on every compile — what made them "slow". Without
|
||||
# --imgsize the image keeps its pHYs DPI, falling back to 96dpi (the same
|
||||
# default pdfLaTeX itself uses).
|
||||
#
|
||||
# Failure handling: convert to a temp + atomic replace, and CONTINUE with
|
||||
# the remaining files after a failure (their valid conversions are
|
||||
# committed by CLSI even though the run is reported failed). A failed
|
||||
# file stays exactly the original PNG CLSI downloaded.
|
||||
|
||||
set -u
|
||||
|
||||
# argv is exactly [--in-place]... ["--"] <files...>. The loop terminates:
|
||||
# every iteration either shifts, breaks, or exits.
|
||||
while [ "$#" -gt 0 ]; do
|
||||
case "$1" in
|
||||
--) shift ; break ;;
|
||||
--in-place) shift ;;
|
||||
*) echo "png2pdf: unexpected argument: $1" >&2 ; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [ "$#" -lt 1 ]; then
|
||||
echo "png2pdf: no input files (usage: --in-place -- <png>...)" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
rc=0
|
||||
for file in "$@"; do
|
||||
# Resolve against CWD so a leading '-' (or '.') never lands in an argv
|
||||
# slot of img2pdf/mv, where it would be parsed as an option.
|
||||
case "$file" in
|
||||
/ | /*) abs="$file" ;;
|
||||
*) abs="$(pwd)/$file" ;;
|
||||
esac
|
||||
tmp="${abs}.tmp"
|
||||
|
||||
if ! img2pdf --output "$tmp" "$abs"; then
|
||||
echo "png2pdf: conversion failed for $file" >&2
|
||||
rm -f -- "$tmp"
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
if ! mv -- "$tmp" "$abs"; then
|
||||
echo "png2pdf: failed to replace $file" >&2
|
||||
rm -f -- "$tmp"
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
echo "Converted $file to PDF"
|
||||
done
|
||||
exit "$rc"
|
||||
@@ -1,5 +1,18 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Grant the service user access to the Docker socket group, same as the CLSI
|
||||
# run script. Needed for CONVERTER=pdftocairo-sandbox (filestore launches the
|
||||
# sibling conversion container through /var/run/docker.sock). Idempotent: a
|
||||
# fresh container may have no such group yet, and CLSI's run script does the
|
||||
# same thing, so just append to the membership if it exists.
|
||||
if [ -e '/var/run/docker.sock' ]; then
|
||||
DOCKER_GROUP=$(stat -c '%g' /var/run/docker.sock)
|
||||
if ! getent group dockeronhost >/dev/null; then
|
||||
groupadd --non-unique --gid "${DOCKER_GROUP}" dockeronhost
|
||||
fi
|
||||
usermod -aG dockeronhost www-data
|
||||
fi
|
||||
|
||||
source /etc/overleaf/env.sh
|
||||
export LISTEN_ADDRESS=127.0.0.1
|
||||
|
||||
|
||||
@@ -48,7 +48,20 @@ const DockerRunner = {
|
||||
image = `${Settings.texliveImageNameOveride}/${img}`
|
||||
}
|
||||
|
||||
if (compileGroup === 'synctex-output') {
|
||||
if (
|
||||
Settings.path.sandboxedCompilesHostDirCache &&
|
||||
directory.startsWith(Settings.path.clsiCacheDir + Path.sep)
|
||||
) {
|
||||
// In: directory = '<clsiCacheDir>/projectId'
|
||||
// sandboxedCompilesHostDirCache = '/host/cache'
|
||||
// Out: directory = '/host/cache/projectId'
|
||||
// (png2pdf & friends operate on CLSI's per-project cache dir, which
|
||||
// is not under the compiles dir)
|
||||
directory = Path.join(
|
||||
Settings.path.sandboxedCompilesHostDirCache,
|
||||
Path.basename(directory)
|
||||
)
|
||||
} else if (compileGroup === 'synctex-output') {
|
||||
// In: directory = '/overleaf/services/clsi/output/projectId-userId/generated-files/buildId'
|
||||
// directory.split('/').slice(-3) === 'projectId-userId/generated-files/buildId'
|
||||
// sandboxedCompilesHostDirOutput = '/host/output'
|
||||
@@ -93,32 +106,35 @@ const DockerRunner = {
|
||||
// logOptions = _.clone(options)
|
||||
// logOptions?.HostConfig?.SecurityOpt = "secomp used, removed in logging"
|
||||
logger.debug({ projectId }, 'running docker container')
|
||||
DockerRunner._runAndWaitForContainer(
|
||||
options,
|
||||
volumes,
|
||||
timeout,
|
||||
(error, output) => {
|
||||
if (error && error.statusCode === 500) {
|
||||
logger.debug(
|
||||
{ err: error, projectId },
|
||||
'error running container so destroying and retrying'
|
||||
)
|
||||
DockerRunner.destroyContainer(name, null, true, error => {
|
||||
if (error != null) {
|
||||
return callback(error)
|
||||
}
|
||||
DockerRunner._runAndWaitForContainer(
|
||||
options,
|
||||
volumes,
|
||||
timeout,
|
||||
callback
|
||||
// Bounded retry: a permanent container create/start failure (e.g.
|
||||
// "unable to find user" on a FROM-scratch image) must not busy-loop
|
||||
// forever — surface the error to the caller after a couple of attempts.
|
||||
let retriesLeft = 2
|
||||
const runAttempt = () => {
|
||||
DockerRunner._runAndWaitForContainer(
|
||||
options,
|
||||
volumes,
|
||||
timeout,
|
||||
(error, output) => {
|
||||
if (error && error.statusCode === 500 && retriesLeft > 0) {
|
||||
retriesLeft -= 1
|
||||
logger.debug(
|
||||
{ err: error, projectId, retriesLeft },
|
||||
'error running container so destroying and retrying'
|
||||
)
|
||||
})
|
||||
} else {
|
||||
callback(error, output)
|
||||
DockerRunner.destroyContainer(name, null, true, err => {
|
||||
if (err != null) {
|
||||
return callback(err)
|
||||
}
|
||||
runAttempt()
|
||||
})
|
||||
} else {
|
||||
callback(error, output)
|
||||
}
|
||||
}
|
||||
}
|
||||
)
|
||||
)
|
||||
}
|
||||
runAttempt()
|
||||
|
||||
// pass back the container name to allow it to be killed
|
||||
return name
|
||||
|
||||
@@ -172,7 +172,15 @@ if ((process.env.DOCKER_RUNNER || process.env.SANDBOXED_COMPILES) === 'true') {
|
||||
wordcount: { 'HostConfig.AutoRemove': true },
|
||||
synctex: { 'HostConfig.AutoRemove': true },
|
||||
'synctex-output': { 'HostConfig.AutoRemove': true },
|
||||
conversions: { 'HostConfig.AutoRemove': true },
|
||||
// The pdftocairo image is also built FROM scratch with no passwd file,
|
||||
// so the default named user (www-data) fails at container create with
|
||||
// "unable to find user ... no matching entries in passwd file". Run it
|
||||
// as the numeric uid/gid of this clsi process (which owns the files it
|
||||
// converts), same as the png2pdf image below.
|
||||
conversions: {
|
||||
'HostConfig.AutoRemove': true,
|
||||
User: `${process.getuid()}:${process.getgid()}`,
|
||||
},
|
||||
// The png2pdf image is built FROM scratch and has no 'tex' user, so run
|
||||
// it as the same uid/gid as this clsi process, which owns the cache files
|
||||
// it converts in place.
|
||||
|
||||
@@ -2,6 +2,7 @@ import metrics from '@overleaf/metrics'
|
||||
import Settings from '@overleaf/settings'
|
||||
import { callbackify } from 'node:util'
|
||||
import SafeExec from './SafeExec.js'
|
||||
import SandboxConverterRunner from './SandboxConverterRunner.js'
|
||||
import Errors from './Errors.js'
|
||||
|
||||
const { ConversionError } = Errors
|
||||
@@ -22,6 +23,9 @@ export default {
|
||||
}
|
||||
|
||||
async function convert(sourcePath, requestedFormat) {
|
||||
if (Settings.converter === 'pdftocairo-sandbox') {
|
||||
return await _convertInSandbox(sourcePath, requestedFormat, 1500)
|
||||
}
|
||||
if (Settings.converter === 'pdftocairo') {
|
||||
const width = 1500
|
||||
return await _convert(sourcePath, requestedFormat, [
|
||||
@@ -49,6 +53,9 @@ async function convert(sourcePath, requestedFormat) {
|
||||
}
|
||||
|
||||
async function thumbnail(sourcePath) {
|
||||
if (Settings.converter === 'pdftocairo-sandbox') {
|
||||
return await _convertInSandbox(sourcePath, 'png', 700)
|
||||
}
|
||||
if (Settings.converter === 'pdftocairo') {
|
||||
const width = 700
|
||||
return await _convert(sourcePath, 'png', [
|
||||
@@ -81,6 +88,9 @@ async function thumbnail(sourcePath) {
|
||||
|
||||
async function preview(sourcePath) {
|
||||
const width = 1000
|
||||
if (Settings.converter === 'pdftocairo-sandbox') {
|
||||
return await _convertInSandbox(sourcePath, 'png', width)
|
||||
}
|
||||
if (Settings.converter === 'pdftocairo') {
|
||||
return await _convert(sourcePath, 'png', [
|
||||
'pdftocairo',
|
||||
@@ -109,6 +119,50 @@ async function preview(sourcePath) {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Run the pdftocairo conversion inside a sandboxed sibling container (no
|
||||
* network, all caps dropped, memory cap — see SandboxConverterRunner.js).
|
||||
*
|
||||
* Keeps the same contract as the local path: the result is written to
|
||||
* `${sourcePath}.${requestedFormat}`.
|
||||
*
|
||||
* @param {string} sourcePath
|
||||
* @param {string} requestedFormat 'png' or 'jpg'
|
||||
* @param {number} width output width in px
|
||||
* @return {Promise<{outputHostPath: string}>}
|
||||
*/
|
||||
async function _convertInSandbox(sourcePath, requestedFormat, width) {
|
||||
if (!APPROVED_FORMATS.includes(requestedFormat)) {
|
||||
throw new ConversionError('invalid format requested', { format: requestedFormat })
|
||||
}
|
||||
const ext = requestedFormat
|
||||
const workDir = '/work'
|
||||
// poppler appends the device extension to the output argument, so the
|
||||
// argument must end WITHOUT an extension; the result lands at
|
||||
// /work/input.pdf.<png|jpg> next to the input.
|
||||
const device = requestedFormat === 'png' ? '-png' : '-jpeg'
|
||||
const result = await SandboxConverterRunner.runConversion({
|
||||
sourcePath,
|
||||
containerArgs: [
|
||||
'pdftocairo',
|
||||
device,
|
||||
'-singlefile',
|
||||
'-scale-to-x',
|
||||
width.toString(),
|
||||
'-scale-to-y',
|
||||
'-1', // maintain aspect ratio
|
||||
`${workDir}/input.pdf`,
|
||||
`${workDir}/input.pdf`,
|
||||
],
|
||||
workDir,
|
||||
outputBase: 'input.pdf',
|
||||
outputExt: ext,
|
||||
destPath: `${sourcePath}.${requestedFormat}`,
|
||||
})
|
||||
// Same contract as the local converters: return the destination path string.
|
||||
return result.outputHostPath
|
||||
}
|
||||
|
||||
async function _convert(sourcePath, requestedFormat, command) {
|
||||
if (!APPROVED_FORMATS.includes(requestedFormat)) {
|
||||
throw new ConversionError('invalid format requested', {
|
||||
|
||||
@@ -0,0 +1,342 @@
|
||||
// SandboxConverterRunner.js
|
||||
//
|
||||
// Runs a PDF→image conversion (pdftocairo) in a throwaway sibling container,
|
||||
// the same isolation profile CLSI's DockerRunner gives its conversion
|
||||
// containers (see services/clsi/app/js/DockerRunner.mjs and the `conversions`
|
||||
// / `png2pdf` compile groups in services/clsi/config/settings.defaults.cjs):
|
||||
//
|
||||
// - no network access (HostConfig.NetworkMode: 'none')
|
||||
// - all capabilities dropped (HostConfig.CapDrop: ['ALL'])
|
||||
// - no new privileges (SecurityOpt: 'no-new-privileges')
|
||||
// - hard memory cap
|
||||
// - numeric uid:gid of the overleaf service user (the scratch-based
|
||||
// converter images have no passwd entries, so named users fail)
|
||||
// - container is always removed afterwards
|
||||
//
|
||||
// Implemented as a minimal Docker Engine API client over the unix socket so
|
||||
// that no new npm dependency is required (this service is installed with
|
||||
// Yarn PnP; cross-service packages such as dockerode are not resolvable
|
||||
// here).
|
||||
//
|
||||
// IMPORTANT (bind mounts): the daemon runs on the Docker host, so the
|
||||
// bind source MUST be a daemon-host-visible path (in this deployment:
|
||||
// under the overleaf `./data` volume), never a container-internal path
|
||||
// such as /tmp. The caller passes a `hostDir` that satisfies this.
|
||||
|
||||
import http from 'node:http'
|
||||
import Path from 'node:path'
|
||||
import Fs from 'node:fs'
|
||||
import crypto from 'node:crypto'
|
||||
import Settings from '@overleaf/settings'
|
||||
import logger from '@overleaf/logger'
|
||||
import Errors from './Errors.js'
|
||||
|
||||
const { ConversionError } = Errors
|
||||
|
||||
const API_VERSION = 'v1.43' // conservative; the engine API is backwards compatible
|
||||
|
||||
/**
|
||||
* One request against the Docker Engine API on a unix socket.
|
||||
*
|
||||
* @param {string} socketPath
|
||||
* @param {string} method
|
||||
* @param {string} apiPath e.g. `/v1.43/containers/create?name=xyz`
|
||||
* @param {object} [opts]
|
||||
* @param {object|string} [opts.body] JSON body
|
||||
* @param {number} [opts.timeoutMs] hard deadline for the whole request
|
||||
* @param {Buffer|null} [opts.expectRaw] pass a truthy value to keep the body
|
||||
* as a Buffer (used for the log stream)
|
||||
* @return {Promise<{status: number, headers: object, body: any, raw: Buffer}>}
|
||||
*/
|
||||
function request(socketPath, method, apiPath, { body, timeoutMs = 60000, raw = false } = {}) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const data = body === undefined || body === null ? null : JSON.stringify(body)
|
||||
let settled = false
|
||||
const req = http.request(
|
||||
{
|
||||
socketPath,
|
||||
method,
|
||||
path: apiPath,
|
||||
headers: data
|
||||
? {
|
||||
'content-type': 'application/json',
|
||||
'content-length': Buffer.byteLength(data),
|
||||
}
|
||||
: {},
|
||||
},
|
||||
(res) => {
|
||||
const chunks = []
|
||||
res.on('data', (c) => chunks.push(c))
|
||||
res.on('end', () => {
|
||||
if (settled) return
|
||||
settled = true
|
||||
clearTimeout(timer)
|
||||
const buf = Buffer.concat(chunks)
|
||||
if (raw) return resolve({ status: res.statusCode, headers: res.headers, body: buf, raw: buf })
|
||||
let parsed = buf
|
||||
if (buf.length && (buf[0] === 0x7b || buf[0] === 0x5b)) {
|
||||
try {
|
||||
parsed = JSON.parse(buf.toString('utf8'))
|
||||
} catch (e) {
|
||||
// non-JSON body (e.g. plain text) — keep buffer
|
||||
parsed = buf
|
||||
}
|
||||
}
|
||||
resolve({ status: res.statusCode, headers: res.headers, body: parsed, raw: buf })
|
||||
})
|
||||
}
|
||||
)
|
||||
const timer = setTimeout(() => {
|
||||
if (settled) return
|
||||
settled = true
|
||||
req.destroy()
|
||||
reject(new Error(`docker api ${method} ${apiPath} timed out after ${timeoutMs}ms`))
|
||||
}, timeoutMs)
|
||||
timer.unref?.()
|
||||
req.on('error', (err) => {
|
||||
if (settled) return
|
||||
settled = true
|
||||
clearTimeout(timer)
|
||||
reject(err)
|
||||
})
|
||||
if (data) req.write(data)
|
||||
req.end()
|
||||
})
|
||||
}
|
||||
|
||||
/** Decode the framed (TTY=false) docker log stream. */
|
||||
function decodeLogStream(buf) {
|
||||
if (!buf || !buf.length) return ''
|
||||
// 8-byte header per frame: [stream(1)] [size_be32(4)] [reserved(3)] + payload
|
||||
// (older daemons use a 4-byte header; try the 8-byte layout first)
|
||||
let out = ''
|
||||
let i = 0
|
||||
while (i + 8 <= buf.length) {
|
||||
const size = buf.readUInt32BE(i + 4)
|
||||
if (i + 8 + size > buf.length) break
|
||||
if (buf[i] === 1 || buf[i] === 2) out += buf.subarray(i + 8, i + 8 + size).toString('utf8')
|
||||
i += 8 + size
|
||||
}
|
||||
return out || buf.toString('utf8')
|
||||
}
|
||||
|
||||
/**
|
||||
* Best-effort sweep of leftover `filestore-convert-*` containers from crashes
|
||||
* or service restarts mid-run (a container that was created but never started
|
||||
* or removed otherwise would leak forever).
|
||||
*/
|
||||
async function _sweepLeftovers(socketPath) {
|
||||
let res
|
||||
try {
|
||||
res = await request(
|
||||
socketPath,
|
||||
'GET',
|
||||
`/${API_VERSION}/containers/json?all=1&limit=200&filters=%7B%22name%22%3A%5B%22filestore-convert-%22%5D%7D`,
|
||||
{ timeoutMs: 10000 }
|
||||
)
|
||||
} catch (err) {
|
||||
return // not fatal
|
||||
}
|
||||
if (res.status !== 200 || !Array.isArray(res.body)) return
|
||||
for (const c of res.body) {
|
||||
if (typeof c.Names?.[0] === 'string' && c.Names[0].startsWith('/filestore-convert-')) {
|
||||
const name = c.Names[0].slice(1)
|
||||
for (let attempt = 0; attempt < 2; attempt += 1) {
|
||||
try {
|
||||
await request(socketPath, 'DELETE', `/${API_VERSION}/containers/${name}?force=1`, { timeoutMs: 10000 })
|
||||
logger.warn({ name }, 'sandbox conversion: removed leftover container')
|
||||
break
|
||||
} catch (err) {
|
||||
if (attempt === 0) continue
|
||||
logger.warn({ err, name }, 'sandbox conversion: failed to remove leftover container')
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Run a conversion in a sandboxed container.
|
||||
*
|
||||
* Layout contract (mirrors CLSI's conversion runner):
|
||||
* host: <hostDir>/<runId>/input.pdf <- the source file (copied in)
|
||||
* host: <hostDir>/<runId>/<outputBase>.<ext> <- produced by the container
|
||||
*
|
||||
* @param {object} opts
|
||||
* @param {string} opts.sourcePath absolute path to the source PDF (must be
|
||||
* inside a daemon-host-visible directory — the bind source is derived from it)
|
||||
* @param {string} opts.containerArgs full argv inside the container, e.g.
|
||||
* ['pdftocairo','-png','-singlefile','-scale-to-x','700','-scale-to-y','-1',
|
||||
* '/work/input.pdf','/work/output']
|
||||
* @param {string} opts.workDir basename mapping used in `containerArgs`; the
|
||||
* source file is placed at <workDir>/input.pdf inside the container
|
||||
* @param {string} opts.outputBase container-side output basename (without ext)
|
||||
* @param {string} opts.outputExt 'png' or 'jpg'
|
||||
* @param {string} opts.destPath absolute host path where the result must end up
|
||||
* @return {Promise<{container: string, exitCode: number, outputHostPath: string, durationMs: number}>}
|
||||
*/
|
||||
export async function runConversion(opts) {
|
||||
const {
|
||||
sourcePath,
|
||||
containerArgs,
|
||||
workDir = '/work',
|
||||
outputBase = 'output',
|
||||
outputExt = 'png',
|
||||
destPath,
|
||||
} = opts
|
||||
|
||||
const sandbox = Settings.sandboxConversion || {}
|
||||
const image = sandbox.image || 'sharelatex/pdftocairo:24.02'
|
||||
const socketPath = sandbox.dockerSocketPath || '/var/run/docker.sock'
|
||||
const hostDir = sandbox.hostDir // daemon-host side of the scratch dir
|
||||
const containerDir = sandbox.containerDir // overleaf-container side
|
||||
const memoryBytes = sandbox.memoryBytes || 512 * 1024 * 1024
|
||||
const timeoutMs = sandbox.timeoutMs || 40000
|
||||
|
||||
if (!hostDir || !containerDir) {
|
||||
throw new ConversionError(
|
||||
'sandbox conversion is not configured: set CONVERSION_SANDBOX_HOST_DIR (Docker-daemon side) and CONVERSION_SANDBOX_DIR (overleaf-container side) to the two views of the same directory'
|
||||
)
|
||||
}
|
||||
|
||||
const runId = crypto.randomBytes(8).toString('hex')
|
||||
const runName = `filestore-convert-${runId}`
|
||||
const runDir = Path.join(containerDir, runId) // path used by THIS process (container side)
|
||||
const hostRunDir = Path.join(hostDir, runId) // same dir from the daemon side (bind source)
|
||||
const inputHostPath = Path.join(runDir, 'input.pdf')
|
||||
const outputHostPath = Path.join(runDir, `${outputBase}.${outputExt}`)
|
||||
|
||||
const t0 = Date.now()
|
||||
let containerCreated = false
|
||||
try {
|
||||
// Guard against leftover containers from an interrupted previous run.
|
||||
await _sweepLeftovers(socketPath)
|
||||
|
||||
Fs.mkdirSync(runDir, { mode: 0o770 })
|
||||
Fs.copyFileSync(sourcePath, inputHostPath)
|
||||
|
||||
// Container-absolute path of the source file, for argv remapping sanity
|
||||
const sourceInContainer = Path.join(workDir, 'input.pdf')
|
||||
if (!containerArgs.some((a) => a === sourceInContainer)) {
|
||||
throw new ConversionError(
|
||||
`sandbox argv does not reference ${sourceInContainer} — refusing to run`
|
||||
)
|
||||
}
|
||||
|
||||
// Create (the converter images are scratch-based: no ENTRYPOINT, so a bare
|
||||
// Cmd list is what CLSI runs successfully too).
|
||||
const createBody = {
|
||||
Image: image,
|
||||
Cmd: containerArgs,
|
||||
WorkingDir: workDir,
|
||||
// scratch-based images have no passwd entries — run as the numeric
|
||||
// uid:gid of this service (it owns the files on both sides of the bind).
|
||||
User: `${process.getuid()}:${process.getgid()}`,
|
||||
HostConfig: {
|
||||
Binds: [`${hostRunDir}:${workDir}:rw`],
|
||||
CapDrop: ['ALL'],
|
||||
SecurityOpt: ['no-new-privileges'],
|
||||
NetworkMode: 'none',
|
||||
Memory: memoryBytes,
|
||||
MemorySwap: 0,
|
||||
PidsLimit: 64,
|
||||
},
|
||||
}
|
||||
const created = await request(socketPath, 'POST', `/${API_VERSION}/containers/create?name=${runName}`, {
|
||||
body: createBody,
|
||||
timeoutMs: 60000,
|
||||
})
|
||||
if (created.status >= 300 || !created.body?.Id) {
|
||||
const msg = created.body?.message || JSON.stringify(created.body) || `status ${created.status}`
|
||||
throw new ConversionError(`failed to create sandbox container: ${msg}`)
|
||||
}
|
||||
containerCreated = true
|
||||
logger.info({ image, runName, hostDir: hostRunDir, containerDir: runDir }, 'sandbox conversion: container created')
|
||||
|
||||
await request(socketPath, 'POST', `/${API_VERSION}/containers/${runName}/start`, { timeoutMs: 60000 })
|
||||
|
||||
// Wait with a real deadline; kill on breach (the /wait call is a blocking
|
||||
// HTTP call that carries no data, so it gets its own generous budget).
|
||||
let exitCode = -1
|
||||
let killed = false
|
||||
try {
|
||||
// NOTE: this daemon (Docker 29 / API ≥1.44) requires POST for /wait; GET
|
||||
// returns 404. (dockerode does the same, which is how CLSI waits.)
|
||||
const waited = await request(socketPath, 'POST', `/${API_VERSION}/containers/${runName}/wait`, {
|
||||
timeoutMs: timeoutMs + 15000,
|
||||
})
|
||||
exitCode = waited.body?.StatusCode ?? -1
|
||||
if (exitCode === -1 && waited.status !== 200) {
|
||||
throw new Error(`unexpected /wait status ${waited.status}: ${waited.raw?.toString?.('utf8') || 'no body'}`)
|
||||
}
|
||||
} catch (waitErr) {
|
||||
killed = true
|
||||
await _killAndRemove(socketPath, runName)
|
||||
throw new ConversionError(`sandbox conversion timed out (${timeoutMs}ms): ${waitErr.message}`)
|
||||
}
|
||||
|
||||
let logs = ''
|
||||
try {
|
||||
const logsRes = await request(
|
||||
socketPath,
|
||||
'GET',
|
||||
`/${API_VERSION}/containers/${runName}/logs?stdout=1&stderr=1&stdin=0&follow=1`,
|
||||
{ raw: true, timeoutMs: 10000 }
|
||||
)
|
||||
logs = decodeLogStream(logsRes.raw)
|
||||
} catch (err) {
|
||||
logger.warn({ err, runName }, 'sandbox conversion: could not read container logs')
|
||||
}
|
||||
|
||||
if (exitCode !== 0) {
|
||||
throw new ConversionError(
|
||||
`sandbox conversion failed (exit ${exitCode}${killed ? ', killed' : ''})`,
|
||||
{ stderr: logs.trim().slice(-4000), exitCode }
|
||||
)
|
||||
}
|
||||
|
||||
let stat
|
||||
try {
|
||||
stat = Fs.statSync(outputHostPath)
|
||||
} catch (err) {
|
||||
throw new ConversionError(`sandbox conversion produced no output file`, { stderr: logs.trim().slice(-4000) })
|
||||
}
|
||||
if (!stat.isFile()) {
|
||||
throw new ConversionError('sandbox conversion output is not a regular file', {
|
||||
stderr: logs.trim().slice(-4000),
|
||||
})
|
||||
}
|
||||
|
||||
Fs.copyFileSync(outputHostPath, destPath)
|
||||
logger.info(
|
||||
{ runName, sizeBytes: stat.size, durationMs: Date.now() - t0 },
|
||||
'sandbox conversion completed'
|
||||
)
|
||||
return { container: runName, exitCode, outputHostPath: destPath, durationMs: Date.now() - t0 }
|
||||
} finally {
|
||||
// Best-effort cleanup: container + scratch dir (the dir holds the original
|
||||
// PDF — do not leave it behind).
|
||||
if (containerCreated) {
|
||||
await _killAndRemove(socketPath, runName).catch((e) => {
|
||||
logger.warn({ err: e, runName }, 'sandbox conversion: cleanup failed')
|
||||
})
|
||||
}
|
||||
Fs.rmSync(runDir, { recursive: true, force: true })
|
||||
}
|
||||
}
|
||||
|
||||
async function _killAndRemove(socketPath, name) {
|
||||
try {
|
||||
// POST /kill (GET is not served by this daemon); 409 = not running → fine
|
||||
await request(socketPath, 'POST', `/${API_VERSION}/containers/${name}/kill`, { timeoutMs: 10000 })
|
||||
} catch (err) {
|
||||
// not running → fine
|
||||
}
|
||||
try {
|
||||
await request(socketPath, 'DELETE', `/${API_VERSION}/containers/${name}?force=1`, { timeoutMs: 10000 })
|
||||
} catch (err) {
|
||||
// already gone → fine
|
||||
}
|
||||
}
|
||||
|
||||
export default { runConversion }
|
||||
@@ -99,8 +99,29 @@ const settings = {
|
||||
|
||||
enableConversions: process.env.ENABLE_CONVERSIONS === 'true',
|
||||
|
||||
// PDF→image conversion backend (template gallery previews/thumbnails):
|
||||
// 'imagemagick' – local, via ImageMagick `convert` (default; binary required)
|
||||
// 'pdftocairo' – local, via the `pdftocairo` binary (poppler)
|
||||
// 'pdftocairo-sandbox' – runs `pdftocairo` in a throwaway sibling container:
|
||||
// no network, CapDrop ALL, no-new-privileges, memory
|
||||
// cap (see app/js/SandboxConverterRunner.js).
|
||||
// requires a Docker socket + a daemon-host-visible
|
||||
// scratch dir (CONVERSION_SANDBOX_HOST_DIR).
|
||||
converter: process.env.CONVERTER || 'imagemagick',
|
||||
|
||||
sandboxConversion: {
|
||||
image: process.env.CONVERSION_SANDBOX_IMAGE || 'sharelatex/pdftocairo:24.02',
|
||||
dockerSocketPath: process.env.CONVERSION_DOCKER_SOCKET || '/var/run/docker.sock',
|
||||
// The scratch dir has two views of the same path (the overleaf container
|
||||
// mounts ./data): `containerDir` for this process, `hostDir` for the Docker
|
||||
// daemon's bind mount. Both must be set when sandbox conversion is used.
|
||||
hostDir: process.env.CONVERSION_SANDBOX_HOST_DIR || null,
|
||||
containerDir: process.env.CONVERSION_SANDBOX_DIR || null,
|
||||
memoryBytes:
|
||||
parseInt(process.env.CONVERSION_SANDBOX_MEMORY_BYTES, 10) || 512 * 1024 * 1024,
|
||||
timeoutMs: parseInt(process.env.CONVERSION_SANDBOX_TIMEOUT_MS, 10) || 40000,
|
||||
},
|
||||
|
||||
gracefulShutdownDelayInMs:
|
||||
parseInt(process.env.GRACEFUL_SHUTDOWN_DELAY_SECONDS ?? '30', 10) * 1000,
|
||||
}
|
||||
|
||||
@@ -1199,8 +1199,8 @@ function _finaliseRequest(projectId, options, project, docs, files) {
|
||||
timeout: options.timeout,
|
||||
imageName: project.imageName,
|
||||
draft: Boolean(options.draft),
|
||||
// enable for premium compiles only
|
||||
png2pdf: Boolean(options.png2pdf) && hasPremiumCompiles,
|
||||
// CE: premium-compile (alpha/priority) gate removed
|
||||
png2pdf: Boolean(options.png2pdf),
|
||||
// enable for premium compiles on an image that has a checkpointing build
|
||||
enableCheckpoint:
|
||||
Boolean(options.checkpointing) &&
|
||||
|
||||
@@ -453,7 +453,8 @@ const _CompileController = {
|
||||
enablePng2Pdf,
|
||||
enableCheckpointCompiles,
|
||||
} = await _getSplitTestOptions(req, res)
|
||||
if (Features.hasFeature('saas')) {
|
||||
// CE: saas feature gate removed so png2pdf / compile-from-history options are sent
|
||||
if (true) {
|
||||
options.compileFromClsiCache = true
|
||||
options.populateClsiCache = true
|
||||
options.compileFromHistory = compileFromHistory
|
||||
|
||||
@@ -1108,10 +1108,8 @@ const _ProjectController = {
|
||||
canUseClsiCache:
|
||||
Features.hasFeature('saas') &&
|
||||
ownerFeatures?.compileGroup === 'priority',
|
||||
canUsePng2Pdf:
|
||||
Features.hasFeature('saas') &&
|
||||
ownerFeatures?.compileGroup === 'priority' &&
|
||||
splitTestAssignments['png2pdf']?.variant === 'enabled',
|
||||
// CE: saas/priority gate removed so the "Optimise images" toggle is available
|
||||
canUsePng2Pdf: true,
|
||||
user: {
|
||||
id: userId,
|
||||
email: user.email,
|
||||
|
||||
@@ -1262,6 +1262,12 @@ module.exports = {
|
||||
'export-html': 'enabled',
|
||||
'export-markdown': 'enabled',
|
||||
} : {}),
|
||||
...(process.env.ENABLE_PNG2PDF_CONVERSIONS === 'true' ? {
|
||||
'png2pdf': 'enabled',
|
||||
// compile-from-history is required for CLSI to record the "slow png" list
|
||||
// (clsi CompileManager.js saveSlowPngList is gated on isCompileFromHistory)
|
||||
'compile-from-history': 'enabled',
|
||||
} : {}),
|
||||
...(process.env.OVERLEAF_HISTORY_RESTORE?.toLowerCase() === 'true' ? {
|
||||
'history-ranges-support': 'enabled',
|
||||
'revert-file': 'enabled',
|
||||
|
||||
Reference in New Issue
Block a user