The hash of empty content is a property of the content rather than of anything
stored: every project's empty file has it, whether or not a blob was ever written
under it, and a file may reference it before anything has. Eight blob stores were
each paying a lookup -- a database query, an HTTP request, a read of a cache that
offline mode may be the only thing standing in for -- to be told what the hash
already says, and clsi had already reinvented the answer on its own.
overleaf-editor-core gains BlobStoreBase, which answers getString for that hash
and delegates the rest to the fetchString its subclasses provide. getObject comes
with it: four stores spelled out the same JSON.parse of getString, and a fifth
wrapped it in a log line.
Adopted by history-v1's BlobStore, BackupBlobStore, HashCheckBlobStore and
recover_zip's RecoveryBlobStore, by project-history's BlobStore, by clsi's
BlobStore, and by web's HistoryBlobStore, SimpleBlobStore and FrontendBlobStore.
history-v1 keeps its own answers in getStream, getBlob and getBlobs, which no
other store implements. Writing and copying a blob are untouched, so a project
that wrote empty content still has a blob for it and a read answers the same
either way.
Two paths could not handle a project holding a tracked change or a comment at
all. Loading such a file needs the ranges blob its rangesHash points at, and
hashing one again needs to write it back, so a store missing getObject or
putObject fails outright:
- recover_zip died on `blobStore.getObject is not a function`, making every such
project unrecoverable. The test covering it asserts a tracked delete is absent
from the zip, which the script already asked for and never reached.
- GET /projects/:project_id/latest/hashed_content returned 500 from the same
cause, then would have hit a missing putObject in snapshot.store.
HashCheckBlobStore becomes a class over the base and gains putObject.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
GitOrigin-RevId: c84f62465beff6cf77ffac26aa772984e1b554cb
* feat: adding 2025 and 2026 incremental compiles options
* feat: remove dmtcp from compiler list and autoselect checkpointing version with labs inclusion
* feat: updating seccomp profile for checkpointing
* feat: disable non-premium dmtcp compiles, and align checkpoint version detection between clsi and web
* feat: adding in reminder of checkpoint experiment, and using session based split test retrieval
* Increase Packer build.sh timeout from 1h15m to 1h30m as build takes approx 1h
* fix: moving checkpointing check from tag name based to hasCheckpointing flag set as part of allowed_images, and removing uneeded seccomp rule
* feat: move variant/ image selection to CLSI, and have web control only what options to set for a compile. removing uneeded extra variants from webs list of allowed images
* feat: adding compile timer as a query string as well
---------
Co-authored-by: Tim Gibbon <tim.gibbon@overleaf.com>
GitOrigin-RevId: cdbdf76a62dcd5a585925f18d64b6cb9cd6ed373
* feat: releasing 2026.1 texlive to general release and promoting to default image
* fix: update clsi image tests so their likeness calcs work locally on different archs
* feat: updating epstopdf fixture and adding note to texlive Dockerfile used for codespaces
* fix: use generated image-eps-convert instead of fixture version that drifts
GitOrigin-RevId: 13c4f6977dbf07ddd34c9a5ab3ea295cddcbe025
* feat: adding in check for if the project could have had a png optimized as part of the compile-result-backend event
* feat: updating using new compile metrics to filter for only eligible project compiles
* move projectHasUnconvertedPngs to stats
* add e2e test for projectHasUnconvertedPngs
* add unit tests for compile-with-optimizable-pngs analytics event
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: 97e9d184420c34fc1d3e9f6cfe63f6b916f0202f
GHSA-72gw-mp4g-v24j / CVE-2026-5079 — DoS via deeply nested field names,
patched in 2.2.0. multer is a direct dep of web and clsi (file uploads).
- bump multer to 2.2.0 in services/web and services/clsi
- re-target the Overleaf local patch (attach files with no filename as
'undefined' instead of skipping) from 2.1.1 to 2.2.0, kept as a scoped
root resolution to match the repo's patch convention
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GitOrigin-RevId: 0d10137b521ebaacee71fc41adba32c4530a4041
* Add png to pdf converter for optimising image loading in pdftex
* refactor
* move fixtures
* refactor and add unit tests
* Remove gamma correction from PNG processing and update related tests
* Add crc32fast dependency and enhance PNG processing tests
* Add fixture test cases for PNG processing
* fix alpha channel to match pdftex
pdftex always uses an 8 bit alpha channel in the output even for 16 bit images
* Refactor PNG processing to use Option for alpha stream and update related logic
* Enhance PDF writing by adding DecodeParms for alpha stream and optimize PNG encoding process
* Add pixel dimensions support in PNG processing and PDF writing
* [copybara] move clsi excludes
* [clsi] move gitignore rules
* [png2pdf] move into dockerfiles folder
* [png2pdf] use atomic writes
* [clsi] integrate png2pdf
* [saas-e2e] add coverage for png2pdf
* [png2pdf] make compatible with seccomp profile in clsi
* [copybara] remove exclude following move of png2pdf
* [copybara] remove spurious excludes
* fix broken detection of malformed chunk headers
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* [project-history] add build script flag for depending on doc-updater types
* [png2pdf] run tests as part of clsi pipeline
* [png2pdf] fix calculation of PDF page size
* [web] put png2pdf behind feature flag and make it opt-in
* [web] fix ClsiManager.promises.sendRequest invocation after merge
* extend is_copy_safe function to follow pdftex
see the conditions in pdftex's writepng.c
https://svn.tug.org:8369/texlive/trunk/Build/source/texk/web2c/pdftexdir/writepng.c?revision=78081&view=markup#l575
the criteria are not an exact match because the png crate does not support all the
png chunk types
https://github.com/image-rs/image-png/issues/116
if the criteria result in a png being converted or skipped the final result should be the same.
in one case the conversion will be done by pdftex (at runtime)
in the other case the conversion will be done by png2pdf -
both conversions should give the same result.
* include tests in png2pdf Dockerfile
* [clsi] explicitly disable png2pdf conversion for project conversion
* [clsi] png2pdf: count converted files
* [png2pdf] bump date in docker image tag
* [web] make draft-setting option wider when png2pdf is available
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: d0d952ee009bf93c72bba2ffe763ae0160d396e4
* [monorepo] fix handling of quoted MOCHA_GREP
* [agents] refine guidance for running tests
- nudge it towards scoped module targets
- nudge it to really run e2e tests when changed
- nudge it towards a11y cypress matchers
- recommend quoting for MOCHA_GREP
* [web] fix forwarding of mocha args from module Makefile
* [monorepo] use BRANCH_NAME_TAG_SAFE for building the ci image name
* [web] also make BRANCH_NAME_TAG_SAFE available in module Makefile
(It's not really needed, but will make copilot happy.)
GitOrigin-RevId: bfda3c8fe2b4f857ef111ac63f94b35a82ad75f8
* [monorepo] provide clsi-sharded dev-env setup
* [monorepo] provide sharded clsi-nginx when using clsi-sharded
* [clsi-perf] run on all instances again
* [monorepo] switch default TL to 2025.1 in clsi (same as web)
* [monorepo] avoid starting base clsi when running clsi-perf with clsi-sharded
* [clsi-lb] use 3s check interval in clsi-sharded mode
GitOrigin-RevId: 3cae8f0c1db69dbd76b94b45dc45b3abb2fa0a83
* [clsi] forward instanceType to web
* [clsi] expose zone in compile response
* [clsi-perf] add new fields for instanceType and zone
* [clsi] expose isPreEmptible flag in compile response
* [clsi-perf] also add isPreEmptible field
* [clsi] rename isPreEmptible -> isSpotInstance
GitOrigin-RevId: 2bc2d772e0b14da83afb51f5f3e4fefd3f60b733
This was useful when iterating on the pdf-caching feature.
PDF caching has been stable for years. We can remove the logging now.
GitOrigin-RevId: 826c91c1e45d3dae1bb3ce5aeb7b6a3f9283392c
* [monorepo] consolidate clsi-lb host/ip env-vars
Target env-var is CLSI_LB_HOST. Keep CLSI_LB_IP populated for a week.
* [clsi] initial version of /convert/pdf-to-jpeg
* [rails] use fake-secrets in CI and Codespaces
* [rails] adapt tests for using clsi to convert PDFs to image
* [rails] add rake task for comparing clsi conversion with transloadit
* [clsi] double check that output.jpg is a regular file
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [clsi] fix composing basename
* [monorepo] fix clsi-lb host env-var post merge
* [monorepo] sort dev-environment.env hosts
* [rails] use local pdf file rather than downloading it again
Download from the old renderer code path still. It's dead code.
* [terraform] clsi: enable pdf to jpg conversion
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: 5ecaa8559d299486340bb3961f06b29f7c4dfcca
* upgrade from eslint version 8 to eslint version 10
* remove unsupported eslint-env directive
* include jsx files in latexqc linting
* use basePath and extends to maintain paths in writefull eslint
* fix yarn.lock
with ./bin/yarn install
* preserve existing glob patterns in web eslint config
* restore original comments
* fix worker path
* corrected comment about eslint-plugin-mocha
* remove unused imports
* remove unused import of includeIgnoreFile
* switch to individual eslit.config.mjs files
* fix lint errors on eslint.config.mjs in web
* update build scripts for eslint.config.mjs
* update volumes for RUN_LINTING_CI_MONOREPO in web Makefile
updated manually as this makefile is not autogenerated
the RUN_LINTING_CI_MONOREPO command is only used for prettier, not eslint, but updating for consistency.
* migrate from mocha/no-skipped-tests to mocha/no-pending-tests
see https://github.com/lo1tuma/eslint-plugin-mocha/pull/365
"rule no-skipped-tests has been removed, its functionality has been merged into the existing no-pending-tests rule"
GitOrigin-RevId: 2c8f25c8049a0dba374a51df1214286bb5093a51
* [clsi] add request flag for isCompileFromHistory
* [clsi] derive cacheKey for history snapshot from compile dir
* [clsi] migrate convert project to document to compile from history
* [clsi] address review feedback
* [web] determine root doc at the time of converting the project
* [web] wait for flush before starting document conversion
* [saas-e2e] add tests for root doc override when converting project
GitOrigin-RevId: 71c578030949b89f3a74e7f7ab882dfa9c98c17a
yu-i-i