* feat: adding 2025 and 2026 incremental compiles options
* feat: remove dmtcp from compiler list and autoselect checkpointing version with labs inclusion
* feat: updating seccomp profile for checkpointing
* feat: disable non-premium dmtcp compiles, and align checkpoint version detection between clsi and web
* feat: adding in reminder of checkpoint experiment, and using session based split test retrieval
* Increase Packer build.sh timeout from 1h15m to 1h30m as build takes approx 1h
* fix: moving checkpointing check from tag name based to hasCheckpointing flag set as part of allowed_images, and removing uneeded seccomp rule
* feat: move variant/ image selection to CLSI, and have web control only what options to set for a compile. removing uneeded extra variants from webs list of allowed images
* feat: adding compile timer as a query string as well
---------
Co-authored-by: Tim Gibbon <tim.gibbon@overleaf.com>
GitOrigin-RevId: cdbdf76a62dcd5a585925f18d64b6cb9cd6ed373
* feat: adding in check for if the project could have had a png optimized as part of the compile-result-backend event
* feat: updating using new compile metrics to filter for only eligible project compiles
* move projectHasUnconvertedPngs to stats
* add e2e test for projectHasUnconvertedPngs
* add unit tests for compile-with-optimizable-pngs analytics event
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: 97e9d184420c34fc1d3e9f6cfe63f6b916f0202f
* Add png to pdf converter for optimising image loading in pdftex
* refactor
* move fixtures
* refactor and add unit tests
* Remove gamma correction from PNG processing and update related tests
* Add crc32fast dependency and enhance PNG processing tests
* Add fixture test cases for PNG processing
* fix alpha channel to match pdftex
pdftex always uses an 8 bit alpha channel in the output even for 16 bit images
* Refactor PNG processing to use Option for alpha stream and update related logic
* Enhance PDF writing by adding DecodeParms for alpha stream and optimize PNG encoding process
* Add pixel dimensions support in PNG processing and PDF writing
* [copybara] move clsi excludes
* [clsi] move gitignore rules
* [png2pdf] move into dockerfiles folder
* [png2pdf] use atomic writes
* [clsi] integrate png2pdf
* [saas-e2e] add coverage for png2pdf
* [png2pdf] make compatible with seccomp profile in clsi
* [copybara] remove exclude following move of png2pdf
* [copybara] remove spurious excludes
* fix broken detection of malformed chunk headers
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* [project-history] add build script flag for depending on doc-updater types
* [png2pdf] run tests as part of clsi pipeline
* [png2pdf] fix calculation of PDF page size
* [web] put png2pdf behind feature flag and make it opt-in
* [web] fix ClsiManager.promises.sendRequest invocation after merge
* extend is_copy_safe function to follow pdftex
see the conditions in pdftex's writepng.c
https://svn.tug.org:8369/texlive/trunk/Build/source/texk/web2c/pdftexdir/writepng.c?revision=78081&view=markup#l575
the criteria are not an exact match because the png crate does not support all the
png chunk types
https://github.com/image-rs/image-png/issues/116
if the criteria result in a png being converted or skipped the final result should be the same.
in one case the conversion will be done by pdftex (at runtime)
in the other case the conversion will be done by png2pdf -
both conversions should give the same result.
* include tests in png2pdf Dockerfile
* [clsi] explicitly disable png2pdf conversion for project conversion
* [clsi] png2pdf: count converted files
* [png2pdf] bump date in docker image tag
* [web] make draft-setting option wider when png2pdf is available
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: d0d952ee009bf93c72bba2ffe763ae0160d396e4
* [clsi] add request flag for isCompileFromHistory
* [clsi] derive cacheKey for history snapshot from compile dir
* [clsi] migrate convert project to document to compile from history
* [clsi] address review feedback
* [web] determine root doc at the time of converting the project
* [web] wait for flush before starting document conversion
* [saas-e2e] add tests for root doc override when converting project
GitOrigin-RevId: 71c578030949b89f3a74e7f7ab882dfa9c98c17a
* [clsi] handle draft mode and tikzexternalize as part of sync phase
* [clsi] emit empty string from SafeReader on ENOENT
* [clsi] persist history state after clearing dirty state without changes
GitOrigin-RevId: d9dcd2e6887017f7935b5e95bdbdc6e11a3b18f5
* [clsi] avoid server error when clearing cache while compiling
* [clsi] tweak API around releasing locks
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
---------
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: d3f171467d3bc26941758dd333f30049b37a05c8
* [saas-e2e] test gallery templates with binary file
* [rails] add make target for fixing rubocop errors
* [rails] migrate compiles of conversions/submissions to history mode
* [rails] forward version to clsi request
* [rails] trim down compile request
* [saas-e2e] source v1 secrets after make install
GitOrigin-RevId: 65269e1df1051c9f3b4f1813d2e9dcf32a01be50
* [clsi] initial implementation of compile from history
* [clsi] copy changes
* [saas-e2e] extend test case with nested folder
* [saas-e2e] add test case for tracked changes
* [web] fix accumulating changes from multiple chunks
* [web] optimize size check for compile request payload
* [clsi] deduplicate globalBlobs
* [clsi] add validation for request body details
* [clsi] add metrics for compile from history
* [clsi] download binary files concurrently
* [clsi] skip download of empty file blob
* [clsi] break down e2e compile time metric by compileFromHistory
GitOrigin-RevId: 0dadef93e89d8a172c35cb130a1042d9d1bec42a
* [clsi] remove all clsi-perf/health-check metrics
* [clsi] always emit E2E compile time metric
* [clsi] do not collect metrics for clsi-cache-template compiles
* [clsi] fix unit tests: request.metricsOpts always exists
* [clsi] use a gauge for the e2e compile time metric of clsi-perf
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
* [clsi] remove metrics for binary file downloads from clsi-perf
---------
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: 7995512e57c802086350e3d1a0ec5213ecdf0a05
* [clsi] add stats and timings to compile response from clsi-cache
* [clsi] set downloadedFromCache when previously downloaded for synctex
Assumption: every compile will emit an output.log. When the output.log
is missing, but the output.synctex.gz exists, it must have been
downloaded from the cache.
GitOrigin-RevId: 41ea34880931e3c43dda3bc9eb26c0d02054894d
* [clsi] tell frontend when synctex mapping was downloaded from clsi-cache
* [web] emit event when synctex mapping was downloaded from clsi-cache
GitOrigin-RevId: 1f6b7e0faaa7dd76449aad566802da971a4cf9ed
* [clsi] drop support for docker-in-docker
* [clsi] run SyncTeX in specific output dir rather than compile dir
* [clsi] store output.synctex.gz outside of tar-ball in clsi-cache
* [clsi] add documentation for rewriting of docker bind-mounts
* [server-pro] update env vars for sandboxed compiles in sample config
GitOrigin-RevId: 8debd7102ac612544961f237aa4ff1c530aa3da3
* [clsi-cache] initial revision of the clsi-cache service
* [clsi] send output files to clsi-cache and import from clsi-cache
* [web] pass editorId to clsi
* [web] clear clsi-cache when clearing clsi cache
* [web] add split-tests for controlling clsi-cache rollout
* [web] populate clsi-cache when cloning/creating project from template
* [clsi-cache] produce less noise when populating cache hits 404
* [clsi-cache] push docker image to AR
* [clsi-cache] push docker image to AR
* [clsi-cache] allow compileGroup in job payload
* [clsi-cache] set X-Zone header from latest endpoint
* [clsi-cache] use method POST for /enqueue endpoint
* [web] populate clsi-cache in zone b with template data
* [clsi-cache] limit number of editors per project/user folder to 10
* [web] clone: populate the clsi-cache unless the TeXLive release changed
* [clsi-cache] keep user folder when clearing cache as anonymous user
* [clsi] download old output.tar.gz when synctex finds empty compile dir
* [web] fix lint
* [clsi-cache] multi-zonal lookup of single build output
* [clsi-cache] add more validation and limits
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [clsi] do not include clsi-cache tar-ball in output.zip
* [clsi-cache] fix reference after remaining constant
Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
* [web] consolidate validation of filename into ClsiCacheHandler
* [clsi-cache] extend metrics and event tracking
- break down most of the clsi metrics by label
- compile=initial - new compile dir without previous output files
- compile=recompile - recompile in existing compile dir
- compile=from-cache - compile using previous clsi-cache
- extend segmentation on compile-result-backend event
- isInitialCompile=true - found new compile dir at start of request
- restoredClsiCache=true - restored compile dir from clsi-cache
* [clsi] rename metrics labels for download of clsi-cache
This is in preparation for synctex changes.
* [clsi] use constant for limit of entries in output.tar.gz
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
* [clsi-cache] fix cloning of project cache
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: 4901a65497af13be1549af7f38ceee3188fcf881
* [clsi] make error copying output files a fatal compile error
Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>
* [clsi] remove unused endpoints for downloading files from compileDir
Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>
* [clsi] avoid useless clone of output files
* [clsi] add test for output files when compile did not produce a PDF
---------
Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>
GitOrigin-RevId: cb998b99b4d96cb48ddd70987958f614ad3b40fc
* Add `unicorn/prefer-node-protocol`
* Fix `unicorn/prefer-node-protocol` ESLint errors
* Run `npm run format:fix`
* Add sandboxed-module sourceTransformers in mocha setups
Fix `no such file or directory, open 'node:fs'` in `sandboxed-module`
* Remove `node:` in the SandboxedModule requires
* Fix new linting errors with `node:`
GitOrigin-RevId: 68f6e31e2191fcff4cb8058dd0a6914c14f59926
