* [monorepo] reduce writes from chrome in cypress
Chrome puts its shared memory in /dev/shm, and docker's default 64MB is far
too small for it - which is why Cypress passes --disable-dev-shm-usage,
spilling those segments to files on the container's writable layer instead.
Under one spec that is a few hundred MB of write-then-delete churn; under the
CI fan-out it was most of the disk bandwidth on the whole box.
So every service built from dockerfiles/cypress gets shm_size 1gb and every
cypress config drops the flag again: the SaaS e2e service, web's CI and local
component-test services plus its standalone cypress one, and server-ce's e2e
and native cypress services. Both e2e-open variants inherit the setting
through `extends`; the native one is a service of its own, and would be left
running Chrome against 64MB once the flag is gone.
* [monorepo] adopt cypress-pnp-reporter
Materializing the reporter packages copied 267 packages / 3779 files
(59MB) into /tmp on every container start, once per spec. Resolving them
through PnP inside Cypress's Electron process instead drops that to
zero: the entrypoint shell went from 55.6MB of write_bytes to 0.
* [web] serve built assets from a prepared buffer between builds
Serving a chunk cost ~2ms whatever its size, and an editor page pulls ~144
of them, so webpack-dev-server burned 2-3 cores through an e2e run purely
answering requests for output that had not changed since the last build.
It is not bytes: cutting the bundle 2.6x with a cheaper devtool moved CPU
by 8%, and an empty 304 cost nearly as much as the full body. It is the
per-request work - reading the asset out of the in-memory filesystem and
re-hashing the whole body for a strong ETag, on every request.
Three things a build does that this has to respect, each found by tracing
the hooks rather than by reading them:
- `done` fires twice per build, for the compilation and for the manifest,
both with the same hash. A flag would have been cleared by the second
while the first was still running, so the gate is a counter.
- An incremental `done` reports only the assets it rewrote - 16, not 630 -
so asset names accumulate. Resetting them on `invalid` left the
manifest's two behind and nothing was served from here again.
- webpack-dev-middleware holds requests during a rebuild. This sits in
front of it, so it steps aside for that window and drops the cache when
a rebuild starts. Whether a response can be kept is then decided as it
ends rather than as it began: an epoch bumped on `invalid` names the
build its bytes were read from, and only a response from the current
build with no rebuild in flight is stored. A rebuild starting
mid-response can no longer write the previous build's bytes in behind
the cache being dropped, which eviction by asset name could not catch
anyway - an incremental build does not name the assets it left alone.
Measured on the editor entrypoint, 16 concurrent clients:
0.26-0.31s -> 0.045-0.050s of webpack CPU per page load, and 0.0455s
immediately after a rebuild. Same chunk graph, byte-identical output.
GitOrigin-RevId: 08b4ab2d665414069bc01b4763dd64faa9b4e5cb
* [editor-core] share the blob hash and the doc-vs-file rule
The git blob hash and the rule that decides whether a file is a doc or a binary
file both have to reach the same answer in every service that writes to history,
so they move into overleaf-editor-core rather than being spelled out again on
each side.
blob_utils takes over lib/blob_string_length.js: hashing and the string-length
rule answered two halves of the same question -- what history needs to know
about content before it can store it -- and each carried the same warning about
staying off index.js, which is bundled for the browser. One module states it
once, and blobForFile joins them, describing content on disk as the blob it
amounts to. The hashing functions are flat rather than a namespace of their own:
in a module that also holds the string-length rule, a bare fromStream says
nothing about what it makes. history-v1's blob store and recover_zip follow the
module to its new name.
file_type_detector holds the doc-vs-file rule, Node-only and deep-imported for
the same reason. It takes textExtensions, editableFilenames and maxDocLength as
configuration rather than reading settings: the limit is web's 2 MiB and not
editor-core's own 3 MiB, and the surrogate check covers both halves of a pair
where editor-core's covers the high half.
buildSetContentOperations' test spelled the blob header out for itself, with the
separator as a literal NUL byte, which had git reading the file as binary. It
hashes through blobHashFromString now, so it does not spell the header at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [editor-core] name the client a change came from on the origin itself
`EditorOrigin` carried an `editorId` that only an editor change could have. What
the id is for -- recognising a change again, so history does not apply a resend
twice and the client does not apply its own change back -- is a property of the
client that submitted it rather than of what it did, so it moves onto `Origin` as
an optional `historyClientId` and `EditorOrigin` goes.
- `Origin` takes `(kind, historyClientId)` and answers `getHistoryClientId()`;
`dropHistoryClientId()` is what the chunk store uses to strip the id from all
but a client's latest change, which now leaves the rest of the origin alone --
a restore that a client submitted stays a restore.
- `toRaw` is built a layer at a time: the base writes the kind and the id, a
subclass spreads that and adds the fields it owns.
- The named kinds are their own schemas and discriminate on `kind`, so a payload
claiming one is checked against that shape and the error says what it got
wrong. The catch-all cannot join them -- its kind is any string -- so it sits
alongside and is refined to refuse the named kinds, which it used to swallow:
`{kind: 'restore'}` validated as a kind and nothing else, then read back as a
restore that recorded nothing about what it restored.
- `lib/origin/*.js` gains `// @ts-check`. Without it the annotations in there
were decorative: the lazily required subclasses were `any`, so nothing handed
to them was checked, and `fromRaw` narrows with `in` to drop the catch-all
variant, which a kind check cannot.
The callers that stamp the id -- real-time's handshake flag, web's applyChange
body -- come with the rest of the branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [editor-core] recognise a change a writer that is not an editor placed
The counterpart of `isSameEditorChange` for writers that are not an editor
client: an integration that commits a batch on a user's behalf has to recognise
that batch coming back, because a lost response does not say whether the commit
landed and resubmitting one that did applies its removes twice.
It compares the same three fields, and for the same reason -- a rebase rewrites a
change's operations but leaves its origin, authors and timestamp alone -- with
the timestamp compared as a point in time rather than as a string, so that two
spellings of the same instant cannot read as different changes. github-sync is
the first caller, and comes with the rest of the branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [editor-core] load an editable file's content, or say it has none
Only an editable file has content: any other kind is bytes addressed by hash, and
hollow data records the size of content it does not have. Both answer
`getContent()` with null, so every caller that wants the content of a doc loaded
it, checked the data's class, and reached for its own error when the check failed.
`File#loadEager` does that once and answers the data that always has the content.
`buildSetContentOperations` takes a `ReadWriteBlobStore` -- everything but
looking a blob up by hash, which is all it takes to build operations over content
-- so a caller holding a store that cannot answer `getBlob` can still use it. Its
metadata argument is `FileMetadata` rather than `Object`, which is what the schema
already says a file's metadata is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [editor-core] say that a pair of operations that do not fit is unprocessable
compose and transform threw a bare Error when the two operations do not describe
the same content. Everything that answers a client for these -- history-v1's
import routes, the git-bridge push, project-history's resync -- decides on
UnprocessableError, so a bare Error reached them as an internal failure: a 500,
which reads as worth retrying, for a pair that fails the same way every time.
A client that resubmits on a 500 resubmits forever.
The guards that cannot be reached through the API keep their bare Error: a
TextOperation composed with something that is not one, and the branch whose
message says it should not happen.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* [history-v1] hash blobs with the rule editor-core holds
storage/lib/blob_hash.js was a bluebird-era implementation of the same git blob
hash editor-core holds, exported from the storage barrel and used across the
storage layer, the api and the tests. All of it reads the shared rule now, and
the test that existed only to check the copy is gone -- editor-core's suite
covers the same non-ASCII case.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
GitOrigin-RevId: 5a0c1e9a06170a0164c01a26a7bb2d205726d531
The hash of empty content is a property of the content rather than of anything
stored: every project's empty file has it, whether or not a blob was ever written
under it, and a file may reference it before anything has. Eight blob stores were
each paying a lookup -- a database query, an HTTP request, a read of a cache that
offline mode may be the only thing standing in for -- to be told what the hash
already says, and clsi had already reinvented the answer on its own.
overleaf-editor-core gains BlobStoreBase, which answers getString for that hash
and delegates the rest to the fetchString its subclasses provide. getObject comes
with it: four stores spelled out the same JSON.parse of getString, and a fifth
wrapped it in a log line.
Adopted by history-v1's BlobStore, BackupBlobStore, HashCheckBlobStore and
recover_zip's RecoveryBlobStore, by project-history's BlobStore, by clsi's
BlobStore, and by web's HistoryBlobStore, SimpleBlobStore and FrontendBlobStore.
history-v1 keeps its own answers in getStream, getBlob and getBlobs, which no
other store implements. Writing and copying a blob are untouched, so a project
that wrote empty content still has a blob for it and a read answers the same
either way.
Two paths could not handle a project holding a tracked change or a comment at
all. Loading such a file needs the ranges blob its rangesHash points at, and
hashing one again needs to write it back, so a store missing getObject or
putObject fails outright:
- recover_zip died on `blobStore.getObject is not a function`, making every such
project unrecoverable. The test covering it asserts a tracked delete is absent
from the zip, which the script already asked for and never reached.
- GET /projects/:project_id/latest/hashed_content returned 500 from the same
cause, then would have hit a missing putObject in snapshot.store.
HashCheckBlobStore becomes a class over the base and gains putObject.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
GitOrigin-RevId: c84f62465beff6cf77ffac26aa772984e1b554cb
* [editor-core] share the pieces the editor, real-time and history-v1 all need
Project-level history-OT has the browser, real-time and history-v1 all reason
about the same changes, so the rules they reason with move into the shared
library rather than being reimplemented on each side.
- `origin/editor_origin.js` plus `EDITOR_ORIGIN_KIND` and its schema entry: the
origin an editor client's change carries, holding the editorId that scopes a
change's timestamp.
- `change_identity.js`: how a submission is recognised when it comes back, so a
resend of a change that already landed is deduplicated rather than applied
twice.
- `rebase.js`: transforming a change onto concurrent ones, in the same order on
both sides, which is what lets the server acknowledge a change with only the
version it landed at.
- `text_file_defaults.js`: the doc-vs-file extension lists, now read from here
by web's settings instead of being spelled out again.
- `blob_string_length.js`: the string length of a blob, used by history-v1's
blob store and by web when it writes an uploaded file into history.
- `file_tree_diff.js`: file-tree operations to a diff, used by
project-history's FileTreeDiffGenerator.
- `Blob.EMPTY_HASH`, so callers can answer for the empty blob without storing
it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* [project-history] build the file-tree diff with the shared helper
FileTreeDiffGenerator's hand-rolled walk over add/edit/move/delete operations
becomes a call to `buildFileTreeDiff` from overleaf-editor-core, so the editor
and project-history read a file-tree change the same way. Adds the unit tests
the decaffeinated original never had.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* [web] read the editable-file lists from overleaf-editor-core
web's settings spelled out the text extensions and editable filenames that
decide whether a file becomes a doc, and every other service that has to make
that call carried its own copy. The lists now live in overleaf-editor-core and
web builds settings.textExtensions and settings.editableFilenames from them,
keeping the ADDITIONAL_TEXT_EXTENSIONS override for Server CE/Pro on top.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* [history-v1] read a blob's string length from overleaf-editor-core
Two copies of getStringLengthOfFile lived here, the one in recover_zip.js
carrying a TODO asking for exactly this. Both now import the shared
implementation, which the blob store keeps re-exporting for backupArchiver.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
GitOrigin-RevId: 31948332f416f56848e3b0dfb5142a4c9f93f508
REQ_VALIDATION_MODE=enforce-log rejects schema failures exactly as 'enforce'
does, but logs them first at instrumented call sites, so the visibility built
for the 'log' rollout (sanitized issues, caller trace, logFields values, per
schema/signature dedup) survives moving out of 'log' mode.
opts no longer change the outcome in this mode: logOnly does not suppress the
throw, the error is classified from the primary schema (not the fallback), and
fallbackSchema is only re-parsed to tag the log entry -- 'enforced-fallback-
passed' marks a rejection the looser pre-refinement schema would have avoided,
i.e. one the tightened schema newly introduces.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
GitOrigin-RevId: 0743322990e6c570ffbae166b4586285a18221df
* Relax split test import schema to allow staging export fields
The staging split test export includes fields (variant.active, version.__v,
splitTest.customerIoEnabled) that aren't otherwise written by the app but
are present in Mongo exports. The strict zod schema was rejecting the
import; mark these as optional so a staging export can be imported into a
local dev environment.
* [web] allow split-test import in dev-env and CI
* [web] ensure that the zod schema for split-tests covers all fields
---------
Co-authored-by: Thomas Mees <thomas.mees@overleaf.com>
GitOrigin-RevId: eab633e8498e351a64dc9a9dad85759e80744da0
* [web] read a linked project file's source from history
Behind the `linked-file-from-history` flag, a linked `project_file` import reads
the source out of the source project's latest history snapshot rather than out of
its file tree in mongo, where docs and files are two kinds of thing.
- `ProjectFileAgent` gains `_importFromHistory` alongside the existing
`_importFromProject`, and the flag picks between them. The snapshot only holds
what has reached history, so the source project is flushed through
document-updater and project-history first. A file that is editable is loaded
eagerly, which applies the edit operations sitting on top of its blob; one that
is not keeps its hash and is streamed. The file is promisified on the way.
- `History/HistoryOTContentManager.mjs` reads the latest chunk and rolls its
changes into a snapshot, retrying the read. `History/HistoryBlobStore.mjs` is
the blob store overleaf-editor-core asks for while loading a file.
- `LinkedFilesController` resolves the flag, including from the referer: the
import endpoints have no query string of their own. Every agent takes the
assignment so that the ones that do not read it still pass it along.
- `linked-file-from-history.spec.ts` drives the flag from the editor url, over
three imports from another project: a file uploaded to the source project and
then refreshed from it, an edit the source project has only made in its editor,
and a binary file. Both sources produce the same file, so which one was read is
asserted in the acceptance tests rather than here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [saas-e2e] add test coverage for tracked-delete in linked file source
* Apply suggestions from code review
Co-authored-by: Mathias Jakobsen <mathias.jakobsen@overleaf.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Mathias Jakobsen <mathias.jakobsen@overleaf.com>
GitOrigin-RevId: 3907d6d399e4b5315e68bf833663c06bc72f4011
* [docstore] use more realistic test stubs for ranges
* [monorepo] define ranges schemas in @overleaf/ranges-tracker
* [ranges-tracker] add legacy orderedRejections field in tracked-changes
* [monorepo] resolve @overleaf/ranges-tracker subpath imports in typescript
* [real-time] validate comment thread ids and tracked change id seeds
* [ranges-tracker] require ObjectId ids and tracked change metadata in ranges
* [monorepo] require ObjectId tracked change ids on the accept routes
* [web] use ObjectId comment thread ids in the DocumentController tests
GitOrigin-RevId: 58b20dec311d85b917ef63f9efbe745a265a2de6
* [editor-core] gracefully handle no-op TextOperation
* [web] avoid sending sync to pdf requests without a ?file
* [validation-tools] log deeper nested violations
* [editor-core] fix round trip of EditNoOperation
* [web] use custom error for sync to pdf w/o file, fix tests
* [validation-tools] include request params in log msg for violation
* Revert "[editor-core] gracefully handle no-op TextOperation"
This reverts commit c1d37261d5db72b37dbf39428c54edcfb5423f41.
* [web] gracefully handle rejected sync to pdf from context menu
GitOrigin-RevId: 21de7dc96b23575b0d67b6eb0c68ed9c2f86292b
* [editor-core] address zod violation for group_id
* [monorepo] use discriminated union in zod schemas where possible
* [editor-core] v1_source_doc_id is a number
GitOrigin-RevId: 38f591b07cca4e22954bfe30bf855545a6af675d
* [document-updater] adopt strict zod types and extend test coverage
* [saas-e2e] add test for project restore with binary and linked file
GitOrigin-RevId: a8e45bd1e46b6ac8a293da24c8b8ad7b236fd9bd
* [notifications] adopt strict zod types and extend test coverage
* [notifications] validate expires date via schema
GitOrigin-RevId: 25dee0578ec443447e4c38a50a7e6c8b44831fe7
- Trigger monorepo-check on Jenkins init scripts and fix violations
- Ignore all the json-dashboards in terraform folders
GitOrigin-RevId: f27b46d6fb81f5ac731f56b065951bede7d1a9e4
* [saas-e2e] run all the spec files
* [saas-e2e] add addComment helper
* [saas-e2e] add replyToComment helper
* [saas-e2e] check latexqc report for errors
* [saas-e2e] fail fast when the login request failed
* [monorepo] upsert the minimal v1 db entries in all environments
* [monorepo] serve the CDN assets from nginx to avoid session mix
* [monorepo] avoid loading fonts from Cypress UI via web
* [monorepo] effectively disable login + loadEditor rate-limit in dev-env
* [saas-e2e] project-list: generate unique project name on re-run
* [saas-e2e] add debug logs for latexqc
GitOrigin-RevId: 778003759414a3c993578062a631d083ecb1d2e6
* [editor-core] extend coverage of zod schemas
* [real-time] adopt strict zod types and extend test coverage
* [project-history] make type check happy
* [web] make type check happy
GitOrigin-RevId: e1a65b9b3560b31ad2af817067a4b549865a5bb1
tid does not exist anywhere else in document-updater at
sha 3d76f4b9bf when
the typo was added. The comment id was read from .t only.
Doc-snapshots are copied the ranges literally without any
normalizing of nested structures.
GitOrigin-RevId: f76fbc930fdf02d9367603b5b1d6414de8d9f559
* [saas-e2e] conversion: wait for initial compile before looking at PDF
* [eslint-plugin] s/before/beforeWithReRunOnTestRetry/ in e2e tests
* [server-ce] remove filestore migration scripts and E2E tests
* [server-ce] test: trim down upgrading tests to 5.x -> latest
GitOrigin-RevId: 2426ec4a4f3d67a5e0e7850ea84a942ae4619f94
With the removal of per-doc rooms, there is nothing to race anymore.
The next lower timeout is 0, i.e. useless check.
GitOrigin-RevId: effb6cc0525f5fd785c7dce8df4e279de1f1b13f
* [web] store ObjectIds in addUserToProject acceptance test helper
* [web] optimize project lookup for tpds operations
GitOrigin-RevId: 03db9fc450556072f177d56f640fcc6114b295d4
* [web] add endpoint for resolving a project for writes from tpds
* [web] return historyId from new endpoint for resolving projects
* [web] add zod validation to new endpoint for resolving projects
GitOrigin-RevId: 0e607e56442117a4150a08373169c44b53dd4fdf
* [agents] allow running all the test-runner skill commands
* [agents] move linting into AGENTS.md
* [agents] double down on permissions and cwd
* [saas-e2e] add support for MOCHA_GREP
* [saas-e2e] fail fast by default outside CI
* [agents] avoid need for Codespace detection
* [agents] work around missing trust prompt in Codespaces
* [saas-e2e] e2e_test_setup: merge split-tests in dev-env
GitOrigin-RevId: f1c1cd2a732e98b2e6125ef86a2b2cda5d3aabb6
* Add png to pdf converter for optimising image loading in pdftex
* refactor
* move fixtures
* refactor and add unit tests
* Remove gamma correction from PNG processing and update related tests
* Add crc32fast dependency and enhance PNG processing tests
* Add fixture test cases for PNG processing
* fix alpha channel to match pdftex
pdftex always uses an 8 bit alpha channel in the output even for 16 bit images
* Refactor PNG processing to use Option for alpha stream and update related logic
* Enhance PDF writing by adding DecodeParms for alpha stream and optimize PNG encoding process
* Add pixel dimensions support in PNG processing and PDF writing
* [copybara] move clsi excludes
* [clsi] move gitignore rules
* [png2pdf] move into dockerfiles folder
* [png2pdf] use atomic writes
* [clsi] integrate png2pdf
* [saas-e2e] add coverage for png2pdf
* [png2pdf] make compatible with seccomp profile in clsi
* [copybara] remove exclude following move of png2pdf
* [copybara] remove spurious excludes
* fix broken detection of malformed chunk headers
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* [project-history] add build script flag for depending on doc-updater types
* [png2pdf] run tests as part of clsi pipeline
* [png2pdf] fix calculation of PDF page size
* [web] put png2pdf behind feature flag and make it opt-in
* [web] fix ClsiManager.promises.sendRequest invocation after merge
* extend is_copy_safe function to follow pdftex
see the conditions in pdftex's writepng.c
https://svn.tug.org:8369/texlive/trunk/Build/source/texk/web2c/pdftexdir/writepng.c?revision=78081&view=markup#l575
the criteria are not an exact match because the png crate does not support all the
png chunk types
https://github.com/image-rs/image-png/issues/116
if the criteria result in a png being converted or skipped the final result should be the same.
in one case the conversion will be done by pdftex (at runtime)
in the other case the conversion will be done by png2pdf -
both conversions should give the same result.
* include tests in png2pdf Dockerfile
* [clsi] explicitly disable png2pdf conversion for project conversion
* [clsi] png2pdf: count converted files
* [png2pdf] bump date in docker image tag
* [web] make draft-setting option wider when png2pdf is available
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: d0d952ee009bf93c72bba2ffe763ae0160d396e4
* [web] add frontend tests for sharejs transform
* [web] extend tests for sharejs transform to flush eventually
* [web] full coverage for sharejs transform
* [web] add tests for sharejs events
* [web] add tests for sharejs callbacks
* [web] avoid sending empty sharejs op
* [web] add tests for resending after reconnecting
* [web] avoid resending of empty sharejs op after reconnecting
GitOrigin-RevId: 226fe47a7069cb877fd5a60454f86cc5855d9a0e
* [real-time] update tests for more realistic applyOtUpdate messages
* [saas-e2e] add tests for reviewer role
GitOrigin-RevId: f0a5307230beb00be3a1387129e06eb5b2106c09
* [web] use client-side filtering for split tests in admin panel
Filtering on non-indexed fields is not allowed with notablescan.
* [web] add initial value for DEFAULT_FILTERS.active
* [web] unset error before fetching split tests
GitOrigin-RevId: 0813873c5a90d57c4fdbfd0043fcc325f00408aa
* [migrations] only apply non-blocking migrations in CI
* [migrations] use "yarn run migrations" instead of "npm run migrations"
GitOrigin-RevId: d9f96db398adf2febcd82333295fb8d5eab741bb
Jakob Ackermann