* [monorepo] reduce writes from chrome in cypress
Chrome puts its shared memory in /dev/shm, and docker's default 64MB is far
too small for it - which is why Cypress passes --disable-dev-shm-usage,
spilling those segments to files on the container's writable layer instead.
Under one spec that is a few hundred MB of write-then-delete churn; under the
CI fan-out it was most of the disk bandwidth on the whole box.
So every service built from dockerfiles/cypress gets shm_size 1gb and every
cypress config drops the flag again: the SaaS e2e service, web's CI and local
component-test services plus its standalone cypress one, and server-ce's e2e
and native cypress services. Both e2e-open variants inherit the setting
through `extends`; the native one is a service of its own, and would be left
running Chrome against 64MB once the flag is gone.
* [monorepo] adopt cypress-pnp-reporter
Materializing the reporter packages copied 267 packages / 3779 files
(59MB) into /tmp on every container start, once per spec. Resolving them
through PnP inside Cypress's Electron process instead drops that to
zero: the entrypoint shell went from 55.6MB of write_bytes to 0.
* [web] serve built assets from a prepared buffer between builds
Serving a chunk cost ~2ms whatever its size, and an editor page pulls ~144
of them, so webpack-dev-server burned 2-3 cores through an e2e run purely
answering requests for output that had not changed since the last build.
It is not bytes: cutting the bundle 2.6x with a cheaper devtool moved CPU
by 8%, and an empty 304 cost nearly as much as the full body. It is the
per-request work - reading the asset out of the in-memory filesystem and
re-hashing the whole body for a strong ETag, on every request.
Three things a build does that this has to respect, each found by tracing
the hooks rather than by reading them:
- `done` fires twice per build, for the compilation and for the manifest,
both with the same hash. A flag would have been cleared by the second
while the first was still running, so the gate is a counter.
- An incremental `done` reports only the assets it rewrote - 16, not 630 -
so asset names accumulate. Resetting them on `invalid` left the
manifest's two behind and nothing was served from here again.
- webpack-dev-middleware holds requests during a rebuild. This sits in
front of it, so it steps aside for that window and drops the cache when
a rebuild starts. Whether a response can be kept is then decided as it
ends rather than as it began: an epoch bumped on `invalid` names the
build its bytes were read from, and only a response from the current
build with no rebuild in flight is stored. A rebuild starting
mid-response can no longer write the previous build's bytes in behind
the cache being dropped, which eviction by asset name could not catch
anyway - an incremental build does not name the assets it left alone.
Measured on the editor entrypoint, 16 concurrent clients:
0.26-0.31s -> 0.045-0.050s of webpack CPU per page load, and 0.0455s
immediately after a rebuild. Same chunk graph, byte-identical output.
GitOrigin-RevId: 08b4ab2d665414069bc01b4763dd64faa9b4e5cb
* [web] open a comment from a URL
* [web] add "View comment" button to digest email
* [web] open a tracked change from a URL
* [web] add "View suggestion" button to digest email
* [web] open the review panel from an open=review-panel URL
* [web] drop the digest email CTA and collapse the deep links to t=review
* [web] add an Open project button to the section overflow row and plain-text view links
* [web] validate the comment and change deep-link ids as object ids
* [web] collapse the deep-link resolvers into a single DeepLink component
* fix: scrolling into view when below the fold
* fix: address claude review
* fix: style adjustments
* fix: updating types to accomodate accepted/rejected changes
---------
Co-authored-by: Jimmy Domagala-Tang <jimmy.domagala-tang@overleaf.com>
GitOrigin-RevId: dab535591a93e1c3849a6b2a0dcc3e3934354385
The hash of empty content is a property of the content rather than of anything
stored: every project's empty file has it, whether or not a blob was ever written
under it, and a file may reference it before anything has. Eight blob stores were
each paying a lookup -- a database query, an HTTP request, a read of a cache that
offline mode may be the only thing standing in for -- to be told what the hash
already says, and clsi had already reinvented the answer on its own.
overleaf-editor-core gains BlobStoreBase, which answers getString for that hash
and delegates the rest to the fetchString its subclasses provide. getObject comes
with it: four stores spelled out the same JSON.parse of getString, and a fifth
wrapped it in a log line.
Adopted by history-v1's BlobStore, BackupBlobStore, HashCheckBlobStore and
recover_zip's RecoveryBlobStore, by project-history's BlobStore, by clsi's
BlobStore, and by web's HistoryBlobStore, SimpleBlobStore and FrontendBlobStore.
history-v1 keeps its own answers in getStream, getBlob and getBlobs, which no
other store implements. Writing and copying a blob are untouched, so a project
that wrote empty content still has a blob for it and a read answers the same
either way.
Two paths could not handle a project holding a tracked change or a comment at
all. Loading such a file needs the ranges blob its rangesHash points at, and
hashing one again needs to write it back, so a store missing getObject or
putObject fails outright:
- recover_zip died on `blobStore.getObject is not a function`, making every such
project unrecoverable. The test covering it asserts a tracked delete is absent
from the zip, which the script already asked for and never reached.
- GET /projects/:project_id/latest/hashed_content returned 500 from the same
cause, then would have hit a missing putObject in snapshot.store.
HashCheckBlobStore becomes a class over the base and gains putObject.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
GitOrigin-RevId: c84f62465beff6cf77ffac26aa772984e1b554cb
* [editor-core] share the pieces the editor, real-time and history-v1 all need
Project-level history-OT has the browser, real-time and history-v1 all reason
about the same changes, so the rules they reason with move into the shared
library rather than being reimplemented on each side.
- `origin/editor_origin.js` plus `EDITOR_ORIGIN_KIND` and its schema entry: the
origin an editor client's change carries, holding the editorId that scopes a
change's timestamp.
- `change_identity.js`: how a submission is recognised when it comes back, so a
resend of a change that already landed is deduplicated rather than applied
twice.
- `rebase.js`: transforming a change onto concurrent ones, in the same order on
both sides, which is what lets the server acknowledge a change with only the
version it landed at.
- `text_file_defaults.js`: the doc-vs-file extension lists, now read from here
by web's settings instead of being spelled out again.
- `blob_string_length.js`: the string length of a blob, used by history-v1's
blob store and by web when it writes an uploaded file into history.
- `file_tree_diff.js`: file-tree operations to a diff, used by
project-history's FileTreeDiffGenerator.
- `Blob.EMPTY_HASH`, so callers can answer for the empty blob without storing
it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* [project-history] build the file-tree diff with the shared helper
FileTreeDiffGenerator's hand-rolled walk over add/edit/move/delete operations
becomes a call to `buildFileTreeDiff` from overleaf-editor-core, so the editor
and project-history read a file-tree change the same way. Adds the unit tests
the decaffeinated original never had.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* [web] read the editable-file lists from overleaf-editor-core
web's settings spelled out the text extensions and editable filenames that
decide whether a file becomes a doc, and every other service that has to make
that call carried its own copy. The lists now live in overleaf-editor-core and
web builds settings.textExtensions and settings.editableFilenames from them,
keeping the ADDITIONAL_TEXT_EXTENSIONS override for Server CE/Pro on top.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* [history-v1] read a blob's string length from overleaf-editor-core
Two copies of getStringLengthOfFile lived here, the one in recover_zip.js
carrying a TODO asking for exactly this. Both now import the shared
implementation, which the blob store keeps re-exporting for backupArchiver.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
GitOrigin-RevId: 31948332f416f56848e3b0dfb5142a4c9f93f508
* Delete library data when an expired deleted user is cleaned up
* Refactor MongoDB session handling
* Remove transaction handling from deleteAllLibraryDataForUser function
* use findUserIdsToKeep rather than filterDanglingUserIds
GitOrigin-RevId: 9e82f6bf943809b55a8b042d9240f99dbc635711
Server Pro/CE admins can set a custom header logo via OVERLEAF_HEADER_IMAGE_URL,
but the same image is used for both the light and dark navbar themes. A logo
with good contrast on one background can be hard to see on the other (e.g. a
dark logo on the dark dashboard navbar).
Add two optional env vars, OVERLEAF_HEADER_IMAGE_URL_LIGHT and
OVERLEAF_HEADER_IMAGE_URL_DARK, each falling back to OVERLEAF_HEADER_IMAGE_URL
when unset, so existing configurations are unaffected. The light/dark fallback is
resolved server-side and exposed to the frontend as customLogo (light-effective)
and customLogoDark (dark-effective); the two theme-aware navbars (project
dashboard and library) pick between them using the existing overall-theme hook,
so the logo tracks both the user's theme setting and live OS theme changes.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
GitOrigin-RevId: 587a5679c650717cba6251e6d612146f434e8730
* Fix dark-mode white flash when navigating between dashboard and library
* Use userSettings object in layout-base.pug
* Move initial theme styles in a new file
* Sanitize data-initial-theme with a generic getInitialTheme helper
* Add getInitialTheme stub
GitOrigin-RevId: 80812093bbf75dc2a6f6f24710324a31a1930265
* Relax split test import schema to allow staging export fields
The staging split test export includes fields (variant.active, version.__v,
splitTest.customerIoEnabled) that aren't otherwise written by the app but
are present in Mongo exports. The strict zod schema was rejecting the
import; mark these as optional so a staging export can be imported into a
local dev environment.
* [web] allow split-test import in dev-env and CI
* [web] ensure that the zod schema for split-tests covers all fields
---------
Co-authored-by: Thomas Mees <thomas.mees@overleaf.com>
GitOrigin-RevId: eab633e8498e351a64dc9a9dad85759e80744da0
Two of the B2B request schemas flagged in #36577 reject bodies we don't
want to reject:
- POST /manage/groups/:id/settings/sso: the group settings frontend
spreads the SSO config document it fetched from the GET route back into
the POST body, so the request carries `_id`, `__v`, `validated`,
`enabled` and `useSettingsUKAMF` that the handler recomputes or ignores.
- POST /manage/groups/:id/invites: an internal script sends an unsupported
`expiresAt` alongside the email; the handler only reads `email`.
Swap z.strictObject() for z.object() on both, with a comment on each
explaining why it's deliberately non-strict.
GitOrigin-RevId: 5ff94ded579bdda07efc59d9ceb1d495964a912d
Stop rejecting unrecognized request fields in the Dropbox, GitHub-sync, git-bridge and reference-manager APIs
GitOrigin-RevId: 5efdee90830b2a48172f839da8e59ef938746f65
* [web] read a linked project file's source from history
Behind the `linked-file-from-history` flag, a linked `project_file` import reads
the source out of the source project's latest history snapshot rather than out of
its file tree in mongo, where docs and files are two kinds of thing.
- `ProjectFileAgent` gains `_importFromHistory` alongside the existing
`_importFromProject`, and the flag picks between them. The snapshot only holds
what has reached history, so the source project is flushed through
document-updater and project-history first. A file that is editable is loaded
eagerly, which applies the edit operations sitting on top of its blob; one that
is not keeps its hash and is streamed. The file is promisified on the way.
- `History/HistoryOTContentManager.mjs` reads the latest chunk and rolls its
changes into a snapshot, retrying the read. `History/HistoryBlobStore.mjs` is
the blob store overleaf-editor-core asks for while loading a file.
- `LinkedFilesController` resolves the flag, including from the referer: the
import endpoints have no query string of their own. Every agent takes the
assignment so that the ones that do not read it still pass it along.
- `linked-file-from-history.spec.ts` drives the flag from the editor url, over
three imports from another project: a file uploaded to the source project and
then refreshed from it, an edit the source project has only made in its editor,
and a binary file. Both sources produce the same file, so which one was read is
asserted in the acceptance tests rather than here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [saas-e2e] add test coverage for tracked-delete in linked file source
* Apply suggestions from code review
Co-authored-by: Mathias Jakobsen <mathias.jakobsen@overleaf.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Mathias Jakobsen <mathias.jakobsen@overleaf.com>
GitOrigin-RevId: 3907d6d399e4b5315e68bf833663c06bc72f4011
* feat: gating labs experiments behind premium compile feature
* feat: move filtering to the labs module, and only propogate experiments that user can opt into on the FE
GitOrigin-RevId: 984aa0339c7dc8d61f0f6830780a599508306d5e
* Move bulk bibtex download to a server-side streaming endpoint
* Wrap download fetch errors in FetchError and parse the error body
* Refactor library references download to use GET request and stream response
* Remove MAX_DOWNLOAD_IDS limit from downloadIdsSchema and related test
* fix merge conflict
GitOrigin-RevId: dee2bae8bfb982cd1b79bbd21c9f1b90e8cff1a4
* [docstore] use more realistic test stubs for ranges
* [monorepo] define ranges schemas in @overleaf/ranges-tracker
* [ranges-tracker] add legacy orderedRejections field in tracked-changes
* [monorepo] resolve @overleaf/ranges-tracker subpath imports in typescript
* [real-time] validate comment thread ids and tracked change id seeds
* [ranges-tracker] require ObjectId ids and tracked change metadata in ranges
* [monorepo] require ObjectId tracked change ids on the accept routes
* [web] use ObjectId comment thread ids in the DocumentController tests
GitOrigin-RevId: 58b20dec311d85b917ef63f9efbe745a265a2de6
Jakob Ackermann