* [editor-core] hold the one threshold that decides which store owns the file tree
HISTORY_FILE_TREE_STAGE is five literals across the services that consume this
library -- web backend, web editor, real-time, third-party-datastore,
github-sync -- linked by comments saying which other copies they are aligned
with. What the docstring here says is what goes wrong: this is the single thing
that decides which storage is authoritative, a project crosses it once with no
way back, and a copy that drifts sends a whole service's writes to the store
nothing reads any more. Silently, and for good.
overleaf-editor-core is where it goes: every one of the five already depends on
it, the editor included, and the library that defines what a file tree is can say
where one lives. The consumers adopt it separately; this is the definition they
adopt.
historyIsSourceOfTruth goes with it, and refuses a stage that is neither absent
nor a number rather than reading it as low -- web's copy coerced, which turned a
corrupt stage into a permanent mongo write. Absent stays "below the gate": a
project that never migrated has no stage.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [editor-core] hold which doc a project opens on
History records it as `main: true` on the file, and nothing derives it: every
writer that creates a project has to record it, or the project compiles nothing
until the user picks a file. So the rule for choosing one, and the operations
that record it, go where a file tree is defined rather than with any one writer.
The rule is the one web has always applied, over a snapshot instead of a mongo
tree: of the docs that declare a document class, the one nearest the project
root, alphabetically among equals, so that the same files always give the same
answer. A project that already records a root doc keeps it -- that is a user's
choice and not something to re-derive.
Whether a doc could be the one is asked of content rather than of a store to read
it from: a writer creating a project has each doc's content in its hands as it
stores it, so reading them back would be a second read of every one. The choice
is then made from the pathnames alone.
The extension list joins the others in text_file_defaults for the reason they are
there: a writer holding a shorter copy leaves a project with nothing to compile.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [editor-core] say why a store built for a consumer does not need getBlob
ReadWriteBlobStore is BlobStore without the lookup by hash, and the type said so
without saying why a consumer can do without it. getBlob is reached only for a
file stored as a bare hash, and one of those does not leave history-v1:
chunk_store's lazyLoadHistoryFiles attaches blob metadata to every snapshot and
change it serves, so a file arrives already knowing its own lengths.
That is what the contract rests on, and a store that reimplemented the
classification rule to answer the lookup anyway would be a second opinion on
whether a file is editable. The reasoning was what was missing, rather than the
type.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [editor-core] say what makes a doc a root doc candidate
The sentence naming the two halves of the check had lost the first one, and read
"named as a root doc may be named". It is the extension that is tested: one of
DEFAULT_ROOT_DOC_EXTENSIONS, and a document class declared in the content.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [editor-core] hold the doc flags a file's metadata can carry in one place
`main` is not the only flag a doc carries -- `mainBibliography` marks the file
references are added to -- and both live in the same metadata, which
SetFileMetadataOperation replaces wholesale rather than merges. So every writer
that sets one has to write the others out again, and every reader that asks
whether a file is a doc has to know the whole list.
root_doc was spelling all three of those for itself against `main` alone: it
tested the key by hand, refused a file whose metadata held anything else, and
cleared the metadata to `{}` when moving the root doc elsewhere. Against a file
that is also the main bibliography, the first two refuse a legitimate root doc
and the third silently drops the bibliography.
The helpers answer it once: which keys are doc flags, whether metadata holds only
those, and what metadata with one flag changed looks like. root_doc uses them
throughout, and the schema takes both flags on one file.
Co-Authored-By: Mathias Jakobsen <mathias.jakobsen@overleaf.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [editor-core] ask the snapshot which doc carries a flag
Finding the file a doc flag is set on is the same search whichever flag is being
asked about, and it belongs to the snapshot rather than to the caller: it walks
the snapshot's own pathnames, and it is the snapshot that knows a file's
metadata without its content being loaded.
root_doc's mainPathname is that search for `main`, and now delegates. It gains
the check it was missing: a file whose metadata holds anything besides doc flags
came from outside the editor, so a stale `main` on it does not make it the root
doc -- which is already what setMainPathnameOperations refuses to write.
Co-Authored-By: Mathias Jakobsen <mathias.jakobsen@overleaf.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [editor-core] drop mainPathname for the snapshot method behind it
It became a second name for `snapshot.getPathnameWithDocFlag('main')` and nothing
in the library was left calling it but chooseRootDoc, which asks the snapshot
directly now. Its two tests go with it: Snapshot's own cover the same two answers,
and more of them.
Callers outside the library ask the snapshot too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Mathias Jakobsen <mathias.jakobsen@overleaf.com>
GitOrigin-RevId: b3f503aa3e94ea8b6524d83f910fe33dc381c0a5
* [editor-core] share the blob hash and the doc-vs-file rule
The git blob hash and the rule that decides whether a file is a doc or a binary
file both have to reach the same answer in every service that writes to history,
so they move into overleaf-editor-core rather than being spelled out again on
each side.
blob_utils takes over lib/blob_string_length.js: hashing and the string-length
rule answered two halves of the same question -- what history needs to know
about content before it can store it -- and each carried the same warning about
staying off index.js, which is bundled for the browser. One module states it
once, and blobForFile joins them, describing content on disk as the blob it
amounts to. The hashing functions are flat rather than a namespace of their own:
in a module that also holds the string-length rule, a bare fromStream says
nothing about what it makes. history-v1's blob store and recover_zip follow the
module to its new name.
file_type_detector holds the doc-vs-file rule, Node-only and deep-imported for
the same reason. It takes textExtensions, editableFilenames and maxDocLength as
configuration rather than reading settings: the limit is web's 2 MiB and not
editor-core's own 3 MiB, and the surrogate check covers both halves of a pair
where editor-core's covers the high half.
buildSetContentOperations' test spelled the blob header out for itself, with the
separator as a literal NUL byte, which had git reading the file as binary. It
hashes through blobHashFromString now, so it does not spell the header at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [editor-core] name the client a change came from on the origin itself
`EditorOrigin` carried an `editorId` that only an editor change could have. What
the id is for -- recognising a change again, so history does not apply a resend
twice and the client does not apply its own change back -- is a property of the
client that submitted it rather than of what it did, so it moves onto `Origin` as
an optional `historyClientId` and `EditorOrigin` goes.
- `Origin` takes `(kind, historyClientId)` and answers `getHistoryClientId()`;
`dropHistoryClientId()` is what the chunk store uses to strip the id from all
but a client's latest change, which now leaves the rest of the origin alone --
a restore that a client submitted stays a restore.
- `toRaw` is built a layer at a time: the base writes the kind and the id, a
subclass spreads that and adds the fields it owns.
- The named kinds are their own schemas and discriminate on `kind`, so a payload
claiming one is checked against that shape and the error says what it got
wrong. The catch-all cannot join them -- its kind is any string -- so it sits
alongside and is refined to refuse the named kinds, which it used to swallow:
`{kind: 'restore'}` validated as a kind and nothing else, then read back as a
restore that recorded nothing about what it restored.
- `lib/origin/*.js` gains `// @ts-check`. Without it the annotations in there
were decorative: the lazily required subclasses were `any`, so nothing handed
to them was checked, and `fromRaw` narrows with `in` to drop the catch-all
variant, which a kind check cannot.
The callers that stamp the id -- real-time's handshake flag, web's applyChange
body -- come with the rest of the branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [editor-core] recognise a change a writer that is not an editor placed
The counterpart of `isSameEditorChange` for writers that are not an editor
client: an integration that commits a batch on a user's behalf has to recognise
that batch coming back, because a lost response does not say whether the commit
landed and resubmitting one that did applies its removes twice.
It compares the same three fields, and for the same reason -- a rebase rewrites a
change's operations but leaves its origin, authors and timestamp alone -- with
the timestamp compared as a point in time rather than as a string, so that two
spellings of the same instant cannot read as different changes. github-sync is
the first caller, and comes with the rest of the branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [editor-core] load an editable file's content, or say it has none
Only an editable file has content: any other kind is bytes addressed by hash, and
hollow data records the size of content it does not have. Both answer
`getContent()` with null, so every caller that wants the content of a doc loaded
it, checked the data's class, and reached for its own error when the check failed.
`File#loadEager` does that once and answers the data that always has the content.
`buildSetContentOperations` takes a `ReadWriteBlobStore` -- everything but
looking a blob up by hash, which is all it takes to build operations over content
-- so a caller holding a store that cannot answer `getBlob` can still use it. Its
metadata argument is `FileMetadata` rather than `Object`, which is what the schema
already says a file's metadata is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [editor-core] say that a pair of operations that do not fit is unprocessable
compose and transform threw a bare Error when the two operations do not describe
the same content. Everything that answers a client for these -- history-v1's
import routes, the git-bridge push, project-history's resync -- decides on
UnprocessableError, so a bare Error reached them as an internal failure: a 500,
which reads as worth retrying, for a pair that fails the same way every time.
A client that resubmits on a 500 resubmits forever.
The guards that cannot be reached through the API keep their bare Error: a
TextOperation composed with something that is not one, and the branch whose
message says it should not happen.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* [history-v1] hash blobs with the rule editor-core holds
storage/lib/blob_hash.js was a bluebird-era implementation of the same git blob
hash editor-core holds, exported from the storage barrel and used across the
storage layer, the api and the tests. All of it reads the shared rule now, and
the test that existed only to check the copy is gone -- editor-core's suite
covers the same non-ASCII case.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
GitOrigin-RevId: 5a0c1e9a06170a0164c01a26a7bb2d205726d531
The hash of empty content is a property of the content rather than of anything
stored: every project's empty file has it, whether or not a blob was ever written
under it, and a file may reference it before anything has. Eight blob stores were
each paying a lookup -- a database query, an HTTP request, a read of a cache that
offline mode may be the only thing standing in for -- to be told what the hash
already says, and clsi had already reinvented the answer on its own.
overleaf-editor-core gains BlobStoreBase, which answers getString for that hash
and delegates the rest to the fetchString its subclasses provide. getObject comes
with it: four stores spelled out the same JSON.parse of getString, and a fifth
wrapped it in a log line.
Adopted by history-v1's BlobStore, BackupBlobStore, HashCheckBlobStore and
recover_zip's RecoveryBlobStore, by project-history's BlobStore, by clsi's
BlobStore, and by web's HistoryBlobStore, SimpleBlobStore and FrontendBlobStore.
history-v1 keeps its own answers in getStream, getBlob and getBlobs, which no
other store implements. Writing and copying a blob are untouched, so a project
that wrote empty content still has a blob for it and a read answers the same
either way.
Two paths could not handle a project holding a tracked change or a comment at
all. Loading such a file needs the ranges blob its rangesHash points at, and
hashing one again needs to write it back, so a store missing getObject or
putObject fails outright:
- recover_zip died on `blobStore.getObject is not a function`, making every such
project unrecoverable. The test covering it asserts a tracked delete is absent
from the zip, which the script already asked for and never reached.
- GET /projects/:project_id/latest/hashed_content returned 500 from the same
cause, then would have hit a missing putObject in snapshot.store.
HashCheckBlobStore becomes a class over the base and gains putObject.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
GitOrigin-RevId: c84f62465beff6cf77ffac26aa772984e1b554cb
* [editor-core] share the pieces the editor, real-time and history-v1 all need
Project-level history-OT has the browser, real-time and history-v1 all reason
about the same changes, so the rules they reason with move into the shared
library rather than being reimplemented on each side.
- `origin/editor_origin.js` plus `EDITOR_ORIGIN_KIND` and its schema entry: the
origin an editor client's change carries, holding the editorId that scopes a
change's timestamp.
- `change_identity.js`: how a submission is recognised when it comes back, so a
resend of a change that already landed is deduplicated rather than applied
twice.
- `rebase.js`: transforming a change onto concurrent ones, in the same order on
both sides, which is what lets the server acknowledge a change with only the
version it landed at.
- `text_file_defaults.js`: the doc-vs-file extension lists, now read from here
by web's settings instead of being spelled out again.
- `blob_string_length.js`: the string length of a blob, used by history-v1's
blob store and by web when it writes an uploaded file into history.
- `file_tree_diff.js`: file-tree operations to a diff, used by
project-history's FileTreeDiffGenerator.
- `Blob.EMPTY_HASH`, so callers can answer for the empty blob without storing
it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* [project-history] build the file-tree diff with the shared helper
FileTreeDiffGenerator's hand-rolled walk over add/edit/move/delete operations
becomes a call to `buildFileTreeDiff` from overleaf-editor-core, so the editor
and project-history read a file-tree change the same way. Adds the unit tests
the decaffeinated original never had.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* [web] read the editable-file lists from overleaf-editor-core
web's settings spelled out the text extensions and editable filenames that
decide whether a file becomes a doc, and every other service that has to make
that call carried its own copy. The lists now live in overleaf-editor-core and
web builds settings.textExtensions and settings.editableFilenames from them,
keeping the ADDITIONAL_TEXT_EXTENSIONS override for Server CE/Pro on top.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* [history-v1] read a blob's string length from overleaf-editor-core
Two copies of getStringLengthOfFile lived here, the one in recover_zip.js
carrying a TODO asking for exactly this. Both now import the shared
implementation, which the blob store keeps re-exporting for backupArchiver.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
GitOrigin-RevId: 31948332f416f56848e3b0dfb5142a4c9f93f508
* [editor-core] gracefully handle no-op TextOperation
* [web] avoid sending sync to pdf requests without a ?file
* [validation-tools] log deeper nested violations
* [editor-core] fix round trip of EditNoOperation
* [web] use custom error for sync to pdf w/o file, fix tests
* [validation-tools] include request params in log msg for violation
* Revert "[editor-core] gracefully handle no-op TextOperation"
This reverts commit c1d37261d5db72b37dbf39428c54edcfb5423f41.
* [web] gracefully handle rejected sync to pdf from context menu
GitOrigin-RevId: 21de7dc96b23575b0d67b6eb0c68ed9c2f86292b
* [editor-core] address zod violation for group_id
* [monorepo] use discriminated union in zod schemas where possible
* [editor-core] v1_source_doc_id is a number
GitOrigin-RevId: 38f591b07cca4e22954bfe30bf855545a6af675d
* [editor-core] extend coverage of zod schemas
* [real-time] adopt strict zod types and extend test coverage
* [project-history] make type check happy
* [web] make type check happy
GitOrigin-RevId: e1a65b9b3560b31ad2af817067a4b549865a5bb1
* [editor-core] avoid duplicate non-bmp check on InsertOp.insertion
* [editor-core] remove non-bmp check from StringFileData
Existing content may contain non-bmp characters. New edits with non-bmp
input are rejected via non-bmp check in InsertOp constructor.
GitOrigin-RevId: e9ef3a4a8c4696e8dc751325608a788351791d1c
* [clsi] initial implementation of compile from history
* [clsi] copy changes
* [saas-e2e] extend test case with nested folder
* [saas-e2e] add test case for tracked changes
* [web] fix accumulating changes from multiple chunks
* [web] optimize size check for compile request payload
* [clsi] deduplicate globalBlobs
* [clsi] add validation for request body details
* [clsi] add metrics for compile from history
* [clsi] download binary files concurrently
* [clsi] skip download of empty file blob
* [clsi] break down e2e compile time metric by compileFromHistory
GitOrigin-RevId: 0dadef93e89d8a172c35cb130a1042d9d1bec42a
* Use snapshot timestamp, remove request for paths at version
* Add timestamp to core Snapshot object
* Avoid mutating function argument
* Explain assumption about editable files
* snapshot.toRaw() in getContentAtVersion
* fix project-history acceptance test
* fix history_v1 test
* fix web tests
* Include the snapshot timestamp in stored chunks
---------
Co-authored-by: Eric Mc Sween <5454374+emcsween@users.noreply.github.com>
GitOrigin-RevId: 376a53e1f927cb56544e6782b47d80345655038c
* Add getDocUpdaterCompatibleRanges utility function
* use history snapshot for file/project restore
* move overleaf-editor-core from devDependencies
GitOrigin-RevId: 62481a5304ada9d931e018418be3c0719bccf1f3
* [history-v1-ot] initial implementation of using doc-level history-v1-ot
* [web] fix advancing of the otMigrationStage
Use 'nextStage' for the user provided, desired stage when advancing.
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [document-updater] document size check in editor-core
* [history-ot] rename history-v1-ot to history-ot and add types
* [history-ot] apply review feedback
- remove extra !!
- merge variable assignment when processing diff-match-match output
- add helper function for getting docstore lines view of StringFileData
Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
* Revert "[document-updater] add safe rollback point for history-ot (#25283)"
This reverts commit d7230dd14a379a27d2c6ab03a006463a18979d06
Signed-off-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
---------
Signed-off-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
GitOrigin-RevId: 89c497782adb0427635d50d02263d6f535b12481
* [project-history] trim logging when projectVersion is out of order
* [editor-core] record reason rather than full pathname when not clean
* [editor-core] remove pathname from error messages
* [project-history] avoid logging updates when historyId is inconsistent
* [project-history] avoid logging failed history-v1 requests repeatedly
* [project-history] log once when doc version is out of order in chunk
GitOrigin-RevId: 5ac431ac87b57c00fac9d5ab486ca305b1775372
