mirror of
https://github.com/yu-i-i/overleaf-cep.git
synced 2026-11-12 13:13:45 +01:00
Jakob AckermannandClaude Opus 5
04871d6049
* [editor-core] share the blob hash and the doc-vs-file rule The git blob hash and the rule that decides whether a file is a doc or a binary file both have to reach the same answer in every service that writes to history, so they move into overleaf-editor-core rather than being spelled out again on each side. blob_utils takes over lib/blob_string_length.js: hashing and the string-length rule answered two halves of the same question -- what history needs to know about content before it can store it -- and each carried the same warning about staying off index.js, which is bundled for the browser. One module states it once, and blobForFile joins them, describing content on disk as the blob it amounts to. The hashing functions are flat rather than a namespace of their own: in a module that also holds the string-length rule, a bare fromStream says nothing about what it makes. history-v1's blob store and recover_zip follow the module to its new name. file_type_detector holds the doc-vs-file rule, Node-only and deep-imported for the same reason. It takes textExtensions, editableFilenames and maxDocLength as configuration rather than reading settings: the limit is web's 2 MiB and not editor-core's own 3 MiB, and the surrogate check covers both halves of a pair where editor-core's covers the high half. buildSetContentOperations' test spelled the blob header out for itself, with the separator as a literal NUL byte, which had git reading the file as binary. It hashes through blobHashFromString now, so it does not spell the header at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [editor-core] name the client a change came from on the origin itself `EditorOrigin` carried an `editorId` that only an editor change could have. What the id is for -- recognising a change again, so history does not apply a resend twice and the client does not apply its own change back -- is a property of the client that submitted it rather than of what it did, so it moves onto `Origin` as an optional `historyClientId` and `EditorOrigin` goes. - `Origin` takes `(kind, historyClientId)` and answers `getHistoryClientId()`; `dropHistoryClientId()` is what the chunk store uses to strip the id from all but a client's latest change, which now leaves the rest of the origin alone -- a restore that a client submitted stays a restore. - `toRaw` is built a layer at a time: the base writes the kind and the id, a subclass spreads that and adds the fields it owns. - The named kinds are their own schemas and discriminate on `kind`, so a payload claiming one is checked against that shape and the error says what it got wrong. The catch-all cannot join them -- its kind is any string -- so it sits alongside and is refined to refuse the named kinds, which it used to swallow: `{kind: 'restore'}` validated as a kind and nothing else, then read back as a restore that recorded nothing about what it restored. - `lib/origin/*.js` gains `// @ts-check`. Without it the annotations in there were decorative: the lazily required subclasses were `any`, so nothing handed to them was checked, and `fromRaw` narrows with `in` to drop the catch-all variant, which a kind check cannot. The callers that stamp the id -- real-time's handshake flag, web's applyChange body -- come with the rest of the branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [editor-core] recognise a change a writer that is not an editor placed The counterpart of `isSameEditorChange` for writers that are not an editor client: an integration that commits a batch on a user's behalf has to recognise that batch coming back, because a lost response does not say whether the commit landed and resubmitting one that did applies its removes twice. It compares the same three fields, and for the same reason -- a rebase rewrites a change's operations but leaves its origin, authors and timestamp alone -- with the timestamp compared as a point in time rather than as a string, so that two spellings of the same instant cannot read as different changes. github-sync is the first caller, and comes with the rest of the branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [editor-core] load an editable file's content, or say it has none Only an editable file has content: any other kind is bytes addressed by hash, and hollow data records the size of content it does not have. Both answer `getContent()` with null, so every caller that wants the content of a doc loaded it, checked the data's class, and reached for its own error when the check failed. `File#loadEager` does that once and answers the data that always has the content. `buildSetContentOperations` takes a `ReadWriteBlobStore` -- everything but looking a blob up by hash, which is all it takes to build operations over content -- so a caller holding a store that cannot answer `getBlob` can still use it. Its metadata argument is `FileMetadata` rather than `Object`, which is what the schema already says a file's metadata is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [editor-core] say that a pair of operations that do not fit is unprocessable compose and transform threw a bare Error when the two operations do not describe the same content. Everything that answers a client for these -- history-v1's import routes, the git-bridge push, project-history's resync -- decides on UnprocessableError, so a bare Error reached them as an internal failure: a 500, which reads as worth retrying, for a pair that fails the same way every time. A client that resubmits on a 500 resubmits forever. The guards that cannot be reached through the API keep their bare Error: a TextOperation composed with something that is not one, and the branch whose message says it should not happen. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * [history-v1] hash blobs with the rule editor-core holds storage/lib/blob_hash.js was a bluebird-era implementation of the same git blob hash editor-core holds, exported from the storage barrel and used across the storage layer, the api and the tests. All of it reads the shared rule now, and the test that existed only to check the copy is gone -- editor-core's suite covers the same non-ASCII case. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> GitOrigin-RevId: 5a0c1e9a06170a0164c01a26a7bb2d205726d531
Database migrations
The history service uses knex to manage PostgreSQL migrations.
To create a new migrations, run:
npx knex migrate:make migration_name
To apply migrations, run:
npx knex migrate:latest
For more information, consult the knex migrations guide.
Global blobs
Global blobs are blobs that are shared between projects. The list of global blobs is stored in the projectHistoryGlobalBlobs Mongo collection and is read when the service starts. Changing the list of global blobs needs to be done carefully.
Adding a blob to the global blobs list
If we identify a blob that appears in many projects, we might want to move that blob to the global blobs list.
- Add a record for the blob to the projectHistoryGlobalBlobs collection.
- Restart the history service.
- Delete any corresponding project blobs.
Removing a blob from the global blobs list
Removing a blob from the global blobs list is trickier. As soon as the global blob is made unavailable, every project that needs the blob will have to get its own copy. To avoid disruptions, follow these steps:
-
In the projectHistoryGlobalBlobs collection, set the
demotedproperty tofalseon the global blob to remove. This will make the history system write new instances of this blob to project blobs, but still read from the global blob. -
Restart the history service.
-
Copy the blob to all projects that need it.
-
Remove the blob from the projectHistoryGlobalBlobs collection.
-
Restart the history service.