a9a9f6ee6b
* migrate recover_zip_from_backup from archiver to zip-stream Replace the `archiver` package with `zip-stream` (the lower-level library that `archiver` wraps) in the `recover_zip_from_backup.mjs` script and `backupArchiver.mjs` library. The `archiver` package has known issues with hanging when creating large zip files and is no longer actively maintained. Changes: - Add `zip-stream@^7.0.2` as a direct dependency - Update `backupArchiver.mjs` to use promisified `ZipStream.entry()` instead of `Archiver.append()` - Rewrite `recover_zip_from_backup.mjs` to use `ZipStream` with `stream/promises.pipeline` for cleaner async flow - Keep `archiver` dependency for `project_archive.js` (separate code path) Agent-Logs-Url: https://github.com/overleaf/internal/sessions/0df27a8b-97f1-43cc-ac26-f5247a84313f Co-authored-by: briangough <7457354+briangough@users.noreply.github.com> * extract finalize timeout to named constant Agent-Logs-Url: https://github.com/overleaf/internal/sessions/0df27a8b-97f1-43cc-ac26-f5247a84313f Co-authored-by: briangough <7457354+briangough@users.noreply.github.com> * convert recover_zip.js to zip-stream, remove finalize timeout, add verbose logging Agent-Logs-Url: https://github.com/overleaf/internal/sessions/9380d08a-d813-4e9f-a2ac-4891122c163b Co-authored-by: briangough <7457354+briangough@users.noreply.github.com> * add acceptance tests for recover_zip_from_backup in raw and latest modes Agent-Logs-Url: https://github.com/overleaf/internal/sessions/9380d08a-d813-4e9f-a2ac-4891122c163b Co-authored-by: briangough <7457354+briangough@users.noreply.github.com> * fix comment formatting in recover_zip_from_backup.mjs Agent-Logs-Url: https://github.com/overleaf/internal/sessions/9380d08a-d813-4e9f-a2ac-4891122c163b Co-authored-by: briangough <7457354+briangough@users.noreply.github.com> * restore EventEmitter.defaultMaxListeners in recover_zip.js, add acceptance test Agent-Logs-Url: https://github.com/overleaf/internal/sessions/e7443126-22d5-4d0e-a176-a7a5dba49ffd Co-authored-by: briangough <7457354+briangough@users.noreply.github.com> * fix formatting * refactor: simplify stream handling by using named imports for pipeline * fix blob hash verification in backup acceptance tests * fix recover_zip script and tests * fix: exit with non-zero status on error in recover_zip.js Agent-Logs-Url: https://github.com/overleaf/internal/sessions/ef3f109b-488f-47c9-84a5-b5269387166a Co-authored-by: briangough <7457354+briangough@users.noreply.github.com> * migrate from npm to yarn --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: briangough <7457354+briangough@users.noreply.github.com> Co-authored-by: Brian Gough <briangough@users.noreply.github.com> GitOrigin-RevId: 6255f9610f3c846790e2ed8b1979ac08b7effece
Database migrations
The history service uses knex to manage PostgreSQL migrations.
To create a new migrations, run:
npx knex migrate:make migration_name
To apply migrations, run:
npx knex migrate:latest
For more information, consult the knex migrations guide.
Global blobs
Global blobs are blobs that are shared between projects. The list of global blobs is stored in the projectHistoryGlobalBlobs Mongo collection and is read when the service starts. Changing the list of global blobs needs to be done carefully.
Adding a blob to the global blobs list
If we identify a blob that appears in many projects, we might want to move that blob to the global blobs list.
- Add a record for the blob to the projectHistoryGlobalBlobs collection.
- Restart the history service.
- Delete any corresponding project blobs.
Removing a blob from the global blobs list
Removing a blob from the global blobs list is trickier. As soon as the global blob is made unavailable, every project that needs the blob will have to get its own copy. To avoid disruptions, follow these steps:
-
In the projectHistoryGlobalBlobs collection, set the
demotedproperty tofalseon the global blob to remove. This will make the history system write new instances of this blob to project blobs, but still read from the global blob. -
Restart the history service.
-
Copy the blob to all projects that need it.
-
Remove the blob from the projectHistoryGlobalBlobs collection.
-
Restart the history service.