Commit Graph

112 Commits

Author SHA1 Message Date
Domagoj Kriskovic da7569e892 Revert "[project-history] migrate from joi to zod (#30187)" (#30681)
This reverts commit f8396a3caadd536edf64c6f971c77f4c941f000a.

GitOrigin-RevId: 8d66aa15a29216736755c3f86954a4d40aa20a6c
2026-01-13 09:07:22 +00:00
Domagoj Kriskovic 6d3dd3601a [project-history] migrate from joi to zod (#30187)
* Add @overleaf/validation-tools in project-history

* use error handler for zod

* Replace joi validations with zod schema

* fix unit tests

* remove Joi validation middleware

* remove optional when there is a default

* fix tests after checking for objectid

* ran build_scripts

* monorepo npm install

GitOrigin-RevId: f8396a3caadd536edf64c6f971c77f4c941f000a
2026-01-13 09:07:00 +00:00
Jakob Ackermann 425e7b1e5b [web] enable mongo notablescan in CI (#29501)
* [monorepo] record ERROR/FATAL log messages in junit report

* [web] put SaaS specific code behind feature flag

* [web] use split test cache for getting user assignments

The unit tests needed updating as they did not replicate any of the
 mongo filtering. The acceptance tests cover this logic.

* [web] make better use of existing indexes

* [web] avoid col-scan in tests of notifications module

* [web] remove cleanup of empty feedbacks collection

* [web] add assertion for reason of rejected request in launchpad test

* [web] add missing indexes

* [web] enable mongo notablescan

* [web] make emailNotifications tests compatible with notablescan

GitOrigin-RevId: b888f2feeb3a0e915f068ae1c4ea23ec17821221
2026-01-13 09:06:38 +00:00
Brian Gough 729e0f5ac9 move migrations to shared location (#28306)
* fix: correct typedef for Document in helpers.mjs

* add move-migrations codemod

* update migration paths to use shared migrations directory

* move migrations to shared location

* fix: update Dockerfile and docker-compose.ci.yml to include migrations directory

* feat: add migrations tool to workspaces in package.json

* [monorepo] Fix order of docker ignore rules

* [web] remove unused docker ignore file

* [monorepo] replace old references to migrations folder

* [server-ce] copy migrations from new place

* [migrations] Inline web scripts

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [migrations] move three web scripts over

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [migrations] add missing collection

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [migrations] remove lodash dependency

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [migrations] avoid mongodb-legacy dependency

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [monorepo] run migrations from tools/migrations

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [migrations] simplify migration for adding gitBridge feature to users

* [monorepo] run migrations from tests in all the services

* [migrations] add Jenkins pipeline for linting/formatting

* [monorepo] fixup running web migrations everywhere

* [monorepo] trigger Jenkins builds on changes to mongo migrations

* [migrations] add Jenkins pipeline for linting/formatting

* [monorepo] build scripts: update devDependencies before deps scanning

* [monorepo] build scripts: formerly depend on tools/migrations

* [monorepo] run eslint on .mjs files

* [migrations] enable more eslint rules and fix all the errors

* [rake] fix migrations:list task

---------

Co-authored-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
GitOrigin-RevId: 14cf69cc1b9405bbc75adbb9a000e555500e0614
2025-10-16 08:07:37 +00:00
Jakob Ackermann fd0d21fbbd [monorepo] migrate build scripts to Jenkins (#29019)
* [monorepo] migrate build scripts to Jenkins

* [fraud-dash] fix integration with monorepo

GitOrigin-RevId: 6a30ec5ea68a1eae9b663faf9ccdc741ec7f9cf8
2025-10-13 08:07:11 +00:00
Domagoj Kriskovic 267fc5393a Promisify ProjectHistoryClient, ProjectHistoryApp, SyncTests and SendingUpdatesTests (#28890)
GitOrigin-RevId: 7bf26c6ed1a172c6506449a821d4e43f424a72bd
2025-10-08 08:06:00 +00:00
Domagoj Kriskovic 6d84dfddcc Skip checking status code for fetchJson
GitOrigin-RevId: 1470fdd5aabde3448c5ac5ed8e9203fc509b13e8
2025-10-07 08:05:43 +00:00
Domagoj Kriskovic 62ec3f9644 Promisify ReadingASnapshotTests
GitOrigin-RevId: 281b3a84466175a38b5ebbf9282b4c996e04d221
2025-10-07 08:05:39 +00:00
Domagoj Kriskovic aed47ef01b Promisify LatestSnapshotTests
GitOrigin-RevId: 40b0811e25ac9ee75c295c89cec7ad07570007c4
2025-10-07 08:05:34 +00:00
Domagoj Kriskovic 64d1f160a1 promisify LabelsTests and SummarisedUpdatesTests
GitOrigin-RevId: 32552c8c63ebbd8e73e8c179fcce67b3b955251d
2025-10-07 08:05:30 +00:00
Domagoj Kriskovic 9150fc332c Promisify SummarisedUpdatesTests
GitOrigin-RevId: 65782b0cf6e552425d6fad716f74f8f3f22cf7e9
2025-10-07 08:05:26 +00:00
Domagoj Kriskovic c1bfd2192d promisify GetChangesInChunkSince
GitOrigin-RevId: 82af36d7cb0301e3bf9527b219c67ceb6fc9b34f
2025-10-07 08:05:21 +00:00
Domagoj Kriskovic 54190e1040 promisidy FlushManagerTests
GitOrigin-RevId: fa26499be5e40d6741ea1d012188a0a70f519b51
2025-10-07 08:05:17 +00:00
Domagoj Kriskovic 1c2aa6251e promisify FileTreeDiffTests
GitOrigin-RevId: f7c9c119f28bdbf98c526d767ee06286c33ecfa2
2025-10-07 08:05:12 +00:00
Domagoj Kriskovic 632a563ed2 convert some helper functions in ProjectHistoryClient
GitOrigin-RevId: a73207c4dfbf13cd96456e024adfa6a046002e00
2025-10-03 08:06:31 +00:00
Domagoj Kriskovic 5944d20340 promisify DiscardingUpdatesTests
GitOrigin-RevId: ea64c162bc05d7f30cf95cc397991d7f448c26e2
2025-10-03 08:06:26 +00:00
Domagoj Kriskovic 5e03da3b58 promisify DiffTests
GitOrigin-RevId: 688f0f5f52917483912ee69779381d1ea2469ad8
2025-10-03 08:06:21 +00:00
Domagoj Kriskovic c1645ebe2c promisify RetryTests
GitOrigin-RevId: 8e748a63bd523087d023be5573e8e23ef3bda3f2
2025-10-03 08:06:16 +00:00
Domagoj Kriskovic 3840eaa0d8 promisify DeleteProjectTests
GitOrigin-RevId: a86e6962253bbf4adf5fc9748e7accb9084b31f9
2025-10-03 08:06:11 +00:00
Domagoj Kriskovic a9d805ef9b delete unused helper
GitOrigin-RevId: ff7d96159e8c71b2a5ebed4041ee44f5cf45ceb1
2025-10-03 08:06:06 +00:00
Domagoj Kriskovic 854536fcc9 use async/await HealthCheckTests, promisify helpers
GitOrigin-RevId: 1016296695eb1c09d74d6f50c765438b83df1024
2025-10-03 08:06:01 +00:00
Eric Mc Sween 98069966ba Merge pull request #28672 from overleaf/em-reapply-restore-optimization
Reapply file restore optimization

GitOrigin-RevId: c4479141d890e3e0af746f406a0186bf21f3d918
2025-09-25 08:05:25 +00:00
Domagoj Kriskovic e9b1c63ed7 Revert "File restore optimization - using snapshot timestamp and file paths (#28546)"
This reverts commit 376a53e1f927cb56544e6782b47d80345655038c.

GitOrigin-RevId: 8e4ab1d4042fec8df1b62ab1c5418d873dc6f5eb
2025-09-24 08:06:06 +00:00
Domagoj Kriskovic 005eba7502 File restore optimization - using snapshot timestamp and file paths (#28546)
* Use snapshot timestamp, remove request for paths at version

* Add timestamp to core Snapshot object

* Avoid mutating function argument

* Explain assumption about editable files

* snapshot.toRaw() in getContentAtVersion

* fix project-history acceptance test

* fix history_v1 test

* fix web tests

* Include the snapshot timestamp in stored chunks

---------

Co-authored-by: Eric Mc Sween <5454374+emcsween@users.noreply.github.com>
GitOrigin-RevId: 376a53e1f927cb56544e6782b47d80345655038c
2025-09-24 08:06:02 +00:00
Jakob Ackermann adf399fb95 [project-history] add support for resync of history-ot ranges (#26475)
* [project-history] add support for resync of history-ot ranges

* [project-history] avoid compressing sharejs and history-ot upgrades

* [document-updater] improve error message of some assertions

... by migrating the assertions like this:
```diff
-stub.calledWith().should.equal(true)
+stub.should.have.been.calledWith()
```
```diff
-stub.called.should.equal(false)
+stub.should.not.have.been.called
```

* [document-updater] move content field in resyncDocContent

* [document-updater] add support for resync of history-ot ranges

GitOrigin-RevId: e6104686a26934a5f25a8f095cbe00c163fbbaa7
2025-06-19 08:05:18 +00:00
Jakob Ackermann 8b937c91f4 [project-history] fix unit tests after merging multiple PRs (#26442)
GitOrigin-RevId: 9ccbadaa807da2f20010930477724e9370bf103e
2025-06-16 08:06:54 +00:00
Jakob Ackermann 0aa56fbe2c [project-history] fix callback signature when processing no updates (#26420)
* [project-history] fix tests and cover callback for processing updates

The before setup was hiding that some tests were not doing what the
assertions were expecting.

* [project-history] fix callback signature when processing no updates

GitOrigin-RevId: 4fa14d47b9a1afd998316b0c9024d49760785a47
2025-06-16 08:06:36 +00:00
Jakob Ackermann 6f516b25af [project-history] add metrics for compression of updates (#26307)
* [project-history] add metrics for compression of updates

* [project-history] sample compression metrics

GitOrigin-RevId: 1cd67dc4ec7b44285afb436c62392b464f007f97
2025-06-16 08:06:28 +00:00
Jakob Ackermann 64984ee86a [history-ot] flush history for projects with short queues ASAP (#25776)
* [document-updater] flush history for projects with short queues ASAP

* [k8s] document-updater: enable short history queue for history-ot demo

* [project-history] flush history for projects with short queues ASAP

* [project-history] wait for mongo before running acceptance tests

* [k8s] project-history: enable short history queue for history-ot demo

* [project-history] change wait-for-mongo step in tests

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>

---------

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: 3e989c409e8e9887655b35f2659ce0829e61b357
2025-05-23 08:05:35 +00:00
Brian Gough 6eada92966 Merge pull request #25406 from overleaf/em-content-hash-validation-resync
Resync project when content hash validation fails

GitOrigin-RevId: ea9b5a78f89c55276fd67835bc262717bc778e92
2025-05-09 08:05:12 +00:00
Eric Mc Sween fb03fe4d26 Merge pull request #24776 from overleaf/em-project-history-unused-endpoints
Remove unused endpoints in project-history

GitOrigin-RevId: 2940c1c1973177e3200cb78884b307f708fd88c3
2025-04-14 08:05:08 +00:00
Eric Mc Sween 35dc7faab6 Merge pull request #24224 from overleaf/em-resync-on-flush-failure
Immediately attempt a resync when a history flush fails

GitOrigin-RevId: 098a0a7edf55c0ed47c48e0a3c080e3562cdceaa
2025-03-18 09:05:36 +00:00
Jakob Ackermann 0a7b2004d2 [misc] silence logger when running tests (#22243)
* [misc] silence logger when running tests

* [misc] re-enable logging in some tests for scripts

* [misc] make it easy to turn on verbose logging for tests

```
LOG_LEVEL=debug make test_unit
LOG_LEVEL=debug make test_acceptance
```
GitOrigin-RevId: 219bc6d1f9cbdb89ddd7d94742920913ddde4514
2025-02-10 09:06:02 +00:00
Eric Mc Sween 064baf31ad Merge pull request #23478 from overleaf/em-doc-hash-diff
Fix doc hash on delete + insert combination

GitOrigin-RevId: ecea5f8ab01f24a9064be88611256b502500d91f
2025-02-10 09:05:38 +00:00
Eric Mc Sween 39842c9857 Merge pull request #23250 from overleaf/em-project-history-doc-hash
Propagate the doc hash from project-history to history

GitOrigin-RevId: 341df52df41f7a5f8dbebbce53a47f9d5e1f8175
2025-02-04 09:05:03 +00:00
Jakob Ackermann b0c9176634 [project-history] add resync for project structure only (#23271)
* [project-history] add resync for project structure only

* [project-history] block resyncProjectStructureOnly that update docs

* [project-history] tweak test description

GitOrigin-RevId: ce2749566c36e04dab21c26dd60dd75d93c0d4c0
2025-02-03 09:05:39 +00:00
Jakob Ackermann ba60f885a4 [project-history] fetch most recent chunk once when processing updates (#23261)
GitOrigin-RevId: 08ddd1f953b0dbae0541a60b45fec0f88e0a1d06
2025-01-31 09:05:14 +00:00
Jakob Ackermann 72b4083318 [project-history] forward all the optional file attributes during resync (#23260)
Notably, 'createdBlob' was missing. And others like metadata should not
be added if not set.

GitOrigin-RevId: c4a6746c4824d637f7c54b4c68a7025b60c15ff2
2025-01-31 09:05:09 +00:00
Eric Mc Sween 3e4e9b298f Merge pull request #23089 from overleaf/em-fix-ranges-snapshot
Fix edge cases when restoring comments

GitOrigin-RevId: 6ce2426bfb7233a3c0915bcc8c53bf0588702847
2025-01-27 09:05:06 +00:00
Jakob Ackermann 762266cd87 [project-history] block filestore reads from old queue entries (#23096)
GitOrigin-RevId: 9952cb66e542b17a6a3b5e3b2609d53dc8c371fd
2025-01-27 09:04:54 +00:00
Jakob Ackermann 2640d4a566 [project-history] fixes for checking blob exists (#23051)
* [project-history] skip HEAD on blob without file-hash

* [project-history] only 404s indicate a missing blob

* [project-history] error checking for existing blob is fatal

* [project-history] always skip creating blob if already exists

Ignore createdBlob detail after performing check on blob. It might not
 exist on old operations that have been queued.

* [project-history] fix unit tests

* [project-history] migrate test to new fetch call

GitOrigin-RevId: 457963042e9d836647d3c122516329d4dc9b544f
2025-01-24 09:05:09 +00:00
Andrew Rumble e64e69f539 Merge pull request #22652 from overleaf/ar-handle-filestore-404-when-copying-in-saas
[web] Do not do filestore copy in SAAS

GitOrigin-RevId: 83456ca57f706246a86b997a038968aecfbae4c5
2024-12-24 09:04:48 +00:00
Andrew Rumble e6a2caa088 Merge pull request #22421 from overleaf/ar-pass-created-blob-for-filetree-uploads
[web] Pass createdBlob through for file creation everywhere

GitOrigin-RevId: 880bebd0a55e351a6a61555a46e80faef22db7cb
2024-12-12 09:05:02 +00:00
Andrew Rumble ad8026038c Merge pull request #22397 from overleaf/revert-22320-revert-22213-ar-avoid-duplicate-blob-writes
Revert "Revert "Avoid duplicate blob writes""

GitOrigin-RevId: f6a0083c0edf9ede14dfb52dbf1809a843aa5ff4
2024-12-10 09:05:07 +00:00
Andrew Rumble ffa28b1a58 Merge pull request #22320 from overleaf/revert-22213-ar-avoid-duplicate-blob-writes
Revert "Avoid duplicate blob writes"

GitOrigin-RevId: 9f86bcea654cd3fa5f66fbdf42080e7f6e2861a7
2024-12-05 09:04:48 +00:00
Andrew Rumble 40b781eb0a Merge pull request #22213 from overleaf/ar-avoid-duplicate-blob-writes
Avoid duplicate blob writes

GitOrigin-RevId: 5e7a472b235bae5ef84389dbbe791189e951908a
2024-12-04 09:05:08 +00:00
Antoine Clausse 7f48c67512 Add prefer-node-protocol ESLint rule (#21532)
* Add `unicorn/prefer-node-protocol`

* Fix `unicorn/prefer-node-protocol` ESLint errors

* Run `npm run format:fix`

* Add sandboxed-module sourceTransformers in mocha setups

Fix `no such file or directory, open 'node:fs'` in `sandboxed-module`

* Remove `node:` in the SandboxedModule requires

* Fix new linting errors with `node:`

GitOrigin-RevId: 68f6e31e2191fcff4cb8058dd0a6914c14f59926
2024-11-11 09:04:51 +00:00
Jakob Ackermann fab69a16c4 Merge pull request #20979 from overleaf/jpa-handle-error
[project-history] gracefully handle errors when loading Changes

GitOrigin-RevId: 214fadb3c779551ea90f38b0fdf77c58ed5df178
2024-10-15 08:05:15 +00:00
Jakob Ackermann 663f6605d7 Merge pull request #20978 from overleaf/jpa-fix-unit-tests
[project-history] fix unit tests for processing updates

GitOrigin-RevId: ce87296b9b97dcfd7598fbf25b3066b7c145fb6f
2024-10-15 08:05:10 +00:00
Domagoj Kriskovic 0998cc04df [project-history] fix diff view on file revert (#20769)
GitOrigin-RevId: 6c95dab9328ecab636272ad507505f5342ec8709
2024-10-14 11:06:36 +00:00