Commit Graph

245 Commits

Author SHA1 Message Date
Brian Gough
89898b0e9a move migrations to shared location (#28306)
* fix: correct typedef for Document in helpers.mjs

* add move-migrations codemod

* update migration paths to use shared migrations directory

* move migrations to shared location

* fix: update Dockerfile and docker-compose.ci.yml to include migrations directory

* feat: add migrations tool to workspaces in package.json

* [monorepo] Fix order of docker ignore rules

* [web] remove unused docker ignore file

* [monorepo] replace old references to migrations folder

* [server-ce] copy migrations from new place

* [migrations] Inline web scripts

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [migrations] move three web scripts over

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [migrations] add missing collection

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [migrations] remove lodash dependency

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [migrations] avoid mongodb-legacy dependency

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [monorepo] run migrations from tools/migrations

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [migrations] simplify migration for adding gitBridge feature to users

* [monorepo] run migrations from tests in all the services

* [migrations] add Jenkins pipeline for linting/formatting

* [monorepo] fixup running web migrations everywhere

* [monorepo] trigger Jenkins builds on changes to mongo migrations

* [migrations] add Jenkins pipeline for linting/formatting

* [monorepo] build scripts: update devDependencies before deps scanning

* [monorepo] build scripts: formerly depend on tools/migrations

* [monorepo] run eslint on .mjs files

* [migrations] enable more eslint rules and fix all the errors

* [rake] fix migrations:list task

---------

Co-authored-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
GitOrigin-RevId: 14cf69cc1b9405bbc75adbb9a000e555500e0614
2025-10-16 08:07:37 +00:00
Jakob Ackermann
81c50f5279 [monorepo] compute IMAGE_CACHE lazily (#29087)
GitOrigin-RevId: de287c89c1cea858d0609265945266be19d7ea26
2025-10-15 08:07:47 +00:00
Jakob Ackermann
79eabd511e [monorepo] use content hash as docker image cache key (#29046)
* [monorepo] remove docker-repos flag from build scripts

* [monorepo] use content hash as docker image cache key

* [packer] jenkins-worker: populate build cache for all services

* [v1] adopt smarter docker caching

* [latexqc] adopt smarter docker caching

* [monorepo] refresh docker cache in Jenkins

* [packer] jenkins-worker: increase disk size

* [monorepo] run jenkins_docker_build_cache as Jenkins user for v1

* [monorepo] define MONOREPO at the top of all the Makefiles

* [monorepo] add --build-arg BUILDKIT_INLINE_CACHE=1 everywhere

GitOrigin-RevId: 8110c6e68727a52ea80f3ec71711d30e17d69499
2025-10-15 08:06:41 +00:00
Jakob Ackermann
edba4eefa4 [monorepo] migrate build scripts to Jenkins (#29019)
* [monorepo] migrate build scripts to Jenkins

* [fraud-dash] fix integration with monorepo

GitOrigin-RevId: 6a30ec5ea68a1eae9b663faf9ccdc741ec7f9cf8
2025-10-13 08:07:11 +00:00
Tim Down
865ce8c3ff Merge pull request #28541 from overleaf/td-notifications-app-ts
Change notifications app.js to TypeScript

GitOrigin-RevId: cb6195e2e8c8cd89e1a954bfcb1911929440d6ca
2025-10-13 08:05:36 +00:00
Domagoj Kriskovic
f3581c9743 Promisify ProjectHistoryClient, ProjectHistoryApp, SyncTests and SendingUpdatesTests (#28890)
GitOrigin-RevId: 7bf26c6ed1a172c6506449a821d4e43f424a72bd
2025-10-08 08:06:00 +00:00
Domagoj Kriskovic
ab0893fb12 Skip checking status code for fetchJson
GitOrigin-RevId: 1470fdd5aabde3448c5ac5ed8e9203fc509b13e8
2025-10-07 08:05:43 +00:00
Domagoj Kriskovic
552464a6ed Promisify ReadingASnapshotTests
GitOrigin-RevId: 281b3a84466175a38b5ebbf9282b4c996e04d221
2025-10-07 08:05:39 +00:00
Domagoj Kriskovic
3c55ed34ee Promisify LatestSnapshotTests
GitOrigin-RevId: 40b0811e25ac9ee75c295c89cec7ad07570007c4
2025-10-07 08:05:34 +00:00
Domagoj Kriskovic
da591924b0 promisify LabelsTests and SummarisedUpdatesTests
GitOrigin-RevId: 32552c8c63ebbd8e73e8c179fcce67b3b955251d
2025-10-07 08:05:30 +00:00
Domagoj Kriskovic
cba45f64fc Promisify SummarisedUpdatesTests
GitOrigin-RevId: 65782b0cf6e552425d6fad716f74f8f3f22cf7e9
2025-10-07 08:05:26 +00:00
Domagoj Kriskovic
d1ff6e13fa promisify GetChangesInChunkSince
GitOrigin-RevId: 82af36d7cb0301e3bf9527b219c67ceb6fc9b34f
2025-10-07 08:05:21 +00:00
Domagoj Kriskovic
b471f5df2d promisidy FlushManagerTests
GitOrigin-RevId: fa26499be5e40d6741ea1d012188a0a70f519b51
2025-10-07 08:05:17 +00:00
Domagoj Kriskovic
8f3d0677c3 promisify FileTreeDiffTests
GitOrigin-RevId: f7c9c119f28bdbf98c526d767ee06286c33ecfa2
2025-10-07 08:05:12 +00:00
Domagoj Kriskovic
36b28fee8d convert some helper functions in ProjectHistoryClient
GitOrigin-RevId: a73207c4dfbf13cd96456e024adfa6a046002e00
2025-10-03 08:06:31 +00:00
Domagoj Kriskovic
226ac6d881 promisify DiscardingUpdatesTests
GitOrigin-RevId: ea64c162bc05d7f30cf95cc397991d7f448c26e2
2025-10-03 08:06:26 +00:00
Domagoj Kriskovic
760ad3b8fa promisify DiffTests
GitOrigin-RevId: 688f0f5f52917483912ee69779381d1ea2469ad8
2025-10-03 08:06:21 +00:00
Domagoj Kriskovic
64e5f31a4c promisify RetryTests
GitOrigin-RevId: 8e748a63bd523087d023be5573e8e23ef3bda3f2
2025-10-03 08:06:16 +00:00
Domagoj Kriskovic
17ff8f3f76 promisify DeleteProjectTests
GitOrigin-RevId: a86e6962253bbf4adf5fc9748e7accb9084b31f9
2025-10-03 08:06:11 +00:00
Domagoj Kriskovic
60e0f6f703 delete unused helper
GitOrigin-RevId: ff7d96159e8c71b2a5ebed4041ee44f5cf45ceb1
2025-10-03 08:06:06 +00:00
Domagoj Kriskovic
92933002e9 use async/await HealthCheckTests, promisify helpers
GitOrigin-RevId: 1016296695eb1c09d74d6f50c765438b83df1024
2025-10-03 08:06:01 +00:00
Jakob Ackermann
d1b531d2f2 [monorepo] migrate monorepo checks to Jenkins (#28765)
GitOrigin-RevId: ebc3db95cebe99226f8de0f66c1830eb5d78e26c
2025-10-01 08:06:28 +00:00
Andrew Rumble
0b3b97e93d [monorepo] format ts files (#28678)
* Add ts files to format package script

* Apply changes to service package.json files

* [monorepo] only support plain .ts files in all the services

---------

Co-authored-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
GitOrigin-RevId: 882cf274da86a0dbe1fff395441f6f2458c1405c
2025-09-25 08:05:30 +00:00
Eric Mc Sween
0015f2cc6c Merge pull request #28672 from overleaf/em-reapply-restore-optimization
Reapply file restore optimization

GitOrigin-RevId: c4479141d890e3e0af746f406a0186bf21f3d918
2025-09-25 08:05:25 +00:00
Jakob Ackermann
4febc27b05 [monorepo] optimize Jenkins build triggers for services (#28477)
Co-authored-by: Andrew Rumble <andrew.rumble@overleaf.com>
Co-authored-by: Mathias Jakobsen <mathias.jakobsen@overleaf.com>
GitOrigin-RevId: ea7d6c435d4a81a0f42ea9b608a0cc79087cdffd
2025-09-24 08:06:43 +00:00
Domagoj Kriskovic
f2502a5143 Revert "File restore optimization - using snapshot timestamp and file paths (#28546)"
This reverts commit 376a53e1f927cb56544e6782b47d80345655038c.

GitOrigin-RevId: 8e4ab1d4042fec8df1b62ab1c5418d873dc6f5eb
2025-09-24 08:06:06 +00:00
Domagoj Kriskovic
fa163bd02c File restore optimization - using snapshot timestamp and file paths (#28546)
* Use snapshot timestamp, remove request for paths at version

* Add timestamp to core Snapshot object

* Avoid mutating function argument

* Explain assumption about editable files

* snapshot.toRaw() in getContentAtVersion

* fix project-history acceptance test

* fix history_v1 test

* fix web tests

* Include the snapshot timestamp in stored chunks

---------

Co-authored-by: Eric Mc Sween <5454374+emcsween@users.noreply.github.com>
GitOrigin-RevId: 376a53e1f927cb56544e6782b47d80345655038c
2025-09-24 08:06:02 +00:00
Andrew Rumble
9d03b59798 Merge pull request #28482 from overleaf/ar-move-notifications-changes-into-buildscripts
[monorepo] move notifications changes into buildscripts

GitOrigin-RevId: b5948702627d0bc87bffd3acc3d7c2ab9f4ac23d
2025-09-23 08:05:53 +00:00
Domagoj Kriskovic
55edbe6606 Use history snapshot when doing file/project restore (#28502)
* Add getDocUpdaterCompatibleRanges utility function

* use history snapshot for file/project restore

* move overleaf-editor-core from devDependencies

GitOrigin-RevId: 62481a5304ada9d931e018418be3c0719bccf1f3
2025-09-19 08:07:58 +00:00
Jakob Ackermann
19b3ded887 [monorepo] run prettier on Jenkinsfiles (#28565)
* [monorepo] consolidate .editorconfig files at the root

* [monorepo] run prettier on Jenkinsfiles

* [saas-e2e] increase timeout for XeTeX compile

GitOrigin-RevId: 48aa82f7c81611899837753ae92c7732998d4ca5
2025-09-19 08:07:49 +00:00
Miguel Serrano
3ef17cb1b6 Upgrade aws-sdk to v3 (re-attempt) (#28410)
* Upgrade `aws-sdk` to v3

Also vendored nodemailer-ses-transport

* Moved default region to CE settings

* Ensure timeout is added to `requestHandler` and `region` is populated

* fix unsigned header in signed URL request

The x-amz-acl header is not needed when using the signed request,
as ACL are already defined when creating the signed URL in
PutObjectCommand constructor.

* Add default AWS region for latexqc service

* remove unnecessary region in compose files

* Use AWS_REGION=us-west-2 for history-v1

* prevent retries uploading streams with PutObjectCommand

* Remove AWS SDK JS message suppression

GitOrigin-RevId: 6fda6f02160023ffed76143397bbd965f86a9509
2025-09-18 08:05:22 +00:00
Miguel Serrano
208b7ccde9 Revert "Merge pull request #27679 from overleaf/msm-aws-sdk-upgrade" (#28151)
This reverts commit 4989ae920d8b7fd9e79623947b7c40bcc2e56d92.

GitOrigin-RevId: 541d95bfeaa7ce820e8af67f646f013fe4fe5d21
2025-08-28 08:05:52 +00:00
Miguel Serrano
0fa25c54dd Merge pull request #27679 from overleaf/msm-aws-sdk-upgrade
Upgrade `aws-sdk` to v3

GitOrigin-RevId: 4989ae920d8b7fd9e79623947b7c40bcc2e56d92
2025-08-28 08:05:36 +00:00
Eric Mc Sween
df1be93ec3 Merge pull request #27936 from overleaf/em-unit-tests-mongo
Make Mongo available to unit tests in all services

GitOrigin-RevId: b65bbb69883d5bba31d09802b89f35bdc523fe4d
2025-08-25 08:05:19 +00:00
Andrew Rumble
c4ef5cf262 [monorepo] Upgrade to node 22.18.0 (#27977)
* Upgrade to node 22.18.0

* Switch projects-data fixture to strip-types compatible form

GitOrigin-RevId: 4a7ec56f4a82395d2be8870526df0c679de38ab5
2025-08-22 08:05:17 +00:00
Jakob Ackermann
2e3ebabb88 [monorepo] add workaround for flaky mongo db init (#27833)
* [monorepo] add workaround for flaky mongo db init

Upstream issue: https://github.com/docker-library/mongo/issues/730
Upstream PR: https://github.com/docker-library/mongo/pull/731

* [server-ce] revert entrypoint changes

* [monorepo] fix typo in path

* [server-ce] run E2E tests from internal monorepo in cloudbuild

* [monorepo] avoid repeated changes to mongo entrypoint

GitOrigin-RevId: 617db9caee1600c8a506a7a147723a306e57ded3
2025-08-14 08:05:58 +00:00
Jakob Ackermann
07eb75fb33 [monorepo] extend cleanup in Jenkins VMs (#27793)
* [monorepo] extend docker compose cleanup

* [web] cleanup docker images

Only retain the branch builds for dependencies.

* [server-ce] jenkins: remove build specific image

Retain the branch image as cache.

* [monorepo] jenkins: prune untagged docker images and docker build cache

* [monorepo] jenkins: remove temporary files from monorepo

* [monorepo] jenkins: remove containers from sandboxed compiles

GitOrigin-RevId: c847015986a060c5c84c39cd06236fd73bbf6f3a
2025-08-12 08:06:27 +00:00
Jakob Ackermann
8e8183b8f0 [monorepo] tweaks for Jenkins/CI (#27761)
* [packer] jenkins-worker: install htop and lsof

* [web] trim tailing space in Jenkinsfile

* [web] cleanup after finishing Jenkins pipeline

* [server-ce] test: cleanup after finishing Jenkins pipeline

* [saas-e2e] add timeout to waiting for rail/web startup

* [monorepo] add 60s timeout for waiting on mongo to startup

* [saas-e2e] cleanup after finishing Jenkins pipeline

GitOrigin-RevId: 9954ab17b4db6a95c444e8e0b55cebb8d5daae59
2025-08-11 08:06:31 +00:00
Alf Eaton
cc6bb54226 Remove "version" from docker-compose files (#27494)
GitOrigin-RevId: 8d9a0e58e3a9c6c4296e4585115d1b259e26a6cd
2025-07-31 08:05:10 +00:00
Christopher Hoskin
5b5afeb69e Merge pull request #27255 from overleaf/revert-27252-revert-26843-csh-issue-26608-mongo8-dev-ci
Revert "Revert "Upgrade the dev environment and CI to mongo 8""

GitOrigin-RevId: 5074b012504e65240017f1fde9b0d8d04c7b8b61
2025-07-22 08:05:25 +00:00
Christopher Hoskin
87ed52e07b Merge pull request #27252 from overleaf/revert-26843-csh-issue-26608-mongo8-dev-ci
Revert "Upgrade the dev environment and CI to mongo 8"

GitOrigin-RevId: f2145812a5c1cf8d3d3ac31c76cc4aed4ea9d46d
2025-07-21 08:05:01 +00:00
Christopher Hoskin
6522ff2bd1 Merge pull request #26843 from overleaf/csh-issue-26608-mongo8-dev-ci
Upgrade the dev environment and CI to mongo 8

GitOrigin-RevId: 5fc33863094ea2bdb9235372efdc7d0cb492c34a
2025-07-21 08:04:57 +00:00
Andrew Rumble
907bc8d3ca Update node to 22.17.0
GitOrigin-RevId: 7a15f7a0e95e3331c589955aca1fbc5dfb13a568
2025-07-09 08:06:01 +00:00
Jakob Ackermann
dc9e7bd12d [project-history] add support for resync of history-ot ranges (#26475)
* [project-history] add support for resync of history-ot ranges

* [project-history] avoid compressing sharejs and history-ot upgrades

* [document-updater] improve error message of some assertions

... by migrating the assertions like this:
```diff
-stub.calledWith().should.equal(true)
+stub.should.have.been.calledWith()
```
```diff
-stub.called.should.equal(false)
+stub.should.not.have.been.called
```

* [document-updater] move content field in resyncDocContent

* [document-updater] add support for resync of history-ot ranges

GitOrigin-RevId: e6104686a26934a5f25a8f095cbe00c163fbbaa7
2025-06-19 08:05:18 +00:00
Jakob Ackermann
53d8a343d2 [misc] make log level configurable when running tests in ESM services (#26465)
GitOrigin-RevId: 153fa99731b2fbff6dfd5edc80ce3e8acf9fc5ce
2025-06-17 08:06:04 +00:00
Jakob Ackermann
68b79ddb5d [project-history] migrate retry_failures script to ESM (#26444)
GitOrigin-RevId: 425e0e6c0a61ef7f8c8c3e07f2d16089f594eb32
2025-06-16 08:07:08 +00:00
Jakob Ackermann
c5f8c0a5e9 [project-history] fix unit tests after merging multiple PRs (#26442)
GitOrigin-RevId: 9ccbadaa807da2f20010930477724e9370bf103e
2025-06-16 08:06:54 +00:00
Jakob Ackermann
e41a03c3b9 [project-history] add script for retrying failures (#26427)
GitOrigin-RevId: 265cec9719825613de01d7f476b20203bbff8d7f
2025-06-16 08:06:41 +00:00
Jakob Ackermann
55b5216a02 [project-history] fix callback signature when processing no updates (#26420)
* [project-history] fix tests and cover callback for processing updates

The before setup was hiding that some tests were not doing what the
assertions were expecting.

* [project-history] fix callback signature when processing no updates

GitOrigin-RevId: 4fa14d47b9a1afd998316b0c9024d49760785a47
2025-06-16 08:06:36 +00:00
Jakob Ackermann
34e948a12b [project-history] add metrics for compression of updates (#26307)
* [project-history] add metrics for compression of updates

* [project-history] sample compression metrics

GitOrigin-RevId: 1cd67dc4ec7b44285afb436c62392b464f007f97
2025-06-16 08:06:28 +00:00