Commit Graph

394 Commits

Author SHA1 Message Date
Jakob Ackermann 18cff6e1ac [object-persistor] import ProjectKey helper from history-v1 (#30600)
GitOrigin-RevId: c72aa4bf91569904a2072c74d6ed2f3c764d97bb
2026-01-13 09:06:55 +00:00
Jakob Ackermann 425e7b1e5b [web] enable mongo notablescan in CI (#29501)
* [monorepo] record ERROR/FATAL log messages in junit report

* [web] put SaaS specific code behind feature flag

* [web] use split test cache for getting user assignments

The unit tests needed updating as they did not replicate any of the
 mongo filtering. The acceptance tests cover this logic.

* [web] make better use of existing indexes

* [web] avoid col-scan in tests of notifications module

* [web] remove cleanup of empty feedbacks collection

* [web] add assertion for reason of rejected request in launchpad test

* [web] add missing indexes

* [web] enable mongo notablescan

* [web] make emailNotifications tests compatible with notablescan

GitOrigin-RevId: b888f2feeb3a0e915f068ae1c4ea23ec17821221
2026-01-13 09:06:38 +00:00
Mathias Jakobsen e895ff8153 Merge pull request #30549 from overleaf/mj-build-script-minio-certs
[build-scripts] Handle expired certs in minio

GitOrigin-RevId: 7e0d03112790fbbfd0c2a1718e272aeeb99c8767
2026-01-12 09:06:36 +00:00
Anna Claire Fields 06a0cf2391 Merge pull request #30339 from overleaf/acf-migration5-cleanup
(5) Remove swagger dependencies from history-v1

GitOrigin-RevId: d3d79dd9d3c8ccb59d6a59aeede7186daf18bdb0
2025-12-18 09:05:37 +00:00
Anna Claire Fields 65c164c73d Merge pull request #30053 from overleaf/acf-migration4-controllers-and-params
(4) Update controllers and tests for Zod migration

GitOrigin-RevId: 876fd64f96e1f5d7244ac1d45053c7db9857d46b
2025-12-16 09:06:44 +00:00
Brian Gough 42d09010aa Merge pull request #30174 from overleaf/bg-history-backup-allow-stalled-jobs-to-be-retried
Allow stalled jobs to be retried in backup queue settings

GitOrigin-RevId: c68605aa89e9f8d4c2bc11fc14739a7d0774096d
2025-12-10 09:06:18 +00:00
Andrew Rumble f3c9b2f2f0 Merge pull request #30182 from overleaf/ar/update-jws-to-3.2.3_or_4.0.1
[monorepo] update jws to 4.0.1

GitOrigin-RevId: b3e778b26b2860112fe80220d6e7354520994082
2025-12-10 09:05:25 +00:00
Brian Gough b9888957e4 Merge pull request #30000 from overleaf/bg-extend-backup-blob-script
extend backup blob script

GitOrigin-RevId: 226e624f0fd93bfe4890acce728ce8005f9787c5
2025-12-08 09:06:05 +00:00
Anna Claire Fields 56c98ac99c Merge pull request #29968 from overleaf/acf-migration3-express-routes
(3) Create Express routes (replaces swagger-router)

GitOrigin-RevId: e27973b630509312b198be899148310e69b94777
2025-12-08 09:05:20 +00:00
Brian Gough 7a57ef00cb Merge pull request #30045 from overleaf/bg-object-persistor-make-list-directory-safer
Improve safety of object persistor

GitOrigin-RevId: bced9814de6613b388ca288a5f72cd42cff6c1d3
2025-12-05 09:06:42 +00:00
Anna Claire Fields c1dc70fc92 Merge pull request #29966 from overleaf/acf-migration2-security-middleware
(2) Create Express authentication middleware (replaces swagger-security)

GitOrigin-RevId: 0a15ec7ca601430a1ae00c7f81da456948650a0c
2025-12-04 09:06:06 +00:00
Anna Claire Fields c25e49782f Merge pull request #29965 from overleaf/acf-migration1-validation
Add Zod validation (replaces swagger-validator + swagger-metadata)

GitOrigin-RevId: 2e4ed742e401bdfe49c6f7dc9d0fceeba20cfc7f
2025-12-03 09:06:15 +00:00
Miguel Serrano a3ec5b2797 Merge pull request #30018 from overleaf/msm-fix-esm-import
[web] Fix ESM import

GitOrigin-RevId: 887f4927248241cb3d237e0bdad36b05928657ef
2025-12-03 09:05:32 +00:00
Brian Gough 06f696ced0 Merge pull request #29980 from overleaf/bg-history-extend-backup-comparison-III
Check file tree hashes in backup comparison

GitOrigin-RevId: 4bd1f36afa34f326d4b8934c8bb0ea00a52cf1d9
2025-12-01 09:06:18 +00:00
Brian Gough e0d6d7cb48 Merge pull request #29952 from overleaf/bg-history-extend-backup-comparison-II
Add batch comparison options for backup verification

GitOrigin-RevId: fe12270b4063c4105f7e7a950c6a58cd39a62843
2025-11-28 09:06:11 +00:00
Brian Gough 70909c126c Merge pull request #29950 from overleaf/bg-history-extend-backup-comparison
Add fast comparison option for blob backups

GitOrigin-RevId: e1383425487ac1b4439248f976e6106567bae07d
2025-11-28 09:06:06 +00:00
Gernot Schulz 74efa0e345 Merge pull request #29898 from overleaf/gs-jenkins-issues-all-pipelines
Enable Jenkins hooks for creating GitHub issues for all pipelines with owners

GitOrigin-RevId: ba46db1d3137db12ee5c78f09b126fb9927d9c49
2025-11-27 09:06:04 +00:00
Brian Gough 1f356754de Merge pull request #29801 from overleaf/bg-history-refactor-backup-worker-shutdown
refactor history-v1-backup worker shutdown

GitOrigin-RevId: 9666a99b00b30e98844e7dd25932f1590d0879b3
2025-11-24 09:05:49 +00:00
Brian Gough e822ac0ee0 Merge pull request #29775 from overleaf/bg-history-use-configurable-concurrency-in-backup-worker
allow configurable concurrency for history-v1 backup worker

GitOrigin-RevId: 59c734b013f99e215cc84688142cb0fbe45b064b
2025-11-24 09:05:44 +00:00
Brian Gough 3227502aeb Merge pull request #29718 from overleaf/bg-history-fix-backup-errors
fix backupBlob error handling and add tests for upload failures

GitOrigin-RevId: 3abda1b70012fda20fc403593d0ee6ce04152e24
2025-11-19 09:05:19 +00:00
Brian Gough 581a3b7ce7 Merge pull request #29700 from overleaf/bg-history-v1-log-stalled-uploads
Add timeout warnings for long-running blob and chunk uploads

GitOrigin-RevId: 5a57ebaaea9dfcaf23153f7e6f27190af541eb16
2025-11-18 09:06:02 +00:00
John Lees-Miller 0d71c825a9 Bump fake-gcs version and fix resumable uploads
GitOrigin-RevId: b1f3cf74f59ff53961665e19826b2f10ac235016
2025-11-18 09:05:21 +00:00
Thomas c059a3c5b0 Handle slashes in branch names for docker build tags (#29529)
* Refactor v1 Makefile to use DOCKER_COMPOSE variable for Docker commands with tag safe branch name

* Handle slashes in branch names for docker build tags

GitOrigin-RevId: 463940e8435845978aced745575905f3bfbb8e1c
2025-11-14 09:05:25 +00:00
Gernot Schulz b834b8bcb7 Merge pull request #29537 from overleaf/gs-jenkins-pr-build-tags-templates
Tag images with branch head instead of merge commit (templates)

GitOrigin-RevId: 53e616aeec058863e25b356841a59f3ebe13c87f
2025-11-10 09:05:00 +00:00
Jakob Ackermann 6691c5dc05 [monorepo] enable mongo notablescan for all services but web (#29497)
* [migrations] make old migrations compatible with --notablescan in mongo

Re-running the migration for db.users.emails.reversedHostname on all
Server Pro/CE users will not be harmfull.

* [monorepo] enable mongo notablescan for all services but web

GitOrigin-RevId: e2a710c9de08de9ac2aa9df1f3082d5f1c33ea05
2025-11-05 09:06:35 +00:00
Brian Gough cc7f0c1b9b Merge pull request #29321 from overleaf/bg-ignore-deleted-projects-in-history-backup
handle deleted projects in history-v1 backup worker

GitOrigin-RevId: f4392045074248137f15d082d922c18b1ef9232f
2025-10-28 09:05:34 +00:00
Brian Gough 729e0f5ac9 move migrations to shared location (#28306)
* fix: correct typedef for Document in helpers.mjs

* add move-migrations codemod

* update migration paths to use shared migrations directory

* move migrations to shared location

* fix: update Dockerfile and docker-compose.ci.yml to include migrations directory

* feat: add migrations tool to workspaces in package.json

* [monorepo] Fix order of docker ignore rules

* [web] remove unused docker ignore file

* [monorepo] replace old references to migrations folder

* [server-ce] copy migrations from new place

* [migrations] Inline web scripts

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [migrations] move three web scripts over

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [migrations] add missing collection

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [migrations] remove lodash dependency

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [migrations] avoid mongodb-legacy dependency

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [monorepo] run migrations from tools/migrations

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [migrations] simplify migration for adding gitBridge feature to users

* [monorepo] run migrations from tests in all the services

* [migrations] add Jenkins pipeline for linting/formatting

* [monorepo] fixup running web migrations everywhere

* [monorepo] trigger Jenkins builds on changes to mongo migrations

* [migrations] add Jenkins pipeline for linting/formatting

* [monorepo] build scripts: update devDependencies before deps scanning

* [monorepo] build scripts: formerly depend on tools/migrations

* [monorepo] run eslint on .mjs files

* [migrations] enable more eslint rules and fix all the errors

* [rake] fix migrations:list task

---------

Co-authored-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
GitOrigin-RevId: 14cf69cc1b9405bbc75adbb9a000e555500e0614
2025-10-16 08:07:37 +00:00
Brian Gough 2cae9c4635 Merge pull request #29097 from overleaf/bg-reduce-retry-backup-script-concurrency
reduce concurrency for retrying failed backups

GitOrigin-RevId: aeb1f05b7ed26b80c00f12fe07a5f30a14786c30
2025-10-16 08:05:56 +00:00
Jakob Ackermann 0182ad0046 [monorepo] compute IMAGE_CACHE lazily (#29087)
GitOrigin-RevId: de287c89c1cea858d0609265945266be19d7ea26
2025-10-15 08:07:47 +00:00
Jakob Ackermann 15a6e8ba04 [monorepo] use content hash as docker image cache key (#29046)
* [monorepo] remove docker-repos flag from build scripts

* [monorepo] use content hash as docker image cache key

* [packer] jenkins-worker: populate build cache for all services

* [v1] adopt smarter docker caching

* [latexqc] adopt smarter docker caching

* [monorepo] refresh docker cache in Jenkins

* [packer] jenkins-worker: increase disk size

* [monorepo] run jenkins_docker_build_cache as Jenkins user for v1

* [monorepo] define MONOREPO at the top of all the Makefiles

* [monorepo] add --build-arg BUILDKIT_INLINE_CACHE=1 everywhere

GitOrigin-RevId: 8110c6e68727a52ea80f3ec71711d30e17d69499
2025-10-15 08:06:41 +00:00
Brian Gough f6229d6fe1 Merge pull request #29032 from overleaf/bg-retry-failed-backups
Add script and cronjobs to retry failed history-v1 backups

GitOrigin-RevId: c32102086235d4e22b947f4a80089b186c093cea
2025-10-14 08:06:28 +00:00
Jakob Ackermann fd0d21fbbd [monorepo] migrate build scripts to Jenkins (#29019)
* [monorepo] migrate build scripts to Jenkins

* [fraud-dash] fix integration with monorepo

GitOrigin-RevId: 6a30ec5ea68a1eae9b663faf9ccdc741ec7f9cf8
2025-10-13 08:07:11 +00:00
Tim Down 7aa66b260c Merge pull request #28541 from overleaf/td-notifications-app-ts
Change notifications app.js to TypeScript

GitOrigin-RevId: cb6195e2e8c8cd89e1a954bfcb1911929440d6ca
2025-10-13 08:05:36 +00:00
Miguel Serrano b6d116e957 Merge pull request #28912 from overleaf/msm-fix-certs-build
Fix `certs` image build

GitOrigin-RevId: 02af78b29915276d55e86001a1cdc4703fc830b5
2025-10-09 08:05:26 +00:00
Miguel Serrano 3a35b8680e Merge pull request #28554 from overleaf/msm-force-s3-lib-storage-uploads
[object-persistor] Use `@aws-sdk/lib-storage` for all uploads

GitOrigin-RevId: ab8e54a7bae843f9e6b05ed9cf936130a36b8c2f
2025-10-08 08:05:55 +00:00
Jakob Ackermann 5fe9d3d6e9 [monorepo] migrate monorepo checks to Jenkins (#28765)
GitOrigin-RevId: ebc3db95cebe99226f8de0f66c1830eb5d78e26c
2025-10-01 08:06:28 +00:00
Andrew Rumble 0200ad7515 [monorepo] format ts files (#28678)
* Add ts files to format package script

* Apply changes to service package.json files

* [monorepo] only support plain .ts files in all the services

---------

Co-authored-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
GitOrigin-RevId: 882cf274da86a0dbe1fff395441f6f2458c1405c
2025-09-25 08:05:30 +00:00
Eric Mc Sween 98069966ba Merge pull request #28672 from overleaf/em-reapply-restore-optimization
Reapply file restore optimization

GitOrigin-RevId: c4479141d890e3e0af746f406a0186bf21f3d918
2025-09-25 08:05:25 +00:00
Jakob Ackermann 9b2fcbe960 [monorepo] optimize Jenkins build triggers for services (#28477)
Co-authored-by: Andrew Rumble <andrew.rumble@overleaf.com>
Co-authored-by: Mathias Jakobsen <mathias.jakobsen@overleaf.com>
GitOrigin-RevId: ea7d6c435d4a81a0f42ea9b608a0cc79087cdffd
2025-09-24 08:06:43 +00:00
Domagoj Kriskovic e9b1c63ed7 Revert "File restore optimization - using snapshot timestamp and file paths (#28546)"
This reverts commit 376a53e1f927cb56544e6782b47d80345655038c.

GitOrigin-RevId: 8e4ab1d4042fec8df1b62ab1c5418d873dc6f5eb
2025-09-24 08:06:06 +00:00
Domagoj Kriskovic 005eba7502 File restore optimization - using snapshot timestamp and file paths (#28546)
* Use snapshot timestamp, remove request for paths at version

* Add timestamp to core Snapshot object

* Avoid mutating function argument

* Explain assumption about editable files

* snapshot.toRaw() in getContentAtVersion

* fix project-history acceptance test

* fix history_v1 test

* fix web tests

* Include the snapshot timestamp in stored chunks

---------

Co-authored-by: Eric Mc Sween <5454374+emcsween@users.noreply.github.com>
GitOrigin-RevId: 376a53e1f927cb56544e6782b47d80345655038c
2025-09-24 08:06:02 +00:00
Brian Gough ff267ca246 Merge pull request #28640 from overleaf/bg-fix-chunk-store-metrics
update metrics for getChangesSinceVersion to include status and hasMore flag

GitOrigin-RevId: 389353425a12d6d7c92aa3ae37d6553fc1c7fc1d
2025-09-23 08:08:09 +00:00
Andrew Rumble 423c2a47e5 Merge pull request #28482 from overleaf/ar-move-notifications-changes-into-buildscripts
[monorepo] move notifications changes into buildscripts

GitOrigin-RevId: b5948702627d0bc87bffd3acc3d7c2ab9f4ac23d
2025-09-23 08:05:53 +00:00
Mathias Jakobsen 6c6048002b Merge pull request #28595 from overleaf/revert-28594-revert-28533-mj-blob-ranges-maxsize
[web+history-v1] Retry: Use range request for binary blobs in project snapshot

GitOrigin-RevId: defc279c9918695135962d65603a2941c5c4b033
2025-09-22 08:05:23 +00:00
Mathias Jakobsen 67d10e96bd Merge pull request #28594 from overleaf/revert-28533-mj-blob-ranges-maxsize
Revert "[web+history-v1] Use range request for binary blobs in project snapshot"

GitOrigin-RevId: 580491611a1b43104846f8509dd1e920f8077c9c
2025-09-22 08:05:15 +00:00
Mathias Jakobsen 6c79ce97cc Merge pull request #28533 from overleaf/mj-blob-ranges-maxsize
[web+history-v1] Use range request for binary blobs in project snapshot

GitOrigin-RevId: e72a2d481bf163b3e85361b42e30e82cf867f1cf
2025-09-22 08:05:04 +00:00
Jakob Ackermann a6e9a5c7e9 [monorepo] run prettier on Jenkinsfiles (#28565)
* [monorepo] consolidate .editorconfig files at the root

* [monorepo] run prettier on Jenkinsfiles

* [saas-e2e] increase timeout for XeTeX compile

GitOrigin-RevId: 48aa82f7c81611899837753ae92c7732998d4ca5
2025-09-19 08:07:49 +00:00
Brian Gough fd82788e61 Merge pull request #28506 from overleaf/bg-size-limits
add size check when cloning project (logging only)

GitOrigin-RevId: 1f56ed80a2d05b28c44fab8532d751ad8e758943
2025-09-18 08:05:52 +00:00
Miguel Serrano f1dc6ec155 Upgrade aws-sdk to v3 (re-attempt) (#28410)
* Upgrade `aws-sdk` to v3

Also vendored nodemailer-ses-transport

* Moved default region to CE settings

* Ensure timeout is added to `requestHandler` and `region` is populated

* fix unsigned header in signed URL request

The x-amz-acl header is not needed when using the signed request,
as ACL are already defined when creating the signed URL in
PutObjectCommand constructor.

* Add default AWS region for latexqc service

* remove unnecessary region in compose files

* Use AWS_REGION=us-west-2 for history-v1

* prevent retries uploading streams with PutObjectCommand

* Remove AWS SDK JS message suppression

GitOrigin-RevId: 6fda6f02160023ffed76143397bbd965f86a9509
2025-09-18 08:05:22 +00:00
Brian Gough 386e133766 Merge pull request #28436 from overleaf/bg-load-changes-directly-from-redis-buffer
load changes directly from redis buffer for getChanges requests

GitOrigin-RevId: 99673c47a137ff4222d331fa88eb6e5103270551
2025-09-12 08:07:35 +00:00