Commit Graph

120 Commits

Author SHA1 Message Date
Rebeka Dekany a648015db8 Centralize prettier configuration to root level (#30501)
* Merge all .prettierignore files into top-level config

* Merge all .prettierrc files into top-level config

* Replace service-specific glob patterns in package.json format scripts with `prettier .`

* Add template files with Jinja2, Go template, envsubst, and Handlebars syntax to .prettierignore

* Ignore GitHub templates

* Ignore PUG templates to format them separately with `format:pug`

* Encourage double quotes for YAML, YML files

* Move prettier for PUG source format script to the root

* Move prettier for styles source format script to the root

* Remove prettier for jenkins files from web

* Remove prettier source format script from all services

* Make .prettierrc more readable

* Update format scripts by file type

* Organise `.prettierignore`

* Add `--cache` flag to prettier scripts for faster runs

* Format all files with prettier

* Format all or format services

* Remove `format`/`format:fix` scripts from services since now it runs from root `package.json`

* Avoid conlficts with yamllint configuration

* Remove `--cache` flag from prettier scripts

* Update all service Makefiles to use root-level prettier configuration

* Update all Jenkinsfile to use root-level prettier configuration

* Ignore auto-generated files by build_scripts

* Update package-lock.json

* Update root Makefile format targets

* Update SP Jenkinsfile format target

* Update E2E Makefile format script

* Udpate `format_js` to work in both local and CI env

* Add docker-mailtrap to .prettierignore

docker-mailtrap is a third-party git-ignored directory used for testing

* Added Docker env detection to prevent nested Docker spawning

* Ignore handlebars templates

* Add cryptographic files and test output to `.prettierignore`

* Add terraform modules to `.gitignore`

* Remove prettier-plugin-groovy

* Use npx directly instead of Docker for local formatting for faster formatting

* Auto-generate Makefiles

* Revert "Remove prettier-plugin-groovy"

This reverts commit 194a33589a2e1e4d2225d10c67e9f025e4222025.

* Mount monorepo root in RUN_LINT_FORMAT for prettier config access

* Prettier ignores all `node_modules` by default regardless of location

* Show only changed files in format output

* Ignore LICENSE files

* Enable prettier on rendered build_scripts outputs

* Ignoring all the template folders by prettier

* Remove the public/minjs entry since it does not exist

* Remove all non-existent paths

* Sync `.prettierignore` with ignored files by `.gitignore` and `.dockerignore` files

* Revert "Auto-generate Makefiles"

This reverts commit c0233e490de1bc95fe437219d65e0b66e0331ec9.

* Revert "Use npx directly instead of Docker for local formatting for faster formatting"

This reverts commit 1d2b2cf1a6c6974c76885852a90dd55e84167e41.

* Ignore dashboard JSON files

* Ignore files generated by bin/update_build_scripts

* Remove unsupported file types from `.prettierignore`

* Ignore test fixture generated files

* Ignore README file types by prettier

* Ignore generate snapshots by prettier

* Allow to format generated bin/update_build_scripts by prettier

* Ensure build script outputs prettier-compatible tsconfig.json

* Fix build script output to match prettier formatting
- Fix Jinja2 whitespace in docker-compose templates
- Change YAML quotes from single to double

* Don't read cryptographic files by prettier

* Ignore google verification files by prettier

* Revert npx prettier formatting

* Ignore domain verification files

* Show only changed files in format output

* Make `.github` prettier

* Allow all files to be formatted in jobs by prettier

* Allow server-ce/server-pro files to be formatted by prettier

* Ignore more folders in clsi, filestory, git-bridge by prettier

* Update build script with `RUN_LINTING_CI_MONOREPO`

* Ignore docker-mailtrap and downloads in server-ce by prettier

* Restore prettier configs and prettierignore for V1 since it has its own prettier (an older version)

* Source format

GitOrigin-RevId: 637adc3cc422d1f20c86d6ebc8ec514d60758287
2026-02-04 09:08:22 +00:00
Jakob Ackermann 18cff6e1ac [object-persistor] import ProjectKey helper from history-v1 (#30600)
GitOrigin-RevId: c72aa4bf91569904a2072c74d6ed2f3c764d97bb
2026-01-13 09:06:55 +00:00
Brian Gough 42d09010aa Merge pull request #30174 from overleaf/bg-history-backup-allow-stalled-jobs-to-be-retried
Allow stalled jobs to be retried in backup queue settings

GitOrigin-RevId: c68605aa89e9f8d4c2bc11fc14739a7d0774096d
2025-12-10 09:06:18 +00:00
Brian Gough b9888957e4 Merge pull request #30000 from overleaf/bg-extend-backup-blob-script
extend backup blob script

GitOrigin-RevId: 226e624f0fd93bfe4890acce728ce8005f9787c5
2025-12-08 09:06:05 +00:00
Brian Gough 7a57ef00cb Merge pull request #30045 from overleaf/bg-object-persistor-make-list-directory-safer
Improve safety of object persistor

GitOrigin-RevId: bced9814de6613b388ca288a5f72cd42cff6c1d3
2025-12-05 09:06:42 +00:00
Brian Gough 06f696ced0 Merge pull request #29980 from overleaf/bg-history-extend-backup-comparison-III
Check file tree hashes in backup comparison

GitOrigin-RevId: 4bd1f36afa34f326d4b8934c8bb0ea00a52cf1d9
2025-12-01 09:06:18 +00:00
Brian Gough e0d6d7cb48 Merge pull request #29952 from overleaf/bg-history-extend-backup-comparison-II
Add batch comparison options for backup verification

GitOrigin-RevId: fe12270b4063c4105f7e7a950c6a58cd39a62843
2025-11-28 09:06:11 +00:00
Brian Gough 70909c126c Merge pull request #29950 from overleaf/bg-history-extend-backup-comparison
Add fast comparison option for blob backups

GitOrigin-RevId: e1383425487ac1b4439248f976e6106567bae07d
2025-11-28 09:06:06 +00:00
Brian Gough 1f356754de Merge pull request #29801 from overleaf/bg-history-refactor-backup-worker-shutdown
refactor history-v1-backup worker shutdown

GitOrigin-RevId: 9666a99b00b30e98844e7dd25932f1590d0879b3
2025-11-24 09:05:49 +00:00
Brian Gough e822ac0ee0 Merge pull request #29775 from overleaf/bg-history-use-configurable-concurrency-in-backup-worker
allow configurable concurrency for history-v1 backup worker

GitOrigin-RevId: 59c734b013f99e215cc84688142cb0fbe45b064b
2025-11-24 09:05:44 +00:00
Brian Gough 581a3b7ce7 Merge pull request #29700 from overleaf/bg-history-v1-log-stalled-uploads
Add timeout warnings for long-running blob and chunk uploads

GitOrigin-RevId: 5a57ebaaea9dfcaf23153f7e6f27190af541eb16
2025-11-18 09:06:02 +00:00
Brian Gough cc7f0c1b9b Merge pull request #29321 from overleaf/bg-ignore-deleted-projects-in-history-backup
handle deleted projects in history-v1 backup worker

GitOrigin-RevId: f4392045074248137f15d082d922c18b1ef9232f
2025-10-28 09:05:34 +00:00
Brian Gough 2cae9c4635 Merge pull request #29097 from overleaf/bg-reduce-retry-backup-script-concurrency
reduce concurrency for retrying failed backups

GitOrigin-RevId: aeb1f05b7ed26b80c00f12fe07a5f30a14786c30
2025-10-16 08:05:56 +00:00
Brian Gough f6229d6fe1 Merge pull request #29032 from overleaf/bg-retry-failed-backups
Add script and cronjobs to retry failed history-v1 backups

GitOrigin-RevId: c32102086235d4e22b947f4a80089b186c093cea
2025-10-14 08:06:28 +00:00
Miguel Serrano 3a35b8680e Merge pull request #28554 from overleaf/msm-force-s3-lib-storage-uploads
[object-persistor] Use `@aws-sdk/lib-storage` for all uploads

GitOrigin-RevId: ab8e54a7bae843f9e6b05ed9cf936130a36b8c2f
2025-10-08 08:05:55 +00:00
Jakob Ackermann b5fc004c0a [history-v1] add script for listing owned projects with size (#28292)
* [history-v1] add script for listing owned projects with size

* [history-v1] handle project with no blobs record

* [history-v1] add plain sumBytes for easy sum in spreadsheet

GitOrigin-RevId: c20af189c223f867a6033cfb904f92a3a10c7cfb
2025-09-05 08:05:33 +00:00
Jakob Ackermann 56f41c77ff [history-v1] add migrations record after full binary file migration (#27932)
* [history-v1] add migrations record after full binary file migration

* [server-pro] add hotfix 5.5.5

* [server-ce] test: build hotfix 5.5.5 and use it in tests

GitOrigin-RevId: fb84e5710c59f466a3305de5f32f78e0ac9ce15d
2025-08-19 08:05:54 +00:00
Jakob Ackermann 1fd4445a34 [history-v1] extend --report with number of files to check/missing (#27397)
* [history-v1] put logs of back_fill_file_hash script in /var/log/overleaf

* [history-v1] do not leave tmp dir behind when using --report

* [history-v1] extend --report with number of files to check/missing

* [history-v1] move all the options into command line flags

Change default values:
- concurrency to 10
- concurrent batches to 1
- stream high water mark to 1MiB

* [web] find_malformed_filetrees: log progress on stderr

* [history-v1] fix percentage of projects that need back-filling

GitOrigin-RevId: ebd7889840f97b24eda43774ffe7524998fb634b
2025-07-30 08:07:12 +00:00
Brian Gough b8e2a8bfdb Merge pull request #27257 from overleaf/bg-filestore-migration-usability
filestore migration usability changes

GitOrigin-RevId: 47e8c8434c35b1b16c41700dfef11ce4602a3063
2025-07-24 08:05:25 +00:00
Jakob Ackermann 81f0807fc6 [web] prepare filestore migration for Server Pro/CE (#27230)
* [web] prepare filestore migration for Server Pro/CE

* [history-v1] remove unused USER_FILES_BUCKET_NAME env var from script

* [server-ce] tests: write default docker-compose.override.yml on startup

* [server-ce] tests: extend access logging of host-admin for response

* [server-ce] tests: test text and binary file upload

* [server-ce] tests: add tests for filestore migration

* [web] simplify feature gate for filestore/project-history-blobs logic

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [server-ce] test: fix flaky test helper

---------

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: f89bdab2749e2b7a49d609e2eac6bf621c727966
2025-07-22 08:06:58 +00:00
Jakob Ackermann bf43d4f709 [history-v1] make back_fill_file_hash_fix_up compatible with Server Pro (#27280)
* [history-v1] move MockFilestore into shared place

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [history-v1] make back_fill_file_hash_fix_up compatible with Server Pro

---------

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: 70ea57e1503031d9f14dcd60c4c110e746450587
2025-07-22 08:06:41 +00:00
Brian Gough 5208ad39ec Merge pull request #27173 from overleaf/bg-filestore-migration-remove-backups
remove backup code from back_fill_file_hash script and tests

GitOrigin-RevId: 364eefc47670e6e7f2314de810ea259b609ff976
2025-07-18 08:06:14 +00:00
Brian Gough 5669a7d1c8 Merge pull request #27147 from overleaf/bg-filestore-migration-for-server-pro-II
add support for fetching files via http from filestore in back_fill_file_hash script and tests

GitOrigin-RevId: 8dea6383ed6fe9ee6786a5695e2deee93b1cdd84
2025-07-17 08:05:30 +00:00
Jakob Ackermann cf472f54d0 [history-v1] use POST requests for expiring redis buffer from cron (#26568)
* [history-v1] use POST requests for expiring redis buffer from cron

(cherry picked from commit 15780ac54e36b96e1aed9fd9eb6dfe9d4fbf842f)

* [history-v1] remove double claim of expire job

GitOrigin-RevId: 8b2eab07006a5819a47eed3f646b2a4d75f86e5b
2025-07-08 08:05:47 +00:00
Andrew Rumble 745043ca92 Add more detail for the modes
GitOrigin-RevId: 44b63b76b1edb8bfb78049c9c8560934e340ef96
2025-06-25 08:06:41 +00:00
Andrew Rumble f2b0a982ac Use BackupBlobStore to get chunk content
GitOrigin-RevId: 91de53101ea289b43bdb08352aecc09ae89d0f74
2025-06-25 08:06:32 +00:00
Andrew Rumble 797f29d40a Handle errors when processing command line arguments
GitOrigin-RevId: 7c1f011e705d6e4b6fbf0afee3dc3aab749f0de4
2025-06-25 08:06:27 +00:00
Andrew Rumble 392037efd6 Improve types and export functions we want to use
GitOrigin-RevId: 25fd93214f88c425bee23d43de113cf754f89524
2025-06-25 08:06:18 +00:00
Andrew Rumble 3600aa4b75 Add script for restoring projects from backup
GitOrigin-RevId: b639d74989afa17638bbcba0f8ee34c978fcd5d9
2025-06-25 08:06:14 +00:00
Eric Mc Sween 1042092144 Merge pull request #26547 from overleaf/em-revert-expiry-post
Revert "use POST requests for expiring redis buffer from cron"

GitOrigin-RevId: 95e9fd1be7b73699d6fac24035437d467c273d0a
2025-06-19 08:06:58 +00:00
Eric Mc Sween 150dfd6cba Merge pull request #26539 from overleaf/jpa-post
[history-v1] use POST requests for expiring redis buffer from cron

GitOrigin-RevId: 51c9a25b998e581ed20c0e113bd4989537a1e6ef
2025-06-19 08:06:50 +00:00
Eric Mc Sween 7bdc4291fc Merge pull request #26533 from overleaf/jpa-queue
[history-v1] use POST requests for flushing history queues

GitOrigin-RevId: ae87a046a7844b25cc123d08ac6c8d1624608394
2025-06-19 08:06:16 +00:00
Eric Mc Sween f68bf5a69f Merge pull request #26477 from overleaf/em-redis-buffer-resync
Handle invalid content hash when persisting changes

GitOrigin-RevId: 5259190396c8c261cad1abcd5de66314c1e871fb
2025-06-18 08:06:17 +00:00
Eric Mc Sween 9cb4ef4d7d Merge pull request #26353 from overleaf/bg-history-redis-extend-persist-worker
extend persist worker to make parallel requests

GitOrigin-RevId: 8def7d5a8b5c9fcbe5fe45ac8f3ace503d31877a
2025-06-17 08:05:55 +00:00
Brian Gough 2f44a4eb5a Merge pull request #26351 from overleaf/bg-history-redis-convert-persist-worker-to-esm
history redis convert persist worker to esm

GitOrigin-RevId: edcbac6e3f1d3dde3fa8239378995f3ff3afcfdd
2025-06-13 08:06:32 +00:00
Brian Gough fdd0d95554 Merge pull request #26293 from overleaf/bg-history-redis-fix-persist-worker
add missing load global blobs from persist worker

GitOrigin-RevId: ae9393f2353fb4d5afe349aa7d0a26bab80c7f53
2025-06-11 08:06:59 +00:00
Brian Gough c23e84eb37 Merge pull request #26273 from overleaf/bg-history-redis-add-persist-worker-to-cron
modify existing run-chunk-lifecycle cron job to persist and expire redis queues

GitOrigin-RevId: afb94b3e2fba7368cfec11997dfd5b2bbd6321a9
2025-06-11 08:06:23 +00:00
Brian Gough 2eb695f4c3 Merge pull request #26122 from overleaf/bg-history-redis-make-persist-buffer-consistent
make persistBuffer export consistent with other methods

GitOrigin-RevId: 24536e521e1d20ef63cc74bd9ba40e095025d512
2025-06-09 08:05:39 +00:00
Brian Gough d280f40885 Merge pull request #26116 from overleaf/bg-history-redis-show-buffer
add script to display redis buffer for a given history ID

GitOrigin-RevId: 71c2e79480c0873d30801ed3c13aa9a7fc7873f6
2025-06-09 08:05:35 +00:00
Brian Gough 393cee7af5 Merge pull request #25993 from overleaf/bg-history-refactor-persist-buffer-limits
refactor persist buffer to add limits

GitOrigin-RevId: 4a40a7a8812acf5bb7f98bfd7b94d81ebe19fc57
2025-06-04 08:06:19 +00:00
Brian Gough 50df3862e9 Merge pull request #25954 from overleaf/bg-history-expire-worker-fix
fix expire_redis_chunks to only clear job on error

GitOrigin-RevId: f7ec435edda95958b453fba501686dcfd84426f7
2025-06-04 08:06:14 +00:00
Brian Gough a80203f748 Merge pull request #25909 from overleaf/bg-history-persist-worker
add history persist worker

GitOrigin-RevId: b9e31e7bdd84570efc0b87b9f5e90b4078551a8c
2025-06-04 08:06:10 +00:00
Jakob Ackermann 11b94593c2 [web] remove deledFiles collection (#25750)
* [history-v1] remove processing of deleted files when back-filling hashes

* [web] remove deledFiles collection

GitOrigin-RevId: 7c080e564f7d7acb33ebe7ebe012f415a847d0df
2025-05-23 08:05:01 +00:00
Brian Gough 0d70223a48 Merge pull request #25482 from overleaf/bg-fix-bull-exporter-errors
retain completed and failed jobs for backup queue

GitOrigin-RevId: 7831ce2565dc493e3ce7f55001207daea2140575
2025-05-13 08:05:40 +00:00
Brian Gough 9e07549ecb Merge pull request #25449 from overleaf/bg-histoy-redis-refactor-expire-worker
refactor the expire worker to make it easier to extend

GitOrigin-RevId: 7b277b243ed51ab3b46316d98b7157af95a9e42b
2025-05-12 08:05:19 +00:00
Eric Mc Sween 391fca9e83 Merge pull request #25361 from overleaf/em-load-latest-raw
Rename loadLatestRaw() to getLatestChunkMetadata()

GitOrigin-RevId: e089dcfa57cbbc43df8666b51eca0d81d595a5a7
2025-05-09 08:06:14 +00:00
Eric Mc Sween 5717ea7f5c Merge pull request #25306 from overleaf/em-redis-buffer-read-operations
Add changes from Redis when reading chunks from the chunk store

GitOrigin-RevId: c0ebf0669b91eb2efc5d1091d025e81efdff9fe4
2025-05-09 08:06:09 +00:00
Brian Gough f9b36cd5be Merge pull request #25241 from overleaf/bg-remove-existing-chunk-buffer
remove existing chunk redis backend and chunk buffer

GitOrigin-RevId: 28fb02d1802312de6892e2fb7dd59191e3fc8914
2025-05-08 08:05:09 +00:00
Brian Gough 850da34778 Merge pull request #25086 from overleaf/bg-history-buffer-use-expire-time
add expire time to redis buffer in history-v1

GitOrigin-RevId: 3d74957c341e62e536dc60869a7ca71ac173e380
2025-04-29 08:05:14 +00:00
Brian Gough 626416ed02 Merge pull request #24945 from overleaf/bg-redis-buffer-stats-script
add script for listing buffer stats from redis

GitOrigin-RevId: 7836563d51a5c6ded264d0e709d5cfcda70596e3
2025-04-24 08:05:49 +00:00