agw/overleaf-cep
1
0
Fork 0
mirror of https://github.com/yu-i-i/overleaf-cep.git synced 2026-11-15 14:43:46 +01:00
Code Issues Packages Projects Releases Wiki Activity
27,223 Commits 8 Branches 23 Tags
da9da896c246dfdab65733a2c9ba2b762b37276f
Commit Graph
229 Commits
Author SHA1 Message Date
Jakob Ackermann 18cff6e1ac [object-persistor] import ProjectKey helper from history-v1 (#30600) 
GitOrigin-RevId: c72aa4bf91569904a2072c74d6ed2f3c764d97bb
 2026-01-13 09:06:55 +00:00
Brian Gough 42d09010aa Merge pull request #30174 from overleaf/bg-history-backup-allow-stalled-jobs-to-be-retried 
Allow stalled jobs to be retried in backup queue settings

GitOrigin-RevId: c68605aa89e9f8d4c2bc11fc14739a7d0774096d
 2025-12-10 09:06:18 +00:00
Brian Gough b9888957e4 Merge pull request #30000 from overleaf/bg-extend-backup-blob-script 
extend backup blob script

GitOrigin-RevId: 226e624f0fd93bfe4890acce728ce8005f9787c5
 2025-12-08 09:06:05 +00:00
Brian Gough 7a57ef00cb Merge pull request #30045 from overleaf/bg-object-persistor-make-list-directory-safer 
Improve safety of object persistor

GitOrigin-RevId: bced9814de6613b388ca288a5f72cd42cff6c1d3
 2025-12-05 09:06:42 +00:00
Brian Gough 06f696ced0 Merge pull request #29980 from overleaf/bg-history-extend-backup-comparison-III 
Check file tree hashes in backup comparison

GitOrigin-RevId: 4bd1f36afa34f326d4b8934c8bb0ea00a52cf1d9
 2025-12-01 09:06:18 +00:00
Brian Gough e0d6d7cb48 Merge pull request #29952 from overleaf/bg-history-extend-backup-comparison-II 
Add batch comparison options for backup verification

GitOrigin-RevId: fe12270b4063c4105f7e7a950c6a58cd39a62843
 2025-11-28 09:06:11 +00:00
Brian Gough 70909c126c Merge pull request #29950 from overleaf/bg-history-extend-backup-comparison 
Add fast comparison option for blob backups

GitOrigin-RevId: e1383425487ac1b4439248f976e6106567bae07d
 2025-11-28 09:06:06 +00:00
Brian Gough 1f356754de Merge pull request #29801 from overleaf/bg-history-refactor-backup-worker-shutdown 
refactor history-v1-backup worker shutdown

GitOrigin-RevId: 9666a99b00b30e98844e7dd25932f1590d0879b3
 2025-11-24 09:05:49 +00:00
Brian Gough e822ac0ee0 Merge pull request #29775 from overleaf/bg-history-use-configurable-concurrency-in-backup-worker 
allow configurable concurrency for history-v1 backup worker

GitOrigin-RevId: 59c734b013f99e215cc84688142cb0fbe45b064b
 2025-11-24 09:05:44 +00:00
Brian Gough 3227502aeb Merge pull request #29718 from overleaf/bg-history-fix-backup-errors 
fix backupBlob error handling and add tests for upload failures

GitOrigin-RevId: 3abda1b70012fda20fc403593d0ee6ce04152e24
 2025-11-19 09:05:19 +00:00
Brian Gough 581a3b7ce7 Merge pull request #29700 from overleaf/bg-history-v1-log-stalled-uploads 
Add timeout warnings for long-running blob and chunk uploads

GitOrigin-RevId: 5a57ebaaea9dfcaf23153f7e6f27190af541eb16
 2025-11-18 09:06:02 +00:00
Brian Gough cc7f0c1b9b Merge pull request #29321 from overleaf/bg-ignore-deleted-projects-in-history-backup 
handle deleted projects in history-v1 backup worker

GitOrigin-RevId: f4392045074248137f15d082d922c18b1ef9232f
 2025-10-28 09:05:34 +00:00
Brian Gough 2cae9c4635 Merge pull request #29097 from overleaf/bg-reduce-retry-backup-script-concurrency 
reduce concurrency for retrying failed backups

GitOrigin-RevId: aeb1f05b7ed26b80c00f12fe07a5f30a14786c30
 2025-10-16 08:05:56 +00:00
Brian Gough f6229d6fe1 Merge pull request #29032 from overleaf/bg-retry-failed-backups 
Add script and cronjobs to retry failed history-v1 backups

GitOrigin-RevId: c32102086235d4e22b947f4a80089b186c093cea
 2025-10-14 08:06:28 +00:00
Miguel Serrano 3a35b8680e Merge pull request #28554 from overleaf/msm-force-s3-lib-storage-uploads 
[object-persistor] Use `@aws-sdk/lib-storage` for all uploads

GitOrigin-RevId: ab8e54a7bae843f9e6b05ed9cf936130a36b8c2f
 2025-10-08 08:05:55 +00:00
Brian Gough ff267ca246 Merge pull request #28640 from overleaf/bg-fix-chunk-store-metrics 
update metrics for getChangesSinceVersion to include status and hasMore flag

GitOrigin-RevId: 389353425a12d6d7c92aa3ae37d6553fc1c7fc1d
 2025-09-23 08:08:09 +00:00
Brian Gough 386e133766 Merge pull request #28436 from overleaf/bg-load-changes-directly-from-redis-buffer 
load changes directly from redis buffer for getChanges requests

GitOrigin-RevId: 99673c47a137ff4222d331fa88eb6e5103270551
 2025-09-12 08:07:35 +00:00
Brian Gough c6df7575d2 Merge pull request #28377 from overleaf/bg-inefficient-query-during-history-backup 
Fix inefficient query during history backup verification

GitOrigin-RevId: c33246dca2436e82056506a2dceb79c465cd8301
 2025-09-11 08:04:57 +00:00
Jakob Ackermann b5fc004c0a [history-v1] add script for listing owned projects with size (#28292) 
* [history-v1] add script for listing owned projects with size

* [history-v1] handle project with no blobs record

* [history-v1] add plain sumBytes for easy sum in spreadsheet

GitOrigin-RevId: c20af189c223f867a6033cfb904f92a3a10c7cfb
 2025-09-05 08:05:33 +00:00
Eric Mc Sween 8f0913fafe Merge pull request #27936 from overleaf/em-unit-tests-mongo 
Make Mongo available to unit tests in all services

GitOrigin-RevId: b65bbb69883d5bba31d09802b89f35bdc523fe4d
 2025-08-25 08:05:19 +00:00
Jakob Ackermann 56f41c77ff [history-v1] add migrations record after full binary file migration (#27932) 
* [history-v1] add migrations record after full binary file migration

* [server-pro] add hotfix 5.5.5

* [server-ce] test: build hotfix 5.5.5 and use it in tests

GitOrigin-RevId: fb84e5710c59f466a3305de5f32f78e0ac9ce15d
 2025-08-19 08:05:54 +00:00
Eric Mc Sween eb2320a8af Merge pull request #27823 from overleaf/em-increase-max-request-size 
Increase max request size in docstore and history

GitOrigin-RevId: 8ce6c023027457a65431ee5deb2f75f467914f14
 2025-08-19 08:05:35 +00:00
Jakob Ackermann 29045b9085 [web] download binary files in clsi from filestore via new endpoints (#27505) 
* [history-v1] use String.padStart instead of lodash.padStart

* [web] download binary files in clsi from filestore via new endpoints

* [server-ce] tests: Cypress.env() is parsing boolean values

* [server-ce] tests: run history migration as root

GitOrigin-RevId: bdf6c0e542531ccc4b3f13d2ed68ca0d31e580e9
 2025-07-31 08:05:28 +00:00
Jakob Ackermann 1fd4445a34 [history-v1] extend --report with number of files to check/missing (#27397) 
* [history-v1] put logs of back_fill_file_hash script in /var/log/overleaf

* [history-v1] do not leave tmp dir behind when using --report

* [history-v1] extend --report with number of files to check/missing

* [history-v1] move all the options into command line flags

Change default values:
- concurrency to 10
- concurrent batches to 1
- stream high water mark to 1MiB

* [web] find_malformed_filetrees: log progress on stderr

* [history-v1] fix percentage of projects that need back-filling

GitOrigin-RevId: ebd7889840f97b24eda43774ffe7524998fb634b
 2025-07-30 08:07:12 +00:00
Brian Gough b8e2a8bfdb Merge pull request #27257 from overleaf/bg-filestore-migration-usability 
filestore migration usability changes

GitOrigin-RevId: 47e8c8434c35b1b16c41700dfef11ce4602a3063
 2025-07-24 08:05:25 +00:00
Jakob AckermannandBrian Gough 81f0807fc6 [web] prepare filestore migration for Server Pro/CE (#27230) 
* [web] prepare filestore migration for Server Pro/CE

* [history-v1] remove unused USER_FILES_BUCKET_NAME env var from script

* [server-ce] tests: write default docker-compose.override.yml on startup

* [server-ce] tests: extend access logging of host-admin for response

* [server-ce] tests: test text and binary file upload

* [server-ce] tests: add tests for filestore migration

* [web] simplify feature gate for filestore/project-history-blobs logic

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [server-ce] test: fix flaky test helper

---------

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: f89bdab2749e2b7a49d609e2eac6bf621c727966
 2025-07-22 08:06:58 +00:00
Jakob AckermannandBrian Gough bf43d4f709 [history-v1] make back_fill_file_hash_fix_up compatible with Server Pro (#27280) 
* [history-v1] move MockFilestore into shared place

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [history-v1] make back_fill_file_hash_fix_up compatible with Server Pro

---------

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: 70ea57e1503031d9f14dcd60c4c110e746450587
 2025-07-22 08:06:41 +00:00
Brian Gough 5208ad39ec Merge pull request #27173 from overleaf/bg-filestore-migration-remove-backups 
remove backup code from back_fill_file_hash script and tests

GitOrigin-RevId: 364eefc47670e6e7f2314de810ea259b609ff976
 2025-07-18 08:06:14 +00:00
Brian Gough 5669a7d1c8 Merge pull request #27147 from overleaf/bg-filestore-migration-for-server-pro-II 
add support for fetching files via http from filestore in back_fill_file_hash script and tests

GitOrigin-RevId: 8dea6383ed6fe9ee6786a5695e2deee93b1cdd84
 2025-07-17 08:05:30 +00:00
Eric Mc Sween 1833bd3d00 Merge pull request #27126 from overleaf/em-fix-persist-changes 
Fix chunk creation over a one change chunk

GitOrigin-RevId: aecae334849522975b83c77224ee27db64de4ed8
 2025-07-16 08:05:40 +00:00
Brian Gough 8fab1b54a3 Merge pull request #27025 from overleaf/bg-delete-redis-buffer-when-project-deleted 
delete redis buffer when project deleted

GitOrigin-RevId: eef7b6fdeb04cb556ae47794379d83e659f89b2e
 2025-07-11 08:06:08 +00:00
Jakob Ackermann d7a0cbefea [history-v1] fix typo in metrics name in redis chunk store (#26865) 
GitOrigin-RevId: 5b37e3b22a3250fcfbd7411e635cc11ed38dcb27
 2025-07-08 08:05:52 +00:00
Jakob Ackermann cf472f54d0 [history-v1] use POST requests for expiring redis buffer from cron (#26568) 
* [history-v1] use POST requests for expiring redis buffer from cron

(cherry picked from commit 15780ac54e36b96e1aed9fd9eb6dfe9d4fbf842f)

* [history-v1] remove double claim of expire job

GitOrigin-RevId: 8b2eab07006a5819a47eed3f646b2a4d75f86e5b
 2025-07-08 08:05:47 +00:00
Eric Mc Sween 735cc2272f Merge pull request #26505 from overleaf/em-persist-buffer-batch 
Persist Redis buffer in batches

GitOrigin-RevId: 1de22807dae3554b3274ec103783b0868b1439d9
 2025-06-30 08:05:25 +00:00
Eric Mc Sween a38eefd2ab Merge pull request #26559 from overleaf/em-redis-buffer-delete 
Delete project from redis buffer when expiring

GitOrigin-RevId: 66c48adc388a4a4e0b1b54b581dc945a38a41356
 2025-06-30 08:05:20 +00:00
Andrew Rumble 903277c222 Handle rangeHashes 
GitOrigin-RevId: 9b9a67d8e45f17e948d003aed33cef6128a25744
 2025-06-25 08:06:50 +00:00
Andrew Rumble 34b674aa6f Handle errors when converting lazyfiles to eagerfiles 
GitOrigin-RevId: 6ada48e2f2b7ffee9c8560856436dc5e2076099d
 2025-06-25 08:06:45 +00:00
Andrew Rumble 745043ca92 Add more detail for the modes 
GitOrigin-RevId: 44b63b76b1edb8bfb78049c9c8560934e340ef96
 2025-06-25 08:06:41 +00:00
Andrew Rumble 6ed488cc65 Change raw mode to just fetch all raw content from backup 
GitOrigin-RevId: 397060d63c824dbe688d187312dd78d1f3e4bf5d
 2025-06-25 08:06:36 +00:00
Andrew Rumble f2b0a982ac Use BackupBlobStore to get chunk content 
GitOrigin-RevId: 91de53101ea289b43bdb08352aecc09ae89d0f74
 2025-06-25 08:06:32 +00:00
Andrew Rumble 797f29d40a Handle errors when processing command line arguments 
GitOrigin-RevId: 7c1f011e705d6e4b6fbf0afee3dc3aab749f0de4
 2025-06-25 08:06:27 +00:00
Andrew Rumble 392037efd6 Improve types and export functions we want to use 
GitOrigin-RevId: 25fd93214f88c425bee23d43de113cf754f89524
 2025-06-25 08:06:18 +00:00
Andrew Rumble 3600aa4b75 Add script for restoring projects from backup 
GitOrigin-RevId: b639d74989afa17638bbcba0f8ee34c978fcd5d9
 2025-06-25 08:06:14 +00:00
Andrew Rumble 25a911d4cb Export loadChunk from verifier to allow code reuse 
GitOrigin-RevId: fc5f2de50583ab2469e8ab3c86776c55501ba2e2
 2025-06-25 08:06:09 +00:00
Eric Mc Sween 1042092144 Merge pull request #26547 from overleaf/em-revert-expiry-post 
Revert "use POST requests for expiring redis buffer from cron"

GitOrigin-RevId: 95e9fd1be7b73699d6fac24035437d467c273d0a
 2025-06-19 08:06:58 +00:00
Eric Mc Sween 150dfd6cba Merge pull request #26539 from overleaf/jpa-post 
[history-v1] use POST requests for expiring redis buffer from cron

GitOrigin-RevId: 51c9a25b998e581ed20c0e113bd4989537a1e6ef
 2025-06-19 08:06:50 +00:00
Eric Mc Sween fd9fd9f0e7 Merge pull request #26545 from overleaf/jpa-fix-resync 
[history-v1] use mongo projectId for initiating resync

GitOrigin-RevId: f93f2358695782fb222d23ba3720d98724b9a291
 2025-06-19 08:06:45 +00:00
Eric Mc Sween 7bdc4291fc Merge pull request #26533 from overleaf/jpa-queue 
[history-v1] use POST requests for flushing history queues

GitOrigin-RevId: ae87a046a7844b25cc123d08ac6c8d1624608394
 2025-06-19 08:06:16 +00:00
Eric Mc Sween f68bf5a69f Merge pull request #26477 from overleaf/em-redis-buffer-resync 
Handle invalid content hash when persisting changes

GitOrigin-RevId: 5259190396c8c261cad1abcd5de66314c1e871fb
 2025-06-18 08:06:17 +00:00
Eric Mc Sween 9cb4ef4d7d Merge pull request #26353 from overleaf/bg-history-redis-extend-persist-worker 
extend persist worker to make parallel requests

GitOrigin-RevId: 8def7d5a8b5c9fcbe5fe45ac8f3ace503d31877a
 2025-06-17 08:05:55 +00:00