Brian Gough
c23e84eb37
Merge pull request #26273 from overleaf/bg-history-redis-add-persist-worker-to-cron
...
modify existing run-chunk-lifecycle cron job to persist and expire redis queues
GitOrigin-RevId: afb94b3e2fba7368cfec11997dfd5b2bbd6321a9
2025-06-11 08:06:23 +00:00
Brian Gough
2eb695f4c3
Merge pull request #26122 from overleaf/bg-history-redis-make-persist-buffer-consistent
...
make persistBuffer export consistent with other methods
GitOrigin-RevId: 24536e521e1d20ef63cc74bd9ba40e095025d512
2025-06-09 08:05:39 +00:00
Brian Gough
d280f40885
Merge pull request #26116 from overleaf/bg-history-redis-show-buffer
...
add script to display redis buffer for a given history ID
GitOrigin-RevId: 71c2e79480c0873d30801ed3c13aa9a7fc7873f6
2025-06-09 08:05:35 +00:00
Brian Gough
393cee7af5
Merge pull request #25993 from overleaf/bg-history-refactor-persist-buffer-limits
...
refactor persist buffer to add limits
GitOrigin-RevId: 4a40a7a8812acf5bb7f98bfd7b94d81ebe19fc57
2025-06-04 08:06:19 +00:00
Brian Gough
50df3862e9
Merge pull request #25954 from overleaf/bg-history-expire-worker-fix
...
fix expire_redis_chunks to only clear job on error
GitOrigin-RevId: f7ec435edda95958b453fba501686dcfd84426f7
2025-06-04 08:06:14 +00:00
Brian Gough
a80203f748
Merge pull request #25909 from overleaf/bg-history-persist-worker
...
add history persist worker
GitOrigin-RevId: b9e31e7bdd84570efc0b87b9f5e90b4078551a8c
2025-06-04 08:06:10 +00:00
Jakob Ackermann
11b94593c2
[web] remove deledFiles collection ( #25750 )
...
* [history-v1] remove processing of deleted files when back-filling hashes
* [web] remove deledFiles collection
GitOrigin-RevId: 7c080e564f7d7acb33ebe7ebe012f415a847d0df
2025-05-23 08:05:01 +00:00
Brian Gough
0d70223a48
Merge pull request #25482 from overleaf/bg-fix-bull-exporter-errors
...
retain completed and failed jobs for backup queue
GitOrigin-RevId: 7831ce2565dc493e3ce7f55001207daea2140575
2025-05-13 08:05:40 +00:00
Brian Gough
9e07549ecb
Merge pull request #25449 from overleaf/bg-histoy-redis-refactor-expire-worker
...
refactor the expire worker to make it easier to extend
GitOrigin-RevId: 7b277b243ed51ab3b46316d98b7157af95a9e42b
2025-05-12 08:05:19 +00:00
Eric Mc Sween
391fca9e83
Merge pull request #25361 from overleaf/em-load-latest-raw
...
Rename loadLatestRaw() to getLatestChunkMetadata()
GitOrigin-RevId: e089dcfa57cbbc43df8666b51eca0d81d595a5a7
2025-05-09 08:06:14 +00:00
Eric Mc Sween
5717ea7f5c
Merge pull request #25306 from overleaf/em-redis-buffer-read-operations
...
Add changes from Redis when reading chunks from the chunk store
GitOrigin-RevId: c0ebf0669b91eb2efc5d1091d025e81efdff9fe4
2025-05-09 08:06:09 +00:00
Brian Gough
f9b36cd5be
Merge pull request #25241 from overleaf/bg-remove-existing-chunk-buffer
...
remove existing chunk redis backend and chunk buffer
GitOrigin-RevId: 28fb02d1802312de6892e2fb7dd59191e3fc8914
2025-05-08 08:05:09 +00:00
Brian Gough
850da34778
Merge pull request #25086 from overleaf/bg-history-buffer-use-expire-time
...
add expire time to redis buffer in history-v1
GitOrigin-RevId: 3d74957c341e62e536dc60869a7ca71ac173e380
2025-04-29 08:05:14 +00:00
Brian Gough
626416ed02
Merge pull request #24945 from overleaf/bg-redis-buffer-stats-script
...
add script for listing buffer stats from redis
GitOrigin-RevId: 7836563d51a5c6ded264d0e709d5cfcda70596e3
2025-04-24 08:05:49 +00:00
Brian Gough
d6c2188f2d
Merge pull request #24903 from overleaf/bg-fix-backup-scheduler
...
remove history.id check from processPendingProjects function
GitOrigin-RevId: 017ab7c9bf693ed4077d4619574154166af04fe3
2025-04-16 08:05:06 +00:00
Brian Gough
3850e97446
Merge pull request #24857 from overleaf/bg-backup-scheduler-fix-for-broken-projects
...
handle broken projects in history backup
GitOrigin-RevId: a295d9d7a29715807a8172dae46e0fe3c14ecf12
2025-04-15 08:06:23 +00:00
Brian Gough
835e14b8b2
Merge pull request #24768 from overleaf/bg-history-redis-buffer
...
test redis caching when loading latest chunk in history-v1
GitOrigin-RevId: f0ee09e5e9e1d7605e228913cb8539be4134e1f7
2025-04-15 08:05:03 +00:00
Brian Gough
4ba0e97b95
Merge pull request #24749 from overleaf/bg-history-redis-buffer
...
add history redis to history-v1
GitOrigin-RevId: 70dc1aee809ad17902c93c020f3804c0f1429238
2025-04-10 08:05:35 +00:00
Brian Gough
040f70471c
Merge pull request #24636 from overleaf/bg-history-backup-fix-broken-projects
...
add --fix option to backup script
GitOrigin-RevId: 568c9158669bb1cede0f0dd75e7507b10e8ff5a2
2025-04-08 08:05:59 +00:00
Brian Gough
f08532dfb0
Merge pull request #24637 from overleaf/bg-history-backup-uninitialised-projects
...
backup uninitialised projects
GitOrigin-RevId: 9310ef9f803decffbd674024a1ffd33d1960a2c4
2025-04-08 08:05:54 +00:00
Andrew Rumble
2779691cd9
Add a script for verifying a set of blobs have been backed up
...
GitOrigin-RevId: 8f7af647ce0ac86ee9800cdf3d386fb78debed7a
2025-03-24 10:48:39 +00:00
Brian Gough
0292bc418d
Merge pull request #24335 from overleaf/bg-warn-old-pending-changes
...
add warning for backups outside rpo threshold
GitOrigin-RevId: a8421529ae64693d860b0325961b010a132426da
2025-03-18 09:05:49 +00:00
Andrew Rumble
f134746c9c
Extend script to allow multiple blobs to be processed from csv
...
GitOrigin-RevId: ad47eb754436ddc7f56b27ceda627268c3a030a1
2025-03-18 09:05:03 +00:00
Andrew Rumble
b5d6484991
Add a script to backup a single blob from a project
...
GitOrigin-RevId: 464e6d69093b87891497e07d1627cd20e2285380
2025-03-18 09:04:59 +00:00
Andrew Rumble
78481e010e
Add verification looper and handle shutdown signals
...
Shutdown signals become more relevant now that we are looping as we want
to gracefully stop processing records rather than continue looping.
GitOrigin-RevId: dbb499388c86d552d77954988f8fc27d140da3f1
2025-03-18 09:04:54 +00:00
Andrew Rumble
9d72eeeeac
Add new strategy to verify_sampled_projects
...
GitOrigin-RevId: d967da41250bb5945d5b8668b212d4a61b4f9d69
2025-03-18 09:04:50 +00:00
Brian Gough
2d48c86e61
Merge pull request #24269 from overleaf/bg-backup-replication-smoothing
...
extend backup scheduler to delay jobs for load levelling
GitOrigin-RevId: be62c458d3be464bf80ca590fcc107c2760aa061
2025-03-13 09:06:28 +00:00
Brian Gough
a178c0f400
Merge pull request #24268 from overleaf/bg-backup-replication-lag-metrics
...
add backup replication metrics
GitOrigin-RevId: a734435e3c6ce56350b2286bd218a5e2324d93a9
2025-03-13 09:06:24 +00:00
Brian Gough
cf105cf01d
Merge pull request #24267 from overleaf/bg-fix-init-option
...
update backup script to use index when finding uninitialised projects
GitOrigin-RevId: 04ce0654b3d42de1c1a9bb542482c2dd53540628
2025-03-13 09:06:20 +00:00
Brian Gough
3b93efdf5c
Merge pull request #24256 from overleaf/bg-backup-fix-sample
...
fix backup sample script
GitOrigin-RevId: 4bf4c15324d9b3b7ebd809f7b212e52d75ab216d
2025-03-13 09:06:16 +00:00
Andrew Rumble
149b590413
Fix script name in usage prompt
...
GitOrigin-RevId: ab4ed7479c96d3d1db82b23ef3c883706c61995e
2025-03-12 09:05:44 +00:00
Andrew Rumble
282f5f92ff
Ensure global blobs are loaded before verifying
...
GitOrigin-RevId: 3f78c44e6b4d98fd6c469b0f4fbacc033b0a0868
2025-03-12 09:05:39 +00:00
Andrew Rumble
afedce1b0e
Add script for verifying sampled sets of projects
...
GitOrigin-RevId: 23a599ea269a528201c67cda66bd2d00c0d376e4
2025-03-12 09:05:35 +00:00
Brian Gough
efd53e567c
Merge pull request #24237 from overleaf/bg-fix-backup-worker
...
fixes for backup worker
GitOrigin-RevId: 79c99e5b2d99d1b0298f69d968e0e2cb89281dd6
2025-03-12 09:05:04 +00:00
Brian Gough
c233243948
Merge pull request #24200 from overleaf/bg-backup-queue-pending-jobs
...
fix backup worker and backup scheduler to handle pending projects
GitOrigin-RevId: a97e011615666b3ae2b8fafe26a96d41b3609edd
2025-03-11 09:06:05 +00:00
Andrew Rumble
441c7a89a7
Merge pull request #24204 from overleaf/ar-jpa-add-chunk-verification
...
[history-v1] add chunk verification
GitOrigin-RevId: 7208ad20872386813bb1c6946283afddb5e8b1cf
2025-03-11 09:05:57 +00:00
Brian Gough
9ff3a45690
Merge pull request #24142 from overleaf/bg-backup-fix-blob-error-handling
...
fix bug which caused errors to be ignored when backing up blobs
GitOrigin-RevId: 64674a484a6e22493cba8f8e47d4671072dc2e9a
2025-03-07 09:06:26 +00:00
Brian Gough
281168fd52
Merge pull request #24080 from overleaf/bg-serialise-backup-initialisation
...
fix backup initialization to avoid incorrect use of batchedUpdate
GitOrigin-RevId: 6984f3510c6b03b3dfda35efea8173f848e58eff
2025-03-07 09:06:22 +00:00
Brian Gough
154de1a0f9
Merge pull request #24102 from overleaf/bg-backup-stats-script
...
add backup sampling script
GitOrigin-RevId: cfd546d421cdcfb9f8ad2111703a437a7272bd54
2025-03-07 09:06:17 +00:00
Brian Gough
893294e6b8
Merge pull request #24069 from overleaf/bg-backup-errors
...
more tweaks for backup errors
GitOrigin-RevId: 0f7c7bb5004923c3c22c6e3471bb7152cc3e05e2
2025-03-05 09:05:50 +00:00
Brian Gough
34b397073f
Merge pull request #24053 from overleaf/bg-backup-queue-tweaks-II
...
additional backup queue tweaks
GitOrigin-RevId: b54e0c5f4a23bb9962c5062bd1b80f9f124c3c87
2025-03-05 09:05:15 +00:00
Brian Gough
c815106d62
Merge pull request #24037 from overleaf/bg-fix-backup-worker-errors
...
backup worker tweaks
GitOrigin-RevId: 7901bb0257c7c7320afd08ef965f124347a1de30
2025-03-04 09:05:12 +00:00
Brian Gough
1dbf5dca10
Merge pull request #24019 from overleaf/bg-backup-add-error-logging
...
add logging of backup errors in batch mode
GitOrigin-RevId: 96ae7a2354f82451b3bbe8c5459c4d4cf47e5459
2025-03-04 09:04:44 +00:00
Brian Gough
4720e8f2c5
Merge pull request #23989 from overleaf/bg-backup-worker-reduce-logging
...
reduce log level from info to debug for backup logging
GitOrigin-RevId: 10f541b0e0068aafd50710226fb8425527fee7c5
2025-03-03 09:05:22 +00:00
Brian Gough
8d0fa5be65
Merge pull request #23977 from overleaf/bg-backup-worker-load-global-blobs
...
ensure global blobs are loaded in backup worker
GitOrigin-RevId: b80791ef38e7fa8db364bdd3a9c461757778409d
2025-03-03 09:05:08 +00:00
Brian Gough
b37b409994
Merge pull request #23946 from overleaf/bg-backup-queue-implementation
...
additional backup queue implementation
GitOrigin-RevId: 03754e57a6c6798a07dcca6a5248dec61b3cdc7a
2025-02-28 09:05:50 +00:00
Brian Gough
763e5ba82c
Merge pull request #23920 from overleaf/bg-backup-queue
...
initial history backup queue worker app
GitOrigin-RevId: e9647a2ec3caeb1fff843cead12164ba89fdb1f8
2025-02-28 09:04:46 +00:00
Brian Gough
6e55a9fa81
Merge pull request #23837 from overleaf/bg-redis-redact-config
...
sanitise password in redis config output
GitOrigin-RevId: 45d8bd13f01c3c4f138549c8a9f47dc0db29d0b0
2025-02-25 09:05:22 +00:00
Brian Gough
0acc067205
Merge pull request #23836 from overleaf/bg-backup-extend-debugging-script
...
add support for fetching chunks in backup debugging script
GitOrigin-RevId: 641298088d62148693525259748dcddf4d67d14b
2025-02-25 09:05:18 +00:00
Brian Gough
9d8f9e5358
Merge pull request #23835 from overleaf/bg-backup-show-errors
...
add error cause logging for missing chunks and blobs in backup comparison
GitOrigin-RevId: 9a1bf84b64621add71fb1d1e0cb013598b226bc8
2025-02-25 09:05:14 +00:00
Brian Gough