* [history-v1] use String.padStart instead of lodash.padStart
* [web] download binary files in clsi from filestore via new endpoints
* [server-ce] tests: Cypress.env() is parsing boolean values
* [server-ce] tests: run history migration as root
GitOrigin-RevId: bdf6c0e542531ccc4b3f13d2ed68ca0d31e580e9
* [history-v1] put logs of back_fill_file_hash script in /var/log/overleaf
* [history-v1] do not leave tmp dir behind when using --report
* [history-v1] extend --report with number of files to check/missing
* [history-v1] move all the options into command line flags
Change default values:
- concurrency to 10
- concurrent batches to 1
- stream high water mark to 1MiB
* [web] find_malformed_filetrees: log progress on stderr
* [history-v1] fix percentage of projects that need back-filling
GitOrigin-RevId: ebd7889840f97b24eda43774ffe7524998fb634b
* [server-pro] check_vulnerabilities.sh: move cache into $HOME/.cache
* [monorepo] move root-owned minio certs into docker volume
* [server-pro] check_vulnerabilities.sh: remove unused vars
* [filestore] fix image name for certs container when using sharding
GitOrigin-RevId: c6756e8f24ccc689c12d5f5be2b7f4eee7d9b989
* [monorepo] ensure DOCKER_USER is set and used on all cypress containers
* [server-ce] test: deduplicate cypress container config
GitOrigin-RevId: ca569a6c95574df7fb29c99577c18f22cdfb83d8
* [server-ce] test filestore migration with upgrade from version 1.x
* [server-ce] tests: drop verbose logs from host-admin in CI
* [server-ce] tests: fix flag following rebase
GitOrigin-RevId: dc00127fc76f87ee3eb5071fd430f4917e8123ff
* [third-party-datastore] use generic serializer for dropboxError
The `err` serializer will not pick up all the dropbox fields.
Co-authored-by: Thomas Mees <thomas.mees@overleaf.com>
* [third-party-datastore] handle user_suspended like insufficient_space
Unlink dropbox and display a notification (same key to clear later).
Co-authored-by: Thomas Mees <thomas.mees@overleaf.com>
* [third-party-datastore] skip retries when rejected with disallowed_name
Co-authored-by: Thomas Mees <thomas.mees@overleaf.com>
* [web] sort translations
* [web] update copy for dropbox_unlinked_because_suspended
Co-authored-by: Kamal Arkinstall <kamal.arkinstall@overleaf.com>
---------
Co-authored-by: Thomas Mees <thomas.mees@overleaf.com>
Co-authored-by: Kamal Arkinstall <kamal.arkinstall@overleaf.com>
GitOrigin-RevId: 8fbb9074d1d6eb879e904d79dd4b2a2c952ff902
* [web] reject upload requests without a file path
* [web] update copy on error message and link to contact form
Co-authored-by: Kamal Arkinstall <kamal.arkinstall@overleaf.com>
* [web] update copy: move dot to the end
---------
Co-authored-by: Kamal Arkinstall <kamal.arkinstall@overleaf.com>
GitOrigin-RevId: ba1ee81a91b046540caeb2f3f3da0e305611b35f
* [history-v1] move MockFilestore into shared place
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [history-v1] make back_fill_file_hash_fix_up compatible with Server Pro
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: 70ea57e1503031d9f14dcd60c4c110e746450587
* [mics] fix "app" label in clsi-cache metrics in dev-env
* [clsi-cache] validate filePath when processing file
* [clsi-cache] meter ingress and egress bandwidth
Files are downloaded directly from nginx, hence we cannot meter egress
in clsi-cache easily.
GitOrigin-RevId: 24de8c41728f0e9c984113c1470dec6153e75f20
* [terraform] clsi: add C3D trial
* [terraform] clsi: switch C3D to standard variant with local SSD
GitOrigin-RevId: 6e6ceeb93ee94800d8ce46557322da82dc99fc2a
* [history-v1] use POST requests for expiring redis buffer from cron
(cherry picked from commit 15780ac54e36b96e1aed9fd9eb6dfe9d4fbf842f)
* [history-v1] remove double claim of expire job
GitOrigin-RevId: 8b2eab07006a5819a47eed3f646b2a4d75f86e5b
* [web] change status page URL to HTTPS everywhere
Also open all links to the status page or admin email in a new tab.
* [server-ce] explicit protocol matching
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: d7d12eef4efc737c5d70fc969c862acae9faf14c
* [project-history] add support for resync of history-ot ranges
* [project-history] avoid compressing sharejs and history-ot upgrades
* [document-updater] improve error message of some assertions
... by migrating the assertions like this:
```diff
-stub.calledWith().should.equal(true)
+stub.should.have.been.calledWith()
```
```diff
-stub.called.should.equal(false)
+stub.should.not.have.been.called
```
* [document-updater] move content field in resyncDocContent
* [document-updater] add support for resync of history-ot ranges
GitOrigin-RevId: e6104686a26934a5f25a8f095cbe00c163fbbaa7
* [web] fix typo in ESM migration of a db migration
* [web] migrate old migration to ESM
* [web] use batchedUpdate for bulk updates in old migrations
GitOrigin-RevId: a984f785c577c2ac4125c947b8a3efffa57e1eb7
* [project-history] fix tests and cover callback for processing updates
The before setup was hiding that some tests were not doing what the
assertions were expecting.
* [project-history] fix callback signature when processing no updates
GitOrigin-RevId: 4fa14d47b9a1afd998316b0c9024d49760785a47
* [docstore] add endpoint for getting a projects comment thread ids
* [web] let docstore determine a projects comment thread ids
Also fetch the comment thread ids once when reverting project.
GitOrigin-RevId: c3ebab976821509c9627962e58918f9c6ebb0e1d
* [misc] migrate remaining references to our GCR repositories to AR
* [server-ce] fix retagging of texlive images
GitOrigin-RevId: 81f955ad4c4486ad42b29cbd6bcc9d5ef4b1a432
* [docstore] add endpoint for getting user ids of tracked changes
* [web] let docstore determine user ids of tracked changes
GitOrigin-RevId: 8d0a131555aa827f7ff80690fedc1aca26cf0817
* [web] skip db query when getting empty list of users
* [web] fetch token users in a single db query per access mode
GitOrigin-RevId: fa5d9edcb761bd5d5e5ea07d137a5a86efdbdd5c
* [web] hide sensitive data from joinProject when building project view
* [web] skip fetching members and invites for restricted users
* [web] fix owner features in joinProject view
* [web] separate invited members from owner
* [web] skip fetching users with empty members list
* [web] split await chain
Co-authored-by: Antoine Clausse <antoine.clausse@overleaf.com>
* [web] remove spurious parentheses
* [web] remove dead code
Co-authored-by: Antoine Clausse <antoine.clausse@overleaf.com>
---------
Co-authored-by: Antoine Clausse <antoine.clausse@overleaf.com>
GitOrigin-RevId: 5b4d874f974971e9c14d7412620805f8ebf63541
* [document-updater] flush history for projects with short queues ASAP
* [k8s] document-updater: enable short history queue for history-ot demo
* [project-history] flush history for projects with short queues ASAP
* [project-history] wait for mongo before running acceptance tests
* [k8s] project-history: enable short history queue for history-ot demo
* [project-history] change wait-for-mongo step in tests
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
---------
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: 3e989c409e8e9887655b35f2659ce0829e61b357
* [clsi] shed load when detecting out-of-disk condition
* [clsi] mark VM as unhealthy when detecting of-of-disk condition
GitOrigin-RevId: 25cda6785c0d973f50ec6206bee389804f35917e
* [history-ot] rename remaining history-v1-ot references to history-ot
* [web] rename History-v1 OT -> History OT in admin panel
* [web] rename OT Migration -> History OT Migration in admin panel
GitOrigin-RevId: 103ce816d5320d6379d51009cdc08b8a71aa48e6
* Revert "[clsi-cache] only use sharding from updated project editor tabs (#25326)"
This reverts commit 1754276bed3186c0536055c983e32476cc90d416.
* [clsi-cache] remove non sharded instances
GitOrigin-RevId: aa3ac46140dfc1722a3350cf7071e5b11af61199
* [clsi] tell frontend when synctex mapping was downloaded from clsi-cache
* [web] emit event when synctex mapping was downloaded from clsi-cache
GitOrigin-RevId: 1f6b7e0faaa7dd76449aad566802da971a4cf9ed
* [web] move building of compile from cache response into manager
* [web] retry fetching initial compile from cache response
GitOrigin-RevId: b4dc89f1b91d99e869c0c7789881dc72d8a5761f
* [web] settle on a single split-test for the clsi-cache rollout
Use the split-test that was used for rolling out the writes so that we
can use their already populated caches.
* [clsi-cache] fix non-sharded clsi-cache in dev-env
GitOrigin-RevId: 6ebd6369183342fe6d5e325b760d011fd1d57516
* [history-v1-ot] initial implementation of using doc-level history-v1-ot
* [web] fix advancing of the otMigrationStage
Use 'nextStage' for the user provided, desired stage when advancing.
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [document-updater] document size check in editor-core
* [history-ot] rename history-v1-ot to history-ot and add types
* [history-ot] apply review feedback
- remove extra !!
- merge variable assignment when processing diff-match-match output
- add helper function for getting docstore lines view of StringFileData
Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
* Revert "[document-updater] add safe rollback point for history-ot (#25283)"
This reverts commit d7230dd14a379a27d2c6ab03a006463a18979d06
Signed-off-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
---------
Signed-off-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
GitOrigin-RevId: 89c497782adb0427635d50d02263d6f535b12481
* [clsi-cache] shard per zone into three instances
Keep the old instance as read fallback. We can remove it in 4 days.
Disk size: 2Ti gives us the maximum write throughput of 240MiB/s on a
N2D instance with fewer than 8 vCPUs.
* [clsi] fix format
* [k8s] clsi-cache: bring back storage-classes
* [k8s] clsi-cache: fix reference to zonal storage-classes
* [k8s] clsi-cache: add logging configs
* [clsi] improve sharding
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [clsi] fix sharding
Index needs to be positive.
* [clsi] fix sharding
The random part is static per machine/process.
* [clsi] restrict clsi-cache to user projects
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [k8s] clsi-cache: align CLSI_CACHE_NGINX_HOST with service LB
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: 1efb1b3245c8194c305420b25e774ea735251fb3
- Remove settings ignore, they are inconsistent and break local prettier
- Remove .dockerignore files, only root ignore file is used
- Move .idea/.run/*.swp/coverage to root
- Remove .npmrc entries, we are no longer writing the rc file
- Remove node_modules/.DS_Store, is contained in root
- Remove cruft
GitOrigin-RevId: 3025fd5acaef343312f55149466c638e759a6e1f
* [k8s] clsi-cache: increase the number of workers
* [clsi-cache] add a global limit on concurrent writes
* [k8s] clsi-cache: increase timeouts for health checks
* [k8s] clsi-cache: align resource requests with current usage
GitOrigin-RevId: 2aba881ac0e581aa8db78a30d2c58afee6702318
* [clsi] merge sandboxed compiles config from Server Pro and SaaS
* [clsi] reorder fallback env vars
Co-authored-by: Mathew Evans <matt.evans@overleaf.com>
* [server-pro] bump version of expected release with these changes
---------
Co-authored-by: Mathew Evans <matt.evans@overleaf.com>
GitOrigin-RevId: bada93fec89bcc3f2bab85b6e60b2e27de88b9c2
* [web] stopOnFirstError=true does not conflict with =false locally
Allow stopOnFirstError to be enabled in the compile from cache and
disabled locally.
Compiles that passed with stopOnFirstError=true will also pass with
stopOnFirstError=false. The inverse does not hold, and we need to
recompile.
* [web] record event when using compile from cache
* [web] record event when falling back to clsi-cache
* [web] make clsi-cache a premium feature
* [k8s] clsi-cache: increase disk size for beta rollout
NOTE: As this is a premium feature and paid servers run in zones c+d, we
do not need to scale up clsi-cache in zone b for now.
* [web] enable full sampling of compile-result-backend events
* [web] fix frontend tests
* [web] be more verbose when determining access to clsi-cache feature
GitOrigin-RevId: 6fd663e16085187876eb225f7f33eeeaf69d2b2a
Jakob Ackermann