* [storybook] invalidate babel cache if needed
* [storybook] defer import until after overriding env var
GitOrigin-RevId: 735e6c794cb61f72b7a5ccbe9d6ae310b577b727
* [clsi] initial implementation of compile from history
* [clsi] copy changes
* [saas-e2e] extend test case with nested folder
* [saas-e2e] add test case for tracked changes
* [web] fix accumulating changes from multiple chunks
* [web] optimize size check for compile request payload
* [clsi] deduplicate globalBlobs
* [clsi] add validation for request body details
* [clsi] add metrics for compile from history
* [clsi] download binary files concurrently
* [clsi] skip download of empty file blob
* [clsi] break down e2e compile time metric by compileFromHistory
GitOrigin-RevId: 0dadef93e89d8a172c35cb130a1042d9d1bec42a
* [web] increase timeout for downloads from clsi
* [web] detach request timeout from streaming timeout
* [web] align timeout for clsi-cache downloads with clsi downloads
* [web] destroy stream when bailing out early from aborted requests
GitOrigin-RevId: cc4cad9d684214b2eee0f5fc43513e430ceb0977
* [monorepo] switch all output file reads to clsi-nginx
* [clsi-lb] allow gallery download requests
* [terraform] clsi: use nginx.conf from clsi service
* [clsi] fix flakey tests
* [clsi] replace alias with rewrite and root in nginx config
* [k8s] clsi-lb: expose download port on internal service
* [web] add explicit endpoint for downloading all output files
Serve the output.zip endpoint from clsi.
* [clsi] fix regex for latexqc submission ids
Previously, we only handled template submission ids.
GitOrigin-RevId: 6c3b21b01ec41ae767530b14aac31fbe3d640dd5
* [dependency-management] check eslint and prettier in Jenkins pipeline
* [npm-overrides-helper] check eslint and prettier in Jenkins pipeline
GitOrigin-RevId: 647f105babd536d0962ac2dbb3d6a0235ba73976
* [web] validate and parse email using EmailHelper when ratelimiting
* [web] use a fake email for rate-limiting ldap logins in Server Pro
GitOrigin-RevId: 27ea7724319e06c4d64ac81e1155dcab558da99c
11 years ago, the db.projects collection was storing doc lines in the
file-tree/rootFolder. Any operations on the project that did not need
those lines were benefitting from excluding all those entries from the
file-tree. These days, the verbose exclusions are not useful anymore and
merely add load on mongo.
REF: 9805c6a9ff
GitOrigin-RevId: 89f544688934c1ed1ca98877ffbe8baefe66c126
* [document-updater] remove_deleted_docs: use promisified RedisManager
* [document-updater] remove_deleted_docs: log all getDoc details
* [document-updater] remove_deleted_docs: add option to run on single doc
GitOrigin-RevId: e2b73defd1d99d14578ce573d0e05c31073ab739
* [monorepo] remove PII and variables from error messages
Exclusions:
- scripts
- tests
- fuzzing
- SplitTestManager (messages are sent to admin frontend)
- Group setup (we may want an error per unique tuple)
- sharejs (unused types; text type errors are shadowed already)
- history-v1 error messages that are used by the ErrorRecorder
- errors that flag issues with configuration/call signatures
I've used these search terms for finding unwanted error messages:
- new Error(`
- new Error\(\n\s+` (regex search)
- new OError(`
- new OError\(\n\s+` (regex search)
* [web] throw NotFoundError from ProjectLocator
* [github-sync] fix OError.tag call in script
Co-authored-by: Jessica Lawshe <jessica.lawshe@overleaf.com>
* [templates] revert changes to test client
---------
Co-authored-by: Jessica Lawshe <jessica.lawshe@overleaf.com>
GitOrigin-RevId: 736857a4fc5d9bfb0f8cb03e0f004eda87e5a220
* [monorepo] move retries from the outside to the per-test level
* [web] increase hookTimeout in CI
* [monorepo] consolidate test retries
- do not retry unit tests
- only retry in ci, optionally locally with "RETRIES=3 make test..."
- add retries for web
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
---------
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: 86e45edcfb087d18e0e957ad6df9a6105dcd5770
* [clsi] tweak logging for clsi-cache
- Use `clsi-cache` identifier on log line
- Add shard to context
- Record nFiles on "too many entries for tar" error
* [clsi] do not trip clsi-cache circuit breaker on ENOENT errors
These can happen when an output/compile-dir is purged while we download
files.
GitOrigin-RevId: ffa73ef312bce5232ef72e3b81966bb6e14d2255
* [monorepo] use ALLOWED_IMAGE_NAMES from production
* [web] add ALLOWED_IMAGE_NAMES to CI env
* [web] throw if ALLOWED_IMAGE_NAMES is not defined in SaaS
* [e2e] update references to ALLOWED_IMAGE_NAMES
* [web] check for ALLOWED_IMAGE_NAMES from the settings-check module
GitOrigin-RevId: 43a7343e80521e3f7e496fb771593b4c22df7d1e
* [monorepo] enable caching for eslint/prettier/stylelint
* [monorepo] speed up prettier by swapping --list-different for --check
--list-different will print each file that it processes. We have a lot
of files in the monorepo. Using --check only prints mismatching files.
Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>
* [monorepo] explicitly configure prettier cache-location
This is the default location. Prettier will only discover that location
if the top level node_modules folder is writable, which is not the case
in CI. We create the .cache folder outside of docker, writable to node
inside docker.
The proper fix would be in prettier, to only check for write access in
the cache folder. Something to raise/upstream another day.
* [monorepo] run top-level format/format_fix in a single container
With the cache in place, it is much faster to use a single container.
As there is a single shared cache file, concurrent processes may see a
partially (re-)written cache file and bail out.
- all in a single container: 24s
- previous with -j4: 41s
- previous with -j8: failed due to corrupted cache file
---------
Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>
GitOrigin-RevId: 7850a3a980ae6c836393d97fe56a6316ffc3fa18
* [e2e] e2e_test_setup: perform health check on all the services
* [monorepo] install curl in nginx-proxy
GitOrigin-RevId: 0d04e2b1012e8b0349cdb297b43a303797fb1968
* [web] deactivate_projects: spread archiving jobs over the cron interval
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [web] deactivate_projects: stop processing after graceful shutdown
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [web] deactivate_projects: exit w/ code 1 when aborting due to max-time
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [web] deactivate_projects: adjust bail out for timeout
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [web] deactivate_projects: handle small number of projects better
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* add comment about use of --maxTime option
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: c62c66725233d391fd2c8d86ce95275cea88f36c
* [monorepo] fix downloads from non-sharded clsi-cache
* [web] check some compile from cache options server-side
* [web] integrate clsi-cache into download pdf from project dashboard
* [web] remove frontend tests for server-side validation
* [web] remove unused fetch mock
* [web] use helper that adds polyfill for AbortSignal.any()
* [web] upgrade fetch-mock to fix leaking AbortSignal
* [web] do not add an extra timeout to clsi-cache request
The web backend service has a low timeout already.
GitOrigin-RevId: a90984b92f5d4f24005db5a09f2c5d2424436886
* [monorepo] also cycle ssl_proxy to avoid hitting circuit break in nginx
* [e2e] use plain pdf.js for extracting PDF text
* [monorepo] manually download canvas binding for MacOS
GitOrigin-RevId: dbadbbaa0e121953c06ab4be9241bf361997865e
* [e2e] ensure user registration runs in a fresh session
* [e2e] align session key in Server Pro and SaaS
Co-authored-by: Antoine Clausse <antoine.clausse@overleaf.com>
---------
Co-authored-by: Antoine Clausse <antoine.clausse@overleaf.com>
GitOrigin-RevId: cc322217dc5a631c2795f477edaa90da8b83959e
* [k8s] clsi-cache: migrate to StatefulSet
* clsi-cache: optimize ILB services for GKE subsetting
Update the new clsi-cache internal load balancer services
to use optimal settings for GKE subsetting (NEG backends):
- set allocateLoadBalancerNodePorts: false (not needed with NEGs)
- set externalTrafficPolicy: Local (preserve source IP, keep traffic in zone)
- add trafficDistribution: PreferClose (zone affinity)
These settings ensure traffic from CLSI VMs stays within the same zone
when possible, reducing latency and cross-zone network costs.
* [k8s] clsi-cache: add missing resource paths
* [clsi] exclude readOnly clsi-cache shards
---------
Co-authored-by: Daniel Kontsek <daniel.kontsek@overleaf.com>
GitOrigin-RevId: 34f18b319a0e859ff149a135131c95a44bc674d6
* [k8s] clsi-cache: switch parent-app label to 'clsi-cache-legacy'
* [k8s] clsi-cache: add service account from kustomization.yaml
* [k8s] clsi-cache: consolidate on a single array of CLSI_CACHE_INSTANCES
* [clsi-cache] make prettier happy
GitOrigin-RevId: 4082a71df591904cfe437c4bde74759ddd83634c
* [e2e] enable --notablescan for Server Pro and SaaS E2E tests
* [server-ce] remove --notablescan override when downgrading
GitOrigin-RevId: 41eb6060e066b8a531d9e815802f1b4464bf5102
* [document-updater] check for flushed docs when fetching historyId
* [document-updater] use doc version key for discovering all docs
GitOrigin-RevId: 122c6786b473c0836a7215ded4fae2819f908cd7
* [e2e] enable --notablescan for Server Pro and SaaS E2E tests
* [web] fix query for expiring projects of E2E test users
* [web] check for --notablescan in e2e test setup
GitOrigin-RevId: f08812899647bc8e4bc0c0d14d3250eb0b9f5c59
* [clsi] remove all clsi-perf/health-check metrics
* [clsi] always emit E2E compile time metric
* [clsi] do not collect metrics for clsi-cache-template compiles
* [clsi] fix unit tests: request.metricsOpts always exists
* [clsi] use a gauge for the e2e compile time metric of clsi-perf
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
* [clsi] remove metrics for binary file downloads from clsi-perf
---------
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: 7995512e57c802086350e3d1a0ec5213ecdf0a05
* [web] override timeout for a specific template
* [web] fix junit integration for vitest
* [web] consolidate on JUNIT_ROOT_SUITE_NAME
* [web] fix unit test
GitOrigin-RevId: 3ba865e4b9c090c7bf8e07dc3d81e9bcee20d821
* [clsi-cache] remove old DiskCrawler
* [clsi-cache] use helpers for populating state during initial crawl
* [clsi-cache] use path helpers everywhere
* [clsi-cache] create project folder as part of the initial crawl
* [clsi-cache] add sharded disk layout
* [k8s] clsi-cache: switch clsi-cache-zone-b-shard-0 to new disk layout
* [clsi-cache] tweak logging context
* [monorepo] add bin/format_project_key util
* [k8s] clsi-cache: use a single nginx config file
GitOrigin-RevId: bb49146502cc6201eef210efae33355321efd77b
* [clsi] try harder at sending files off to a working clsi-cache shard
* [clsi] use a crc for generating a stable sequence of shards to try
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [clsi] gradually migrate to crc based shard assigment
* [clsi] tweak selecting clsi-cache shard from crc
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [clsi] bump rollout dates of new clsi-cache shard change
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: 9386e170503b405580e4d0a8641832f3fcb1fa83
* [monorepo] record ERROR/FATAL log messages in junit report
* [web] put SaaS specific code behind feature flag
* [web] use split test cache for getting user assignments
The unit tests needed updating as they did not replicate any of the
mongo filtering. The acceptance tests cover this logic.
* [web] make better use of existing indexes
* [web] avoid col-scan in tests of notifications module
* [web] remove cleanup of empty feedbacks collection
* [web] add assertion for reason of rejected request in launchpad test
* [web] add missing indexes
* [web] enable mongo notablescan
* [web] make emailNotifications tests compatible with notablescan
GitOrigin-RevId: b888f2feeb3a0e915f068ae1c4ea23ec17821221
* [clsi] fix circuit breaker for clsi-cache
* [clsi] enable ts-check for CLSICacheHandler
* [clsi] limit the number of .blg files in clsi-cache to 50
* [clsi-cache] limit the number of files per job to 100
* [clsi-cache] explain early registration of buildId
* [clsi-cache] lock down downloads via nginx to project folder
GitOrigin-RevId: 081d0c40b08db3a384c4d765b71a50b973f42151
* [clsi] gracefully handle fast exit of synctex/wordcount containers
* [clsi] do not change container options in-place for logging
GitOrigin-RevId: 0b685310a3c72f8f46125fefaa30c1ddb19e7b07
* [migrations] make old migrations compatible with --notablescan in mongo
Re-running the migration for db.users.emails.reversedHostname on all
Server Pro/CE users will not be harmfull.
* [monorepo] enable mongo notablescan for all services but web
GitOrigin-RevId: e2a710c9de08de9ac2aa9df1f3082d5f1c33ea05
* [web] tweak Jenkins pipeline following move to faster CI VMs
The webpack build time has been reduced by about 30%. Adjust the sleep
statement for delaying the webpack build accordingly.
Push the production docker image layers from the parallel steps already.
Use a shared "scratch" tag to avoid generating cruft per branch/build.
* [web] avoid sharing the cypress/downloads folder in CI
GitOrigin-RevId: 52fdf420ec04fd26e3823ff9fca8c52f7a7764d7
Stage timeouts:
- frontend waits 5s
- web/clsi waits 4s
- clsi-cache waits 3s
This should ensure that the frontend can receive a valid response after
any of the backend requests failed.
The circuit breaker will remain closed for TIMEOUT + jitter of 0-3 times
the TIMEOUT of the respective service. This should avoid the bulk of
traffic to fail and occasionally issue retries without hammering the
instances while down.
Also do not try the next backend when the abort signal has expired.
GitOrigin-RevId: d612125616a9e416beff2f4c6d7f30066b5b9d6d
* [web] add clsi-cache prompts
* [web] add new editor variant to segmentation
* [web] add tests for useNewEditorVariant
* [web] adjust start of using clsi-cache in split-test
GitOrigin-RevId: c9c5b1eff2ceefb65ef82516d9074cb971cb4c48
* [clsi] add stats and timings to compile response from clsi-cache
* [clsi] set downloadedFromCache when previously downloaded for synctex
Assumption: every compile will emit an output.log. When the output.log
is missing, but the output.synctex.gz exists, it must have been
downloaded from the cache.
GitOrigin-RevId: 41ea34880931e3c43dda3bc9eb26c0d02054894d
* [monorepo] switch from cypress-reports to reports
* [saas-e2e] collect junit reports from new editor tests
* [saas-e2e] simplify setup for semaphore variable in Jenkins pipeline
We do not retry pipelines anymore.
* [saas-e2e] put new editor tests into their own namespace
GitOrigin-RevId: 1869310bf58f0b5b0081dcae2f16ec1a267caec6
* [monorepo] remove docker-repos flag from build scripts
* [monorepo] use content hash as docker image cache key
* [packer] jenkins-worker: populate build cache for all services
* [v1] adopt smarter docker caching
* [latexqc] adopt smarter docker caching
* [monorepo] refresh docker cache in Jenkins
* [packer] jenkins-worker: increase disk size
* [monorepo] run jenkins_docker_build_cache as Jenkins user for v1
* [monorepo] define MONOREPO at the top of all the Makefiles
* [monorepo] add --build-arg BUILDKIT_INLINE_CACHE=1 everywhere
GitOrigin-RevId: 8110c6e68727a52ea80f3ec71711d30e17d69499
The rake deploy:app task is expecting a build for all web changes.
Triggering a few extra Server Pro E2E test runs is cheaper than making
the rake task smarter.
GitOrigin-RevId: a8380417d4a323b3d2a008eeb6ee26bc0ad151d9
* [libraries] move all the unit tests into test/unit
* [settings] remove empty test file
* [mongo-utils] remove test setup as there are no tests
* [libraries] ensure that all libraries with tests have test/setup.js
* [monorepo] remove test step from Cloud build for libraries without tests
GitOrigin-RevId: b9a0a8a3330db7d8646c7a272b03fbb8c5e44bc0
* [monorepo] refactor retries in Jenkins to step level
Change the junit reports to use step specific file names. The [hash]
template option was neat in getting unique file names, but results in
duplicate test reports on retry.
* [patches] add support for .cjs config files for mocha-multi-reporters
GitOrigin-RevId: 3a749441470b5ba633e71319589606cfbe860952
shouldFlushHistoryOps has a default value for 'threshold', which keeps
the exports simpler and still lets the unit tests override it.
GitOrigin-RevId: 1c6d4a2778052b5af40e2e338589a230ac2f4646
* [server-ce] tests: migrate host-admin to ESM, zod and npm-workspaces
* [server-ce] test: use import.meta.dirname
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
* [server-ce] test: fix zod schema for docker compose endpoint
---------
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: d490948693f341210c0ab5c2947db7c9a17775ef
* [web] add junit integration for Jenkins
* [web] integrate junit test report for writefull into Jenkins
* [web] put all the junit test results into namespaces
GitOrigin-RevId: ba1ff07b5ea7bcfa97bb4d6bf7fa9e5291ab7b0f
Jakob Ackermann