* [clsi] initial implementation of compile from history
* [clsi] copy changes
* [saas-e2e] extend test case with nested folder
* [saas-e2e] add test case for tracked changes
* [web] fix accumulating changes from multiple chunks
* [web] optimize size check for compile request payload
* [clsi] deduplicate globalBlobs
* [clsi] add validation for request body details
* [clsi] add metrics for compile from history
* [clsi] download binary files concurrently
* [clsi] skip download of empty file blob
* [clsi] break down e2e compile time metric by compileFromHistory
GitOrigin-RevId: 0dadef93e89d8a172c35cb130a1042d9d1bec42a
* [monorepo] switch all output file reads to clsi-nginx
* [clsi-lb] allow gallery download requests
* [terraform] clsi: use nginx.conf from clsi service
* [clsi] fix flakey tests
* [clsi] replace alias with rewrite and root in nginx config
* [k8s] clsi-lb: expose download port on internal service
* [web] add explicit endpoint for downloading all output files
Serve the output.zip endpoint from clsi.
* [clsi] fix regex for latexqc submission ids
Previously, we only handled template submission ids.
GitOrigin-RevId: 6c3b21b01ec41ae767530b14aac31fbe3d640dd5
* [monorepo] remove PII and variables from error messages
Exclusions:
- scripts
- tests
- fuzzing
- SplitTestManager (messages are sent to admin frontend)
- Group setup (we may want an error per unique tuple)
- sharejs (unused types; text type errors are shadowed already)
- history-v1 error messages that are used by the ErrorRecorder
- errors that flag issues with configuration/call signatures
I've used these search terms for finding unwanted error messages:
- new Error(`
- new Error\(\n\s+` (regex search)
- new OError(`
- new OError\(\n\s+` (regex search)
* [web] throw NotFoundError from ProjectLocator
* [github-sync] fix OError.tag call in script
Co-authored-by: Jessica Lawshe <jessica.lawshe@overleaf.com>
* [templates] revert changes to test client
---------
Co-authored-by: Jessica Lawshe <jessica.lawshe@overleaf.com>
GitOrigin-RevId: 736857a4fc5d9bfb0f8cb03e0f004eda87e5a220
* [monorepo] move retries from the outside to the per-test level
* [web] increase hookTimeout in CI
* [monorepo] consolidate test retries
- do not retry unit tests
- only retry in ci, optionally locally with "RETRIES=3 make test..."
- add retries for web
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
---------
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: 86e45edcfb087d18e0e957ad6df9a6105dcd5770
* [clsi] tweak logging for clsi-cache
- Use `clsi-cache` identifier on log line
- Add shard to context
- Record nFiles on "too many entries for tar" error
* [clsi] do not trip clsi-cache circuit breaker on ENOENT errors
These can happen when an output/compile-dir is purged while we download
files.
GitOrigin-RevId: ffa73ef312bce5232ef72e3b81966bb6e14d2255
* [monorepo] enable caching for eslint/prettier/stylelint
* [monorepo] speed up prettier by swapping --list-different for --check
--list-different will print each file that it processes. We have a lot
of files in the monorepo. Using --check only prints mismatching files.
Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>
* [monorepo] explicitly configure prettier cache-location
This is the default location. Prettier will only discover that location
if the top level node_modules folder is writable, which is not the case
in CI. We create the .cache folder outside of docker, writable to node
inside docker.
The proper fix would be in prettier, to only check for write access in
the cache folder. Something to raise/upstream another day.
* [monorepo] run top-level format/format_fix in a single container
With the cache in place, it is much faster to use a single container.
As there is a single shared cache file, concurrent processes may see a
partially (re-)written cache file and bail out.
- all in a single container: 24s
- previous with -j4: 41s
- previous with -j8: failed due to corrupted cache file
---------
Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>
GitOrigin-RevId: 7850a3a980ae6c836393d97fe56a6316ffc3fa18
* Merge all .prettierignore files into top-level config
* Merge all .prettierrc files into top-level config
* Replace service-specific glob patterns in package.json format scripts with `prettier .`
* Add template files with Jinja2, Go template, envsubst, and Handlebars syntax to .prettierignore
* Ignore GitHub templates
* Ignore PUG templates to format them separately with `format:pug`
* Encourage double quotes for YAML, YML files
* Move prettier for PUG source format script to the root
* Move prettier for styles source format script to the root
* Remove prettier for jenkins files from web
* Remove prettier source format script from all services
* Make .prettierrc more readable
* Update format scripts by file type
* Organise `.prettierignore`
* Add `--cache` flag to prettier scripts for faster runs
* Format all files with prettier
* Format all or format services
* Remove `format`/`format:fix` scripts from services since now it runs from root `package.json`
* Avoid conlficts with yamllint configuration
* Remove `--cache` flag from prettier scripts
* Update all service Makefiles to use root-level prettier configuration
* Update all Jenkinsfile to use root-level prettier configuration
* Ignore auto-generated files by build_scripts
* Update package-lock.json
* Update root Makefile format targets
* Update SP Jenkinsfile format target
* Update E2E Makefile format script
* Udpate `format_js` to work in both local and CI env
* Add docker-mailtrap to .prettierignore
docker-mailtrap is a third-party git-ignored directory used for testing
* Added Docker env detection to prevent nested Docker spawning
* Ignore handlebars templates
* Add cryptographic files and test output to `.prettierignore`
* Add terraform modules to `.gitignore`
* Remove prettier-plugin-groovy
* Use npx directly instead of Docker for local formatting for faster formatting
* Auto-generate Makefiles
* Revert "Remove prettier-plugin-groovy"
This reverts commit 194a33589a2e1e4d2225d10c67e9f025e4222025.
* Mount monorepo root in RUN_LINT_FORMAT for prettier config access
* Prettier ignores all `node_modules` by default regardless of location
* Show only changed files in format output
* Ignore LICENSE files
* Enable prettier on rendered build_scripts outputs
* Ignoring all the template folders by prettier
* Remove the public/minjs entry since it does not exist
* Remove all non-existent paths
* Sync `.prettierignore` with ignored files by `.gitignore` and `.dockerignore` files
* Revert "Auto-generate Makefiles"
This reverts commit c0233e490de1bc95fe437219d65e0b66e0331ec9.
* Revert "Use npx directly instead of Docker for local formatting for faster formatting"
This reverts commit 1d2b2cf1a6c6974c76885852a90dd55e84167e41.
* Ignore dashboard JSON files
* Ignore files generated by bin/update_build_scripts
* Remove unsupported file types from `.prettierignore`
* Ignore test fixture generated files
* Ignore README file types by prettier
* Ignore generate snapshots by prettier
* Allow to format generated bin/update_build_scripts by prettier
* Ensure build script outputs prettier-compatible tsconfig.json
* Fix build script output to match prettier formatting
- Fix Jinja2 whitespace in docker-compose templates
- Change YAML quotes from single to double
* Don't read cryptographic files by prettier
* Ignore google verification files by prettier
* Revert npx prettier formatting
* Ignore domain verification files
* Show only changed files in format output
* Make `.github` prettier
* Allow all files to be formatted in jobs by prettier
* Allow server-ce/server-pro files to be formatted by prettier
* Ignore more folders in clsi, filestory, git-bridge by prettier
* Update build script with `RUN_LINTING_CI_MONOREPO`
* Ignore docker-mailtrap and downloads in server-ce by prettier
* Restore prettier configs and prettierignore for V1 since it has its own prettier (an older version)
* Source format
GitOrigin-RevId: 637adc3cc422d1f20c86d6ebc8ec514d60758287
* [k8s] clsi-cache: migrate to StatefulSet
* clsi-cache: optimize ILB services for GKE subsetting
Update the new clsi-cache internal load balancer services
to use optimal settings for GKE subsetting (NEG backends):
- set allocateLoadBalancerNodePorts: false (not needed with NEGs)
- set externalTrafficPolicy: Local (preserve source IP, keep traffic in zone)
- add trafficDistribution: PreferClose (zone affinity)
These settings ensure traffic from CLSI VMs stays within the same zone
when possible, reducing latency and cross-zone network costs.
* [k8s] clsi-cache: add missing resource paths
* [clsi] exclude readOnly clsi-cache shards
---------
Co-authored-by: Daniel Kontsek <daniel.kontsek@overleaf.com>
GitOrigin-RevId: 34f18b319a0e859ff149a135131c95a44bc674d6
* [k8s] clsi-cache: switch parent-app label to 'clsi-cache-legacy'
* [k8s] clsi-cache: add service account from kustomization.yaml
* [k8s] clsi-cache: consolidate on a single array of CLSI_CACHE_INSTANCES
* [clsi-cache] make prettier happy
GitOrigin-RevId: 4082a71df591904cfe437c4bde74759ddd83634c
* [clsi] remove all clsi-perf/health-check metrics
* [clsi] always emit E2E compile time metric
* [clsi] do not collect metrics for clsi-cache-template compiles
* [clsi] fix unit tests: request.metricsOpts always exists
* [clsi] use a gauge for the e2e compile time metric of clsi-perf
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
* [clsi] remove metrics for binary file downloads from clsi-perf
---------
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: 7995512e57c802086350e3d1a0ec5213ecdf0a05
* [clsi] try harder at sending files off to a working clsi-cache shard
* [clsi] use a crc for generating a stable sequence of shards to try
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [clsi] gradually migrate to crc based shard assigment
* [clsi] tweak selecting clsi-cache shard from crc
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [clsi] bump rollout dates of new clsi-cache shard change
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: 9386e170503b405580e4d0a8641832f3fcb1fa83
* [monorepo] record ERROR/FATAL log messages in junit report
* [web] put SaaS specific code behind feature flag
* [web] use split test cache for getting user assignments
The unit tests needed updating as they did not replicate any of the
mongo filtering. The acceptance tests cover this logic.
* [web] make better use of existing indexes
* [web] avoid col-scan in tests of notifications module
* [web] remove cleanup of empty feedbacks collection
* [web] add assertion for reason of rejected request in launchpad test
* [web] add missing indexes
* [web] enable mongo notablescan
* [web] make emailNotifications tests compatible with notablescan
GitOrigin-RevId: b888f2feeb3a0e915f068ae1c4ea23ec17821221
* [clsi] fix circuit breaker for clsi-cache
* [clsi] enable ts-check for CLSICacheHandler
* [clsi] limit the number of .blg files in clsi-cache to 50
* [clsi-cache] limit the number of files per job to 100
* [clsi-cache] explain early registration of buildId
* [clsi-cache] lock down downloads via nginx to project folder
GitOrigin-RevId: 081d0c40b08db3a384c4d765b71a50b973f42151
* Refactor v1 Makefile to use DOCKER_COMPOSE variable for Docker commands with tag safe branch name
* Handle slashes in branch names for docker build tags
GitOrigin-RevId: 463940e8435845978aced745575905f3bfbb8e1c
* [clsi] gracefully handle fast exit of synctex/wordcount containers
* [clsi] do not change container options in-place for logging
GitOrigin-RevId: 0b685310a3c72f8f46125fefaa30c1ddb19e7b07
Stage timeouts:
- frontend waits 5s
- web/clsi waits 4s
- clsi-cache waits 3s
This should ensure that the frontend can receive a valid response after
any of the backend requests failed.
The circuit breaker will remain closed for TIMEOUT + jitter of 0-3 times
the TIMEOUT of the respective service. This should avoid the bulk of
traffic to fail and occasionally issue retries without hammering the
instances while down.
Also do not try the next backend when the abort signal has expired.
GitOrigin-RevId: d612125616a9e416beff2f4c6d7f30066b5b9d6d
* [clsi] add stats and timings to compile response from clsi-cache
* [clsi] set downloadedFromCache when previously downloaded for synctex
Assumption: every compile will emit an output.log. When the output.log
is missing, but the output.synctex.gz exists, it must have been
downloaded from the cache.
GitOrigin-RevId: 41ea34880931e3c43dda3bc9eb26c0d02054894d
* fix: correct typedef for Document in helpers.mjs
* add move-migrations codemod
* update migration paths to use shared migrations directory
* move migrations to shared location
* fix: update Dockerfile and docker-compose.ci.yml to include migrations directory
* feat: add migrations tool to workspaces in package.json
* [monorepo] Fix order of docker ignore rules
* [web] remove unused docker ignore file
* [monorepo] replace old references to migrations folder
* [server-ce] copy migrations from new place
* [migrations] Inline web scripts
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [migrations] move three web scripts over
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [migrations] add missing collection
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [migrations] remove lodash dependency
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [migrations] avoid mongodb-legacy dependency
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [monorepo] run migrations from tools/migrations
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [migrations] simplify migration for adding gitBridge feature to users
* [monorepo] run migrations from tests in all the services
* [migrations] add Jenkins pipeline for linting/formatting
* [monorepo] fixup running web migrations everywhere
* [monorepo] trigger Jenkins builds on changes to mongo migrations
* [migrations] add Jenkins pipeline for linting/formatting
* [monorepo] build scripts: update devDependencies before deps scanning
* [monorepo] build scripts: formerly depend on tools/migrations
* [monorepo] run eslint on .mjs files
* [migrations] enable more eslint rules and fix all the errors
* [rake] fix migrations:list task
---------
Co-authored-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
GitOrigin-RevId: 14cf69cc1b9405bbc75adbb9a000e555500e0614
* [monorepo] remove docker-repos flag from build scripts
* [monorepo] use content hash as docker image cache key
* [packer] jenkins-worker: populate build cache for all services
* [v1] adopt smarter docker caching
* [latexqc] adopt smarter docker caching
* [monorepo] refresh docker cache in Jenkins
* [packer] jenkins-worker: increase disk size
* [monorepo] run jenkins_docker_build_cache as Jenkins user for v1
* [monorepo] define MONOREPO at the top of all the Makefiles
* [monorepo] add --build-arg BUILDKIT_INLINE_CACHE=1 everywhere
GitOrigin-RevId: 8110c6e68727a52ea80f3ec71711d30e17d69499
