* [clsi-cache] remove old DiskCrawler
* [clsi-cache] use helpers for populating state during initial crawl
* [clsi-cache] use path helpers everywhere
* [clsi-cache] create project folder as part of the initial crawl
* [clsi-cache] add sharded disk layout
* [k8s] clsi-cache: switch clsi-cache-zone-b-shard-0 to new disk layout
* [clsi-cache] tweak logging context
* [monorepo] add bin/format_project_key util
* [k8s] clsi-cache: use a single nginx config file
GitOrigin-RevId: bb49146502cc6201eef210efae33355321efd77b
* [clsi] try harder at sending files off to a working clsi-cache shard
* [clsi] use a crc for generating a stable sequence of shards to try
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [clsi] gradually migrate to crc based shard assigment
* [clsi] tweak selecting clsi-cache shard from crc
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [clsi] bump rollout dates of new clsi-cache shard change
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: 9386e170503b405580e4d0a8641832f3fcb1fa83
* [monorepo] record ERROR/FATAL log messages in junit report
* [web] put SaaS specific code behind feature flag
* [web] use split test cache for getting user assignments
The unit tests needed updating as they did not replicate any of the
mongo filtering. The acceptance tests cover this logic.
* [web] make better use of existing indexes
* [web] avoid col-scan in tests of notifications module
* [web] remove cleanup of empty feedbacks collection
* [web] add assertion for reason of rejected request in launchpad test
* [web] add missing indexes
* [web] enable mongo notablescan
* [web] make emailNotifications tests compatible with notablescan
GitOrigin-RevId: b888f2feeb3a0e915f068ae1c4ea23ec17821221
* [clsi] fix circuit breaker for clsi-cache
* [clsi] enable ts-check for CLSICacheHandler
* [clsi] limit the number of .blg files in clsi-cache to 50
* [clsi-cache] limit the number of files per job to 100
* [clsi-cache] explain early registration of buildId
* [clsi-cache] lock down downloads via nginx to project folder
GitOrigin-RevId: 081d0c40b08db3a384c4d765b71a50b973f42151
* [clsi] gracefully handle fast exit of synctex/wordcount containers
* [clsi] do not change container options in-place for logging
GitOrigin-RevId: 0b685310a3c72f8f46125fefaa30c1ddb19e7b07
* [migrations] make old migrations compatible with --notablescan in mongo
Re-running the migration for db.users.emails.reversedHostname on all
Server Pro/CE users will not be harmfull.
* [monorepo] enable mongo notablescan for all services but web
GitOrigin-RevId: e2a710c9de08de9ac2aa9df1f3082d5f1c33ea05
* [web] tweak Jenkins pipeline following move to faster CI VMs
The webpack build time has been reduced by about 30%. Adjust the sleep
statement for delaying the webpack build accordingly.
Push the production docker image layers from the parallel steps already.
Use a shared "scratch" tag to avoid generating cruft per branch/build.
* [web] avoid sharing the cypress/downloads folder in CI
GitOrigin-RevId: 52fdf420ec04fd26e3823ff9fca8c52f7a7764d7
Stage timeouts:
- frontend waits 5s
- web/clsi waits 4s
- clsi-cache waits 3s
This should ensure that the frontend can receive a valid response after
any of the backend requests failed.
The circuit breaker will remain closed for TIMEOUT + jitter of 0-3 times
the TIMEOUT of the respective service. This should avoid the bulk of
traffic to fail and occasionally issue retries without hammering the
instances while down.
Also do not try the next backend when the abort signal has expired.
GitOrigin-RevId: d612125616a9e416beff2f4c6d7f30066b5b9d6d
* [web] add clsi-cache prompts
* [web] add new editor variant to segmentation
* [web] add tests for useNewEditorVariant
* [web] adjust start of using clsi-cache in split-test
GitOrigin-RevId: c9c5b1eff2ceefb65ef82516d9074cb971cb4c48
* [clsi] add stats and timings to compile response from clsi-cache
* [clsi] set downloadedFromCache when previously downloaded for synctex
Assumption: every compile will emit an output.log. When the output.log
is missing, but the output.synctex.gz exists, it must have been
downloaded from the cache.
GitOrigin-RevId: 41ea34880931e3c43dda3bc9eb26c0d02054894d
* [monorepo] switch from cypress-reports to reports
* [saas-e2e] collect junit reports from new editor tests
* [saas-e2e] simplify setup for semaphore variable in Jenkins pipeline
We do not retry pipelines anymore.
* [saas-e2e] put new editor tests into their own namespace
GitOrigin-RevId: 1869310bf58f0b5b0081dcae2f16ec1a267caec6
* [monorepo] remove docker-repos flag from build scripts
* [monorepo] use content hash as docker image cache key
* [packer] jenkins-worker: populate build cache for all services
* [v1] adopt smarter docker caching
* [latexqc] adopt smarter docker caching
* [monorepo] refresh docker cache in Jenkins
* [packer] jenkins-worker: increase disk size
* [monorepo] run jenkins_docker_build_cache as Jenkins user for v1
* [monorepo] define MONOREPO at the top of all the Makefiles
* [monorepo] add --build-arg BUILDKIT_INLINE_CACHE=1 everywhere
GitOrigin-RevId: 8110c6e68727a52ea80f3ec71711d30e17d69499
The rake deploy:app task is expecting a build for all web changes.
Triggering a few extra Server Pro E2E test runs is cheaper than making
the rake task smarter.
GitOrigin-RevId: a8380417d4a323b3d2a008eeb6ee26bc0ad151d9
* [libraries] move all the unit tests into test/unit
* [settings] remove empty test file
* [mongo-utils] remove test setup as there are no tests
* [libraries] ensure that all libraries with tests have test/setup.js
* [monorepo] remove test step from Cloud build for libraries without tests
GitOrigin-RevId: b9a0a8a3330db7d8646c7a272b03fbb8c5e44bc0
* [monorepo] refactor retries in Jenkins to step level
Change the junit reports to use step specific file names. The [hash]
template option was neat in getting unique file names, but results in
duplicate test reports on retry.
* [patches] add support for .cjs config files for mocha-multi-reporters
GitOrigin-RevId: 3a749441470b5ba633e71319589606cfbe860952
shouldFlushHistoryOps has a default value for 'threshold', which keeps
the exports simpler and still lets the unit tests override it.
GitOrigin-RevId: 1c6d4a2778052b5af40e2e338589a230ac2f4646
* [server-ce] tests: migrate host-admin to ESM, zod and npm-workspaces
* [server-ce] test: use import.meta.dirname
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
* [server-ce] test: fix zod schema for docker compose endpoint
---------
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: d490948693f341210c0ab5c2947db7c9a17775ef
* [web] add junit integration for Jenkins
* [web] integrate junit test report for writefull into Jenkins
* [web] put all the junit test results into namespaces
GitOrigin-RevId: ba1ff07b5ea7bcfa97bb4d6bf7fa9e5291ab7b0f
* [monorepo] consolidate .editorconfig files at the root
* [monorepo] run prettier on Jenkinsfiles
* [saas-e2e] increase timeout for XeTeX compile
GitOrigin-RevId: 48aa82f7c81611899837753ae92c7732998d4ca5
* [monorepo] rebuild lezer output as part of 'make install'
* [monorepo] run 'make install' with high concurrency in Jenkins
* [monorepo] rebuild lezer parser output once
* [monorepo] headless npm run
* [web] update docs for lezer
GitOrigin-RevId: f385bb8a9cdab1a6bfc28a8843b5128ceec364ec
* [history-v1] add endpoint for getting blob stats of projects
* [web] display project history blob size on admin pages
* [web] break down history storage size by text and binary blob
GitOrigin-RevId: bfa4d56cf2c503d03005c13a0f7ac38158156cd2
* [history-v1] add script for listing owned projects with size
* [history-v1] handle project with no blobs record
* [history-v1] add plain sumBytes for easy sum in spreadsheet
GitOrigin-RevId: c20af189c223f867a6033cfb904f92a3a10c7cfb
When reconfiguring fails, the config snapshot in the frontend is not
updated. When the next test uses the previous config, it will not
attempt to restore that config again, and instead use the other config.
Reset the config before making changes to ensure all tests get their
desired config, even when reconfiguring fails.
GitOrigin-RevId: 0939e709cf93da820a86fef2599940669138d728
* [history-v1] add migrations record after full binary file migration
* [server-pro] add hotfix 5.5.5
* [server-ce] test: build hotfix 5.5.5 and use it in tests
GitOrigin-RevId: fb84e5710c59f466a3305de5f32f78e0ac9ce15d
* [web] reduce noise from CollectPayPalPastDueInvoiceTest
* [web] reduce noise for frontend tests that use Range.getClientRects
* [web] avoid nesting <li> elements in ProjectToolsMoreDropdownButton
RenameProjectMenuItem/CopyProjectMenuItem will return <li> elements.
* [web] avoid clicking on links that navigate away in JSDom
* [web] add mock for location change in JSDom
* [web] add fallback stripe subscription data for test
3.8k lines of log noise!
GitOrigin-RevId: 7adab182487a6c07ac9809c60802e498a0edb0ec
* [web] move sentry upload into a Makefile target
* [web] fix typos in jenkins pipeline
* [web] fix BRANCH_NAME for PR builds
This is currently "true". Just use PR-xxx for now.
* [web] remove explicit pull of main-deps for Jenkins
In Jenkins we do not need to wait for the cloud-builder image. The
docker build can start right away. Let docker validate the remote cache
and pull it if needed.
* [packer] jenkins-worker: prefetch sentry image
* [packer] jenkins-worker: switch to production AR
* [packer] jenkins-worker: wait for monorepo setup before pulling sentry
* [web] cloudbuild: fix directory for prefetch_sentry
GitOrigin-RevId: ec394bbf51efd516f0ade8813014dec9b63fb732
* [saas-e2e] adjust project name for new SpamSafe limits
* [saas-e2e] stay below 6 digits for spam safe project name
Co-authored-by: Antoine Clausse <antoine.clausse@overleaf.com>
---------
Co-authored-by: Antoine Clausse <antoine.clausse@overleaf.com>
GitOrigin-RevId: 77c039312c01b33aeb3b57661ec6ca5b88a0a36b
* [packer] jenkins-worker: install htop and lsof
* [web] trim tailing space in Jenkinsfile
* [web] cleanup after finishing Jenkins pipeline
* [server-ce] test: cleanup after finishing Jenkins pipeline
* [saas-e2e] add timeout to waiting for rail/web startup
* [monorepo] add 60s timeout for waiting on mongo to startup
* [saas-e2e] cleanup after finishing Jenkins pipeline
GitOrigin-RevId: 9954ab17b4db6a95c444e8e0b55cebb8d5daae59
* [web] add missing await statement in admin-panel tests
* [web] add audit log for changing admin access in Server Pro via SSO
GitOrigin-RevId: 69e994d0b2e0f0e0281ade80acbdf7fb30bd2e89
* [server-ce] tests: optimize sharding
* [server-ce] tests: remove shard CE_CUSTOM_2
* [server-ce] tests: move project sharing tests onto a new shard
Previously they were on their own CE_CUSTOM_1 shard.
GitOrigin-RevId: de256e04f5956b4651a5dc4ab47cdb5972723c5d
Jakob Ackermann