Commit Graph

1141 Commits

Author SHA1 Message Date
Jakob Ackermann c560f7ffde [clsi] prepare for clsi-cache survey (#29274)
* [clsi] add stats and timings to compile response from clsi-cache

* [clsi] set downloadedFromCache when previously downloaded for synctex

Assumption: every compile will emit an output.log. When the output.log
is missing, but the output.synctex.gz exists, it must have been
downloaded from the cache.
GitOrigin-RevId: 41ea34880931e3c43dda3bc9eb26c0d02054894d
2025-10-23 08:05:43 +00:00
Eric Mc Sween 6a5b06f77e Merge pull request #29176 from overleaf/em-clsi-image-timings
CLSI: Capture image processing timings
GitOrigin-RevId: 28c2f73f260f2e82a64751bb46655e7546a458ef
2025-10-20 08:05:42 +00:00
Brian Gough 89898b0e9a move migrations to shared location (#28306)
* fix: correct typedef for Document in helpers.mjs

* add move-migrations codemod

* update migration paths to use shared migrations directory

* move migrations to shared location

* fix: update Dockerfile and docker-compose.ci.yml to include migrations directory

* feat: add migrations tool to workspaces in package.json

* [monorepo] Fix order of docker ignore rules

* [web] remove unused docker ignore file

* [monorepo] replace old references to migrations folder

* [server-ce] copy migrations from new place

* [migrations] Inline web scripts

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [migrations] move three web scripts over

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [migrations] add missing collection

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [migrations] remove lodash dependency

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [migrations] avoid mongodb-legacy dependency

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [monorepo] run migrations from tools/migrations

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [migrations] simplify migration for adding gitBridge feature to users

* [monorepo] run migrations from tests in all the services

* [migrations] add Jenkins pipeline for linting/formatting

* [monorepo] fixup running web migrations everywhere

* [monorepo] trigger Jenkins builds on changes to mongo migrations

* [migrations] add Jenkins pipeline for linting/formatting

* [monorepo] build scripts: update devDependencies before deps scanning

* [monorepo] build scripts: formerly depend on tools/migrations

* [monorepo] run eslint on .mjs files

* [migrations] enable more eslint rules and fix all the errors

* [rake] fix migrations:list task

---------

Co-authored-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
GitOrigin-RevId: 14cf69cc1b9405bbc75adbb9a000e555500e0614
2025-10-16 08:07:37 +00:00
Eric Mc Sween 3ddf88a031 Merge pull request #29106 from overleaf/bg-fix-capdrop-in-docker-runner
fix capdrop in docker runner

GitOrigin-RevId: 1e8c81723a9e152ec85a3a2776965891fbe07606
2025-10-16 08:06:47 +00:00
Eric Mc Sween 8a8a0fbf3c Merge pull request #29085 from overleaf/em-metrics-dependencies
Log noteworthy dependencies in the CLSI performance log

GitOrigin-RevId: 8412251e0cc77f305867d645ad5d9d3bbb9b0890
2025-10-16 08:06:42 +00:00
Eric Mc Sween 5ccedbc528 Merge pull request #29076 from overleaf/em-metrics-compile-passes
Add number of passes to compile metrics

GitOrigin-RevId: b9a6b6691f2feb7f376cd1bb94c81ecb7c3bc580
2025-10-16 08:06:37 +00:00
Eric Mc Sween c1f375a867 Merge pull request #29117 from overleaf/jpa-clsi-logging
[clsi] log high level details for large clsi-cache requests

GitOrigin-RevId: e368d745554c925a665f8794514cc8bfed78b7b3
2025-10-16 08:06:32 +00:00
Jakob Ackermann 81c50f5279 [monorepo] compute IMAGE_CACHE lazily (#29087)
GitOrigin-RevId: de287c89c1cea858d0609265945266be19d7ea26
2025-10-15 08:07:47 +00:00
Jakob Ackermann 79eabd511e [monorepo] use content hash as docker image cache key (#29046)
* [monorepo] remove docker-repos flag from build scripts

* [monorepo] use content hash as docker image cache key

* [packer] jenkins-worker: populate build cache for all services

* [v1] adopt smarter docker caching

* [latexqc] adopt smarter docker caching

* [monorepo] refresh docker cache in Jenkins

* [packer] jenkins-worker: increase disk size

* [monorepo] run jenkins_docker_build_cache as Jenkins user for v1

* [monorepo] define MONOREPO at the top of all the Makefiles

* [monorepo] add --build-arg BUILDKIT_INLINE_CACHE=1 everywhere

GitOrigin-RevId: 8110c6e68727a52ea80f3ec71711d30e17d69499
2025-10-15 08:06:41 +00:00
Jakob Ackermann abe568c300 [monorepo] fixes for Jenkins (#29027)
* [packer] jenkins-worker: pull TL2017 for clsi

* [tpdsworker] add stub file for test/setup.js

* [linked-url-proxy] update package-lock.json

* [packer] jenkins: discover common Jenkinsfiles from jobs and libraries

* [linked-url-proxy] fix mocharc path for mocha-multi-reporters

* [linked-url-proxy] install prettier and eslint inside docker

* [linked-url-proxy] make prettier happy

* [linked-url-proxy] add patch for mocha-multi-reporters

* [clsi] pull TL2017

* [monorepo] add bind mount for reports folder for unit tests

GitOrigin-RevId: d0d3d92f963c15549125a298ca4fd50d8a0e83da
2025-10-14 08:05:48 +00:00
Jakob Ackermann edba4eefa4 [monorepo] migrate build scripts to Jenkins (#29019)
* [monorepo] migrate build scripts to Jenkins

* [fraud-dash] fix integration with monorepo

GitOrigin-RevId: 6a30ec5ea68a1eae9b663faf9ccdc741ec7f9cf8
2025-10-13 08:07:11 +00:00
Eric Mc Sween 3ccae0e852 Merge pull request #28992 from overleaf/em-compile-metrics-runs
Add metric measuring the execution time of each latexmk rule

GitOrigin-RevId: fcb7215f7f53063e6fe046c01bbcc81e6441c064
2025-10-13 08:07:07 +00:00
Eric Mc Sween 1b544263fd Merge pull request #28909 from overleaf/em-compile-metrics
Use histograms to track CLSI compile times

GitOrigin-RevId: cf25f1e6d2094186f419acc70748f0c71b6c3240
2025-10-13 08:07:02 +00:00
Tim Down 865ce8c3ff Merge pull request #28541 from overleaf/td-notifications-app-ts
Change notifications app.js to TypeScript

GitOrigin-RevId: cb6195e2e8c8cd89e1a954bfcb1911929440d6ca
2025-10-13 08:05:36 +00:00
Brian Gough 97cabd4fea Merge pull request #28988 from overleaf/bg-add-file-info-to-performance-logs
add latexmk fdb file info to performance logs

GitOrigin-RevId: 3cc5709cd10fd55c2cd8aff7754fb7868aacdf0c
2025-10-13 08:05:23 +00:00
Brian Gough f4b42d7bd0 Merge pull request #28959 from overleaf/bg-exclude-health-checks-from-performance-logs
exclude health checks from performance logs

GitOrigin-RevId: 88db63e00b32b2b015ee25c7d555546ed7d9a95b
2025-10-13 08:05:18 +00:00
Brian Gough f67d5fe006 Merge pull request #28880 from overleaf/bg-add-time-option-to-clsi
add latexmk `-time` option to clsi and record performance logs

GitOrigin-RevId: 467473859359913da73f83e10b63b45603ea175c
2025-10-09 08:06:12 +00:00
Miguel Serrano 98bb3a5ba8 Merge pull request #28779 from overleaf/msm-clsi-loadtest-async-await
[clsi] Replaced callbacks with async/await in `loadTests`

GitOrigin-RevId: 81e84dd77f71560f765625dfdbeafcf14312a3ff
2025-10-09 08:05:16 +00:00
Miguel Serrano 919a0a17e8 Merge pull request #28826 from overleaf/msm-bump-dockerode-4-0-9
[clsi] Bump dockerode 4.0.7 -> 4.0.9

GitOrigin-RevId: ec07c7c1d9e95f415b528a7b61b390f95014ea15
2025-10-08 08:05:45 +00:00
Miguel Serrano a0f0641cfd Merge pull request #28752 from overleaf/msm-clsi-acceptance-async-await
[clsi] async/await migration in acceptance tests

GitOrigin-RevId: d614fabb6d568dc5c955603fb923fb40b871a703
2025-10-02 08:06:04 +00:00
Jakob Ackermann d1b531d2f2 [monorepo] migrate monorepo checks to Jenkins (#28765)
GitOrigin-RevId: ebc3db95cebe99226f8de0f66c1830eb5d78e26c
2025-10-01 08:06:28 +00:00
Brian Gough 613906531e Merge pull request #28719 from overleaf/dependabot-npm_and_yarn-tar-fs-3.1.1
Bump tar-fs from 3.0.9 to 3.1.1

GitOrigin-RevId: abbdf1a27eb9b22aba3fa3ce68b8edd3a7944547
2025-09-29 08:05:47 +00:00
Andrew Rumble 0b3b97e93d [monorepo] format ts files (#28678)
* Add ts files to format package script

* Apply changes to service package.json files

* [monorepo] only support plain .ts files in all the services

---------

Co-authored-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
GitOrigin-RevId: 882cf274da86a0dbe1fff395441f6f2458c1405c
2025-09-25 08:05:30 +00:00
Jakob Ackermann 4febc27b05 [monorepo] optimize Jenkins build triggers for services (#28477)
Co-authored-by: Andrew Rumble <andrew.rumble@overleaf.com>
Co-authored-by: Mathias Jakobsen <mathias.jakobsen@overleaf.com>
GitOrigin-RevId: ea7d6c435d4a81a0f42ea9b608a0cc79087cdffd
2025-09-24 08:06:43 +00:00
Jimmy Domagala-Tang 38c702ddd4 add persistent directory for rolling builds texlive location (#28563)
GitOrigin-RevId: ea131bc99f27be32055d40a92a967f524f29d02d
2025-09-23 08:07:59 +00:00
Jakob Ackermann 19b3ded887 [monorepo] run prettier on Jenkinsfiles (#28565)
* [monorepo] consolidate .editorconfig files at the root

* [monorepo] run prettier on Jenkinsfiles

* [saas-e2e] increase timeout for XeTeX compile

GitOrigin-RevId: 48aa82f7c81611899837753ae92c7732998d4ca5
2025-09-19 08:07:49 +00:00
Miguel Serrano f17ee6a9fb Merge pull request #28138 from overleaf/msm-remove-volumes-dockerode
[clsi] Remove `Volumes` from container options

GitOrigin-RevId: 53a60f69e9689ee777d9b300127885de7b88c1fb
2025-09-01 08:05:03 +00:00
Eric Mc Sween df1be93ec3 Merge pull request #27936 from overleaf/em-unit-tests-mongo
Make Mongo available to unit tests in all services

GitOrigin-RevId: b65bbb69883d5bba31d09802b89f35bdc523fe4d
2025-08-25 08:05:19 +00:00
Andrew Rumble c4ef5cf262 [monorepo] Upgrade to node 22.18.0 (#27977)
* Upgrade to node 22.18.0

* Switch projects-data fixture to strip-types compatible form

GitOrigin-RevId: 4a7ec56f4a82395d2be8870526df0c679de38ab5
2025-08-22 08:05:17 +00:00
Jakob Ackermann 07eb75fb33 [monorepo] extend cleanup in Jenkins VMs (#27793)
* [monorepo] extend docker compose cleanup

* [web] cleanup docker images

Only retain the branch builds for dependencies.

* [server-ce] jenkins: remove build specific image

Retain the branch image as cache.

* [monorepo] jenkins: prune untagged docker images and docker build cache

* [monorepo] jenkins: remove temporary files from monorepo

* [monorepo] jenkins: remove containers from sandboxed compiles

GitOrigin-RevId: c847015986a060c5c84c39cd06236fd73bbf6f3a
2025-08-12 08:06:27 +00:00
Alf Eaton cc6bb54226 Remove "version" from docker-compose files (#27494)
GitOrigin-RevId: 8d9a0e58e3a9c6c4296e4585115d1b259e26a6cd
2025-07-31 08:05:10 +00:00
Jakob Ackermann 812a2704fa [clsi-cache] meter ingress and egress bandwidth (#27143)
* [mics] fix "app" label in clsi-cache metrics in dev-env

* [clsi-cache] validate filePath when processing file

* [clsi-cache] meter ingress and egress bandwidth

Files are downloaded directly from nginx, hence we cannot meter egress
in clsi-cache easily.

GitOrigin-RevId: 24de8c41728f0e9c984113c1470dec6153e75f20
2025-07-16 08:05:59 +00:00
Brian Gough f45031c6f9 Merge pull request #26637 from overleaf/bg-clsi-fix-process-group-for-local-compiles
fix "stop compile" option for local command runner in CE/SP

GitOrigin-RevId: 7986b505362aaf33ac6e161b3b54458baba1e2e6
2025-07-11 08:07:11 +00:00
Andrew Rumble 907bc8d3ca Update node to 22.17.0
GitOrigin-RevId: 7a15f7a0e95e3331c589955aca1fbc5dfb13a568
2025-07-09 08:06:01 +00:00
Andrew Rumble 0008eca842 Remove @types/workerpool
This is a stub

GitOrigin-RevId: de776c11044c42bb191424f1d614442da22c3197
2025-07-04 08:04:51 +00:00
Jakob Ackermann 1facb76e54 [clsi] adopt Server Pro env vars for sandboxed compiles (#26430)
GitOrigin-RevId: b7c0d9f4ff37d6cef77694a6a9ea4d50dcb148b2
2025-06-16 08:06:50 +00:00
Jakob Ackermann c72ae362dc [misc] migrate remaining references to our GCR repositories to AR (#26370)
* [misc] migrate remaining references to our GCR repositories to AR

* [server-ce] fix retagging of texlive images

GitOrigin-RevId: 81f955ad4c4486ad42b29cbd6bcc9d5ef4b1a432
2025-06-13 08:06:18 +00:00
Miguel Serrano 71bf86a152 Merge pull request #26218 from overleaf/msm-bump-tar-fs-multer
[clsi/web/history-v1] Bump `tar-fs` and `multer`

GitOrigin-RevId: c76b964224c8367d68dc1190ff29627cc6919ade
2025-06-10 08:06:05 +00:00
Jakob Ackermann 2107805c47 [clsi] add env var override for seccomp profile (#25894)
GitOrigin-RevId: 6ef9a5c1f9149147641abb9fe1798b1b41a14c05
2025-05-23 13:25:58 +00:00
Antoine Clausse 611999e1ae Update Node to 22.15.1 (#25785)
GitOrigin-RevId: 52428d2d7e67c3135a1604fa487dd142aa08bf15
2025-05-22 08:07:42 +00:00
Antoine Clausse 79e58a79a7 Revert "Update defaultHighWaterMark to 64KiB (Node 22's default) (#25522)" (#25789)
This reverts commit 19d731abf683066654027de3a4f9ac0b8916f22c.

GitOrigin-RevId: eb7c45ab45e02054601b607a4bfeb432424a1837
2025-05-22 08:07:38 +00:00
Jakob Ackermann 09057984bf [clsi] mark VM as unhealthy when detecting of-of-disk condition (#25721)
* [clsi] shed load when detecting out-of-disk condition

* [clsi] mark VM as unhealthy when detecting of-of-disk condition

GitOrigin-RevId: 25cda6785c0d973f50ec6206bee389804f35917e
2025-05-21 08:05:34 +00:00
Jakob Ackermann 142da2c850 [clsi] avoid creating legacy db folder (#25760)
GitOrigin-RevId: eaa97813f0e20d14d7d6dafefe7b8b4d002fa41c
2025-05-20 08:06:36 +00:00
Antoine Clausse 108a0a7c0d Update Node to 22.15.0 (#24699)
* Run `bin/update_node 20.18.2 22.15.0`

* Remove expects on `fetchMock.callHistory.done()` to fix tests: are they necessary?

* Set node version to `22.x` in linked-url-proxy

* Increase test timeout to 30s in `github-sync`, Add waiting steps

* Define `navigator.onLine` in tests setup

GitOrigin-RevId: 75eb556e9f51b665e57497a0879b6915d14069ce
2025-05-16 08:06:26 +00:00
Jakob Ackermann 291dd2fa85 [clsi-cache] remove non sharded instances (#25645)
* Revert "[clsi-cache] only use sharding from updated project editor tabs (#25326)"

This reverts commit 1754276bed3186c0536055c983e32476cc90d416.

* [clsi-cache] remove non sharded instances

GitOrigin-RevId: aa3ac46140dfc1722a3350cf7071e5b11af61199
2025-05-16 08:05:02 +00:00
Antoine Clausse 7242b554c1 Update defaultHighWaterMark to 64KiB (Node 22's default) (#25522)
* Set defaultHighWaterMark to 16KiB

This is already the default in Node 20

* Set defaultHighWaterMark to 64KiB

Per https://github.com/overleaf/internal/pull/25522#issuecomment-2872035192

GitOrigin-RevId: 19d731abf683066654027de3a4f9ac0b8916f22c
2025-05-14 08:05:16 +00:00
Christopher Hoskin 5679083c19 Merge pull request #25543 from overleaf/csh-issue-23942-more-migrate
Remove more CR refs

GitOrigin-RevId: 7ba91712943c309d40613b0f7247f20d45d7fbb1
2025-05-13 08:08:22 +00:00
Jakob Ackermann 6244616fc1 [web] avoid trying to fetch synctex.gz from clsi-cache in free projects (#25445)
* [web] avoid trying to fetch synctex.gz from clsi-cache in free projects

* [clsi] parse boolean query parameter

GitOrigin-RevId: 99c98aac8147a626b704e9a888b7fc660cc5ab17
2025-05-12 08:05:24 +00:00
Jakob Ackermann 35adaf19b6 [web] emit event when synctex mapping was downloaded from clsi-cache (#25424)
* [clsi] tell frontend when synctex mapping was downloaded from clsi-cache

* [web] emit event when synctex mapping was downloaded from clsi-cache

GitOrigin-RevId: 1f6b7e0faaa7dd76449aad566802da971a4cf9ed
2025-05-09 08:06:00 +00:00
Jakob Ackermann e673b1f3e6 [clsi] enable keepAlive on global HTTP agents (#25350)
GitOrigin-RevId: c9478b405ac32ca55aeb3bcf9f24052477464667
2025-05-07 08:08:10 +00:00