Commit Graph

154 Commits

Author SHA1 Message Date
Jakob Ackermann
205573c01a [web] fetch file-tree in fallback to old compile mode (#32403)
* [web] fetch file-tree in fallback to old compile mode

* [web] tweak unit test

GitOrigin-RevId: 1c5f8cb1f0cb15c91efc46babec343d228631019
2026-03-24 09:06:22 +00:00
Jakob Ackermann
81b7121408 [clsi] initial implementation of compile from history (#31883)
* [clsi] initial implementation of compile from history

* [clsi] copy changes

* [saas-e2e] extend test case with nested folder

* [saas-e2e] add test case for tracked changes

* [web] fix accumulating changes from multiple chunks

* [web] optimize size check for compile request payload

* [clsi] deduplicate globalBlobs

* [clsi] add validation for request body details

* [clsi] add metrics for compile from history

* [clsi] download binary files concurrently

* [clsi] skip download of empty file blob

* [clsi] break down e2e compile time metric by compileFromHistory

GitOrigin-RevId: 0dadef93e89d8a172c35cb130a1042d9d1bec42a
2026-03-06 09:12:07 +00:00
Jakob Ackermann
6c6e8d9a97 [monorepo] switch all output file reads to clsi-nginx (#31691)
* [monorepo] switch all output file reads to clsi-nginx

* [clsi-lb] allow gallery download requests

* [terraform] clsi: use nginx.conf from clsi service

* [clsi] fix flakey tests

* [clsi] replace alias with rewrite and root in nginx config

* [k8s] clsi-lb: expose download port on internal service

* [web] add explicit endpoint for downloading all output files

Serve the output.zip endpoint from clsi.

* [clsi] fix regex for latexqc submission ids

Previously, we only handled template submission ids.

GitOrigin-RevId: 6c3b21b01ec41ae767530b14aac31fbe3d640dd5
2026-02-24 09:07:12 +00:00
Jakob Ackermann
ddeafa1d7c [web] fallback to clsi-cache from regular output file download (#31555)
* [web] fallback to clsi-cache from regular output file download

* [web] add comment on req.params assignment

GitOrigin-RevId: 9d0c2e0a8e12d322aa5754c732a38f19adeac716
2026-02-17 09:05:08 +00:00
Brian Gough
dbff5ae559 Merge pull request #30813 from overleaf/bg-fix-compile-from-redis
exclude buildId from project state hash

GitOrigin-RevId: da7424378422bec348adeaa5da0fe2d3581744d9
2026-01-19 09:07:04 +00:00
Jakob Ackermann
14a7e935b8 [web] override timeout for a specific template (#30711)
* [web] override timeout for a specific template

* [web] fix junit integration for vitest

* [web] consolidate on JUNIT_ROOT_SUITE_NAME

* [web] fix unit test

GitOrigin-RevId: 3ba865e4b9c090c7bf8e07dc3d81e9bcee20d821
2026-01-15 14:19:33 +00:00
Jakob Ackermann
72ad614b25 [web] remove fallback for old clsi-server-id persistence (#30528)
GitOrigin-RevId: bd29aa1f77d61b5ff0c2c0a7d6fa893509c89e37
2026-01-13 09:06:43 +00:00
Andrew Rumble
f1e788d9b3 Merge pull request #29967 from overleaf/ar/web-remove-mocha-unit-tests
[web] remove mocha unit tests

GitOrigin-RevId: eda753af3470dbd1f385cb0bc3f06d78ade8a764
2025-12-03 09:05:37 +00:00
Andrew Rumble
07c827e9fd Merge pull request #29928 from overleaf/ar-last-infrastructure-conversions
[web] last infrastructure conversions

GitOrigin-RevId: ad1aff9b7df0610ed0303157d9e2c8032f32c02b
2025-11-28 09:05:56 +00:00
Andrew Rumble
18f44866e5 Merge pull request #29919 from overleaf/revert-29795-ar-last-infrastructure-conversions
Revert "[web] last infrastructure conversions"

GitOrigin-RevId: 48dc64553012afb5d2db4b2eb9c9898489b7e5ef
2025-11-27 09:05:54 +00:00
Andrew Rumble
d748d8d606 Merge pull request #29795 from overleaf/ar-last-infrastructure-conversions
[web] last infrastructure conversions

GitOrigin-RevId: 68aa11625a9bc6d0d5324ecd95bb5ac52af8ee96
2025-11-27 09:05:30 +00:00
Andrew Rumble
beb6f6d484 Merge pull request #29597 from overleaf/ar-last-features-esm-conversion
[web] last features esm conversion

GitOrigin-RevId: a35ab995bf654f1cdfe0e0062d8806761ecccf2d
2025-11-21 09:05:36 +00:00
Jakob Ackermann
3586b37491 [web] Remove clsi-cache-prompt/survey, split-tests and events (#29510)
* [web] Remove clsi-cache-prompt/survey and split-tests

* [web] Remove initial 50/50 clsi-cache split-test

* [web] Remove synctex-downloaded-from-cache event

* [web] Remove fallback-to-clsi-cache event

* [saas-e2e] fix tests with clsi-cache enabled

GitOrigin-RevId: b5cf2ab073dc866fe398b81fd5afe46422134c80
2025-11-05 09:06:59 +00:00
Jakob Ackermann
a80b69abe5 [web] move free compiles up one bracket (#29411)
GitOrigin-RevId: 613b703d677d91110cd6dcf085e9291008e05f5c
2025-11-03 09:05:23 +00:00
Jakob Ackermann
3a206074f2 [web] move premium compiles up one bracket (#29410)
GitOrigin-RevId: 53423f855b2c215f6cc926b480422b2ce3477a74
2025-10-31 09:06:32 +00:00
Jakob Ackermann
f3f6a8a8e0 [web] add clsi-cache prompts (#29281)
* [web] add clsi-cache prompts

* [web] add new editor variant to segmentation

* [web] add tests for useNewEditorVariant

* [web] adjust start of using clsi-cache in split-test

GitOrigin-RevId: c9c5b1eff2ceefb65ef82516d9074cb971cb4c48
2025-10-24 08:05:42 +00:00
Andrew Rumble
3d3be18f57 Features ESM conversion
GitOrigin-RevId: d659326723a90ac0789f4f7acc7f00aa9eaa63e2
2025-10-21 08:05:51 +00:00
Andrew Rumble
e909995ce0 Rename features files
GitOrigin-RevId: fc4ce037343b02d002fa07b7cdc283f2d7d4f89b
2025-10-21 08:05:47 +00:00
Jakob Ackermann
ad3e8bd65d [web] fix clearing of clsi persistence during migration to new key (#29172)
GitOrigin-RevId: 1d9e6eb87f8f27792fdc866e7dcc29b47e1b2547
2025-10-17 08:06:06 +00:00
Jakob Ackermann
706971ef41 [web] shard compile server persistence by compileBackendClass (#29156)
* [web] shard compile server persistence by compileBackendClass

* [web] make prettier happy

GitOrigin-RevId: d7cf8bde07e7110053d9d7531c007111af5cc46b
2025-10-17 08:06:02 +00:00
MoxAmber
986a728569 Merge pull request #28895 from overleaf/as-promisify-clsiformatchecker
Promisify ClsiFormatChecker and tests

GitOrigin-RevId: 89196708b97ff936660cc50a0b31f549d9723c4b
2025-10-17 08:05:40 +00:00
Jakob Ackermann
fbea855690 [web] switch dark compiles to n4 (#29133)
* [clsi-perf] run compiles on any backend with prefix clsi_compile

* [clsi-lb] add n4 backend

* [terraform] clsi: add n4 instances

* [web] switch dark compiles to n4

* [terraform] clsi: remove c4d instances

* [terraform] clsi: scale up n4 instances

GitOrigin-RevId: 8f2d98ef6c3c503366c92c4f148a8dd7394cebd2
2025-10-16 08:07:21 +00:00
Antoine Clausse
33e63d79fc Merge pull request #28584 from overleaf/ac-some-web-esm-migration-5
[web] Convert some Features files to ES modules (part 5)

GitOrigin-RevId: 0cad67f9afe0095e2b066bf2f4d3717c00540dab
2025-10-08 08:06:15 +00:00
Antoine Clausse
46715191e3 Merge pull request #28544 from overleaf/ac-some-web-esm-migration-4
[web] Convert some Features files to ES modules (part 4)

GitOrigin-RevId: cf11a7584e39c4d4de08e2f924240e488a4066c4
2025-09-24 08:05:58 +00:00
Antoine Clausse
71f0b28a84 [web] Convert some Features files to ES modules (part 3) (#28494)
* Rename files to mjs

* Rename test files to mjs

* Update CODEOWNERS

* Update files to ESM

* Update test files to ESM

* Update RestoreManager.test.mjs

* Remove unused `AdminAuthorizationHelper` mock and stub

* Remove unnecessary return

GitOrigin-RevId: 2b9ef126de1d8964afbc6e5641cca36712655866
2025-09-17 08:05:02 +00:00
Andrew Rumble
7f908915aa Merge pull request #28367 from overleaf/ar-migrate-routermjs-to-zod
[web] migrate router.mjs to zod

GitOrigin-RevId: d3fc21a11351f3e2deb5011cd1beeb86286a300b
2025-09-16 08:05:26 +00:00
Jakob Ackermann
16422f972b Merge pull request #28265 from overleaf/jpa-double-compile
[web] add c2d -> c4d double compile test

GitOrigin-RevId: 186cfa867d3db5786c6d4888dfe1ca2f46a5bb0c
2025-09-04 08:05:10 +00:00
Jakob Ackermann
03320bb377 Merge pull request #28264 from overleaf/jpa-synctex
[web] use standard request handling for SyncTeX requests

GitOrigin-RevId: ad5ba1834241d5939675f2533940ade741fc5abf
2025-09-04 08:05:06 +00:00
Jakob Ackermann
a85b2b34f5 Merge pull request #28233 from overleaf/jpa-clsi-lb-clear
[web] clear clsi server id/output files in both clsi-lb backends

GitOrigin-RevId: b73ccc2017800d9abbb8f571efeb34f51c9f96c1
2025-09-04 08:04:58 +00:00
Jakob Ackermann
bd4130c147 [filestore] remove user files endpoints (#28125)
* [filestore] remove user files endpoints

* [web] remove user files integration for filestore

GitOrigin-RevId: 565fa68a659c07420ee6141d0f276b4e4d2972e0
2025-09-02 08:04:52 +00:00
MoxAmber
fc5ae92be1 Merge pull request #27677 from overleaf/as-compile-timeout-enforcement
[web] 10s Compile Timeout - Enforcement Phase

GitOrigin-RevId: 3930eb376cc1293409259e073032218e09d5270e
2025-08-20 08:06:01 +00:00
Jakob Ackermann
29045b9085 [web] download binary files in clsi from filestore via new endpoints (#27505)
* [history-v1] use String.padStart instead of lodash.padStart

* [web] download binary files in clsi from filestore via new endpoints

* [server-ce] tests: Cypress.env() is parsing boolean values

* [server-ce] tests: run history migration as root

GitOrigin-RevId: bdf6c0e542531ccc4b3f13d2ed68ca0d31e580e9
2025-07-31 08:05:28 +00:00
Eric Mc Sween
17d1b0b8d6 Merge pull request #25646 from overleaf/em-ds-mobile-app-compile
Compile endpoint for the DS mobile app

GitOrigin-RevId: 2fd9f4a6e8c2ed4ee868b0c1293f6760b9d113c8
2025-05-21 08:06:36 +00:00
Jakob Ackermann
ec1bd69605 [clsi-cache] remove non sharded instances (#25645)
* Revert "[clsi-cache] only use sharding from updated project editor tabs (#25326)"

This reverts commit 1754276bed3186c0536055c983e32476cc90d416.

* [clsi-cache] remove non sharded instances

GitOrigin-RevId: aa3ac46140dfc1722a3350cf7071e5b11af61199
2025-05-16 08:05:02 +00:00
Antoine Clausse
bc4c3c4ef8 [web] Promisify ClsiCookieManager and CompileController (reapply again) (#25280)
* Reapply "[web] Promisify ClsiCookieManager and CompileController (reapply and fix)"

This reverts commit 98cb9127ff2b7c7c347c560766f749265d712490.

* Fix: Use query parameters correctly (!!)

* Add unit test on `checkIsLoadSheddingEvent`

* Remove interference between tests: rename to `ClsiCookieManager2` when it's re-sandboxed

* Add test: 'should report "cycle" when other is UP'

GitOrigin-RevId: 3146b149954b908830226cb03b51d9adfa08ec2e
2025-05-07 08:07:07 +00:00
Jakob Ackermann
6881ba956a [clsi-cache] only use sharding from updated project editor tabs (#25326)
GitOrigin-RevId: 1754276bed3186c0536055c983e32476cc90d416
2025-05-07 08:06:39 +00:00
Antoine Clausse
666481d8b2 Merge pull request #25221 from overleaf/revert-25207-ac-promisify-compile-controller-2
Revert "[web] Promisify ClsiCookieManager and CompileController (reapply and fix)"

GitOrigin-RevId: 4495b0fdee22c9c2a7eb39250aef498883389427
2025-05-01 08:06:55 +00:00
Antoine Clausse
61db35ac8f Merge pull request #25207 from overleaf/ac-promisify-compile-controller-2
[web] Promisify ClsiCookieManager and CompileController (reapply and fix)

GitOrigin-RevId: 0737f30c24bf92b33327dc7d0e015ac2cd7d751d
2025-05-01 08:06:47 +00:00
Antoine Clausse
d7d60f9d4c Merge pull request #25200 from overleaf/revert-25023-ac-promisify-compile-controller
Revert "[web] Promisify ClsiCookieManager and CompileController"

GitOrigin-RevId: 190ee8d2be23687f092e762c5199a34bcdf37cf9
2025-05-01 08:06:00 +00:00
Antoine Clausse
73476180d4 [web] Promisify ClsiCookieManager and CompileController (#25023)
* Refactor `ClsiCookieManager` to use async/await

* Refactor `CompileController` to use async/await

* Fix tests: CompileControllerTests.js

* Fix tests: ClsiCookieManagerTests.js

* Fixup: keep old object key (`serverId`->`clsiServerId`)

* Prefix non-express methods with an underscore

* Use async versions of `rclient` methods

* Fix: `canContinue` must be `false` when rate limited (!!)

* Remove unused `ClsiCookieManager.getCookieJar`

* Remove bad comments

* Fix linting

* Replace `request` by `@overleaf/fetch-utils`

* Replace `callsFake` by `resolves`

* Update `catch` block after request->fetch update: check status code

* Re-add timeout of 30s (was removed in 3df75ab5ccc8a63bd69a6a6e6052ef0451b76220)

* `npm run format:fix`

* Don't throw on 4xx errors; keep functionality similar to the current implementation using `request`.

GitOrigin-RevId: ef161f6c252d13f407e9cf28029b62061d6b992f
2025-05-01 08:05:42 +00:00
Andrew Rumble
2ad9f36706 Promisify tests
GitOrigin-RevId: 6f413f4c5ef8d034b4e94afacdf2d7b43c3a8830
2025-04-29 08:05:18 +00:00
Jakob Ackermann
f0edc7ba00 [web] update the projects lastUpdated timestamp when changing file-tree (#24867)
* [misc] freeze time before any other unit test setup steps

Freezing it after other work (notably sandboxed-module imports) will
result in flaky tests.

* [web] update the projects lastUpdated timestamp when changing file-tree

GitOrigin-RevId: b82b2ff74dc31886f3c4bd300375117eead6e0cd
2025-04-16 08:05:14 +00:00
Jakob Ackermann
b538d56591 [clsi-cache] backend (#24388)
* [clsi-cache] initial revision of the clsi-cache service

* [clsi] send output files to clsi-cache and import from clsi-cache

* [web] pass editorId to clsi

* [web] clear clsi-cache when clearing clsi cache

* [web] add split-tests for controlling clsi-cache rollout

* [web] populate clsi-cache when cloning/creating project from template

* [clsi-cache] produce less noise when populating cache hits 404

* [clsi-cache] push docker image to AR

* [clsi-cache] push docker image to AR

* [clsi-cache] allow compileGroup in job payload

* [clsi-cache] set X-Zone header from latest endpoint

* [clsi-cache] use method POST for /enqueue endpoint

* [web] populate clsi-cache in zone b with template data

* [clsi-cache] limit number of editors per project/user folder to 10

* [web] clone: populate the clsi-cache unless the TeXLive release changed

* [clsi-cache] keep user folder when clearing cache as anonymous user

* [clsi] download old output.tar.gz when synctex finds empty compile dir

* [web] fix lint

* [clsi-cache] multi-zonal lookup of single build output

* [clsi-cache] add more validation and limits

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [clsi] do not include clsi-cache tar-ball in output.zip

* [clsi-cache] fix reference after remaining constant

Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>

* [web] consolidate validation of filename into ClsiCacheHandler

* [clsi-cache] extend metrics and event tracking

- break down most of the clsi metrics by label
  - compile=initial - new compile dir without previous output files
  - compile=recompile - recompile in existing compile dir
  - compile=from-cache - compile using previous clsi-cache
- extend segmentation on compile-result-backend event
  - isInitialCompile=true - found new compile dir at start of request
  - restoredClsiCache=true - restored compile dir from clsi-cache

* [clsi] rename metrics labels for download of clsi-cache

This is in preparation for synctex changes.

* [clsi] use constant for limit of entries in output.tar.gz

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>

* [clsi-cache] fix cloning of project cache

---------

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: 4901a65497af13be1549af7f38ceee3188fcf881
2025-04-10 08:05:17 +00:00
Jakob Ackermann
13bf214a3c [web] generate clsi buildId ahead of fetching project content (#24337)
* [web] generate clsi buildId ahead of fetching project content

The buildIds timestamp component will be used for cache invalidation.

* [clsi] strict validation for buildId

* [clsi] validate buildId parameter

GitOrigin-RevId: 88e8b2d48e78fa137b6dca7f2e6b93bbcf88a777
2025-03-24 10:46:02 +00:00
Jakob Ackermann
195a12a3dc [web] fix download of output.pdf when creating Server Pro template (#23988)
GitOrigin-RevId: 899b9bfd54af8d3f345393068663d769ec3a869b
2025-03-03 09:05:26 +00:00
Brian Gough
2ce82fabab Merge pull request #22533 from overleaf/ar-only-use-history-for-blobs-when-enabled
[web] only use history for blobs when enabled

GitOrigin-RevId: 010983e9b29657d4c594e03945dca5700577bf0a
2025-01-10 09:05:32 +00:00
Brian Gough
bb1072a3a4 Merge pull request #22219 from overleaf/jpa-handle-missing-hash
[web] gracefully handle missing hash when compiling

GitOrigin-RevId: a9b121716319117ead84b7fed9a67098f63fd9a7
2024-11-29 09:05:43 +00:00
Jakob Ackermann
d19c5e236f Merge pull request #22208 from overleaf/jpa-clsi-hash
[misc] clsi: read files from history-v1 with fallback to filestore

GitOrigin-RevId: c54bb128780198c14e7a63818f39fad62ce65d4e
2024-11-29 09:05:39 +00:00
Andrew Rumble
d1a58e6b77 Merge pull request #18538 from overleaf/ar-add-download-all-link-in-web
[web] add download all link for output files

GitOrigin-RevId: 3d574d75d53e577cb0f8fd3caa4f757d9e1b7889
2024-06-03 08:04:07 +00:00
Andrew Rumble
9c3d9ef590 Merge pull request #17935 from overleaf/ar-refactor-compile-async
[web] make CompileManager async

GitOrigin-RevId: 617bde1f429fa9aafc7d4bf4ec628b2a22386b19
2024-04-30 08:04:43 +00:00