Commit Graph

122 Commits

Author SHA1 Message Date
Eric Mc Sween
57681c10c8 Merge pull request #25646 from overleaf/em-ds-mobile-app-compile
Compile endpoint for the DS mobile app

GitOrigin-RevId: 2fd9f4a6e8c2ed4ee868b0c1293f6760b9d113c8
2025-05-21 08:06:36 +00:00
Jakob Ackermann
291dd2fa85 [clsi-cache] remove non sharded instances (#25645)
* Revert "[clsi-cache] only use sharding from updated project editor tabs (#25326)"

This reverts commit 1754276bed3186c0536055c983e32476cc90d416.

* [clsi-cache] remove non sharded instances

GitOrigin-RevId: aa3ac46140dfc1722a3350cf7071e5b11af61199
2025-05-16 08:05:02 +00:00
Antoine Clausse
d389cca007 [web] Promisify ClsiCookieManager and CompileController (reapply again) (#25280)
* Reapply "[web] Promisify ClsiCookieManager and CompileController (reapply and fix)"

This reverts commit 98cb9127ff2b7c7c347c560766f749265d712490.

* Fix: Use query parameters correctly (!!)

* Add unit test on `checkIsLoadSheddingEvent`

* Remove interference between tests: rename to `ClsiCookieManager2` when it's re-sandboxed

* Add test: 'should report "cycle" when other is UP'

GitOrigin-RevId: 3146b149954b908830226cb03b51d9adfa08ec2e
2025-05-07 08:07:07 +00:00
Jakob Ackermann
2c67a76115 [clsi-cache] only use sharding from updated project editor tabs (#25326)
GitOrigin-RevId: 1754276bed3186c0536055c983e32476cc90d416
2025-05-07 08:06:39 +00:00
Antoine Clausse
19d012bcee Merge pull request #25221 from overleaf/revert-25207-ac-promisify-compile-controller-2
Revert "[web] Promisify ClsiCookieManager and CompileController (reapply and fix)"

GitOrigin-RevId: 4495b0fdee22c9c2a7eb39250aef498883389427
2025-05-01 08:06:55 +00:00
Antoine Clausse
42ae34e830 Merge pull request #25207 from overleaf/ac-promisify-compile-controller-2
[web] Promisify ClsiCookieManager and CompileController (reapply and fix)

GitOrigin-RevId: 0737f30c24bf92b33327dc7d0e015ac2cd7d751d
2025-05-01 08:06:47 +00:00
Antoine Clausse
c0759da78e Merge pull request #25200 from overleaf/revert-25023-ac-promisify-compile-controller
Revert "[web] Promisify ClsiCookieManager and CompileController"

GitOrigin-RevId: 190ee8d2be23687f092e762c5199a34bcdf37cf9
2025-05-01 08:06:00 +00:00
Antoine Clausse
5106b8b54e [web] Promisify ClsiCookieManager and CompileController (#25023)
* Refactor `ClsiCookieManager` to use async/await

* Refactor `CompileController` to use async/await

* Fix tests: CompileControllerTests.js

* Fix tests: ClsiCookieManagerTests.js

* Fixup: keep old object key (`serverId`->`clsiServerId`)

* Prefix non-express methods with an underscore

* Use async versions of `rclient` methods

* Fix: `canContinue` must be `false` when rate limited (!!)

* Remove unused `ClsiCookieManager.getCookieJar`

* Remove bad comments

* Fix linting

* Replace `request` by `@overleaf/fetch-utils`

* Replace `callsFake` by `resolves`

* Update `catch` block after request->fetch update: check status code

* Re-add timeout of 30s (was removed in 3df75ab5ccc8a63bd69a6a6e6052ef0451b76220)

* `npm run format:fix`

* Don't throw on 4xx errors; keep functionality similar to the current implementation using `request`.

GitOrigin-RevId: ef161f6c252d13f407e9cf28029b62061d6b992f
2025-05-01 08:05:42 +00:00
Andrew Rumble
34ae228d99 Promisify tests
GitOrigin-RevId: 6f413f4c5ef8d034b4e94afacdf2d7b43c3a8830
2025-04-29 08:05:18 +00:00
Jakob Ackermann
2825083ec8 [web] update the projects lastUpdated timestamp when changing file-tree (#24867)
* [misc] freeze time before any other unit test setup steps

Freezing it after other work (notably sandboxed-module imports) will
result in flaky tests.

* [web] update the projects lastUpdated timestamp when changing file-tree

GitOrigin-RevId: b82b2ff74dc31886f3c4bd300375117eead6e0cd
2025-04-16 08:05:14 +00:00
Jakob Ackermann
ddc15e4c8d [clsi-cache] backend (#24388)
* [clsi-cache] initial revision of the clsi-cache service

* [clsi] send output files to clsi-cache and import from clsi-cache

* [web] pass editorId to clsi

* [web] clear clsi-cache when clearing clsi cache

* [web] add split-tests for controlling clsi-cache rollout

* [web] populate clsi-cache when cloning/creating project from template

* [clsi-cache] produce less noise when populating cache hits 404

* [clsi-cache] push docker image to AR

* [clsi-cache] push docker image to AR

* [clsi-cache] allow compileGroup in job payload

* [clsi-cache] set X-Zone header from latest endpoint

* [clsi-cache] use method POST for /enqueue endpoint

* [web] populate clsi-cache in zone b with template data

* [clsi-cache] limit number of editors per project/user folder to 10

* [web] clone: populate the clsi-cache unless the TeXLive release changed

* [clsi-cache] keep user folder when clearing cache as anonymous user

* [clsi] download old output.tar.gz when synctex finds empty compile dir

* [web] fix lint

* [clsi-cache] multi-zonal lookup of single build output

* [clsi-cache] add more validation and limits

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [clsi] do not include clsi-cache tar-ball in output.zip

* [clsi-cache] fix reference after remaining constant

Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>

* [web] consolidate validation of filename into ClsiCacheHandler

* [clsi-cache] extend metrics and event tracking

- break down most of the clsi metrics by label
  - compile=initial - new compile dir without previous output files
  - compile=recompile - recompile in existing compile dir
  - compile=from-cache - compile using previous clsi-cache
- extend segmentation on compile-result-backend event
  - isInitialCompile=true - found new compile dir at start of request
  - restoredClsiCache=true - restored compile dir from clsi-cache

* [clsi] rename metrics labels for download of clsi-cache

This is in preparation for synctex changes.

* [clsi] use constant for limit of entries in output.tar.gz

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>

* [clsi-cache] fix cloning of project cache

---------

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: 4901a65497af13be1549af7f38ceee3188fcf881
2025-04-10 08:05:17 +00:00
Jakob Ackermann
7adf52694c [web] generate clsi buildId ahead of fetching project content (#24337)
* [web] generate clsi buildId ahead of fetching project content

The buildIds timestamp component will be used for cache invalidation.

* [clsi] strict validation for buildId

* [clsi] validate buildId parameter

GitOrigin-RevId: 88e8b2d48e78fa137b6dca7f2e6b93bbcf88a777
2025-03-24 10:46:02 +00:00
Jakob Ackermann
302788e50f [web] fix download of output.pdf when creating Server Pro template (#23988)
GitOrigin-RevId: 899b9bfd54af8d3f345393068663d769ec3a869b
2025-03-03 09:05:26 +00:00
Brian Gough
f8de8da7f7 Merge pull request #22533 from overleaf/ar-only-use-history-for-blobs-when-enabled
[web] only use history for blobs when enabled

GitOrigin-RevId: 010983e9b29657d4c594e03945dca5700577bf0a
2025-01-10 09:05:32 +00:00
Brian Gough
cfbac60ab0 Merge pull request #22219 from overleaf/jpa-handle-missing-hash
[web] gracefully handle missing hash when compiling

GitOrigin-RevId: a9b121716319117ead84b7fed9a67098f63fd9a7
2024-11-29 09:05:43 +00:00
Jakob Ackermann
a9f10f013e Merge pull request #22208 from overleaf/jpa-clsi-hash
[misc] clsi: read files from history-v1 with fallback to filestore

GitOrigin-RevId: c54bb128780198c14e7a63818f39fad62ce65d4e
2024-11-29 09:05:39 +00:00
Andrew Rumble
2d6b19fa9a Merge pull request #18538 from overleaf/ar-add-download-all-link-in-web
[web] add download all link for output files

GitOrigin-RevId: 3d574d75d53e577cb0f8fd3caa4f757d9e1b7889
2024-06-03 08:04:07 +00:00
Andrew Rumble
1de4fa1340 Merge pull request #17935 from overleaf/ar-refactor-compile-async
[web] make CompileManager async

GitOrigin-RevId: 617bde1f429fa9aafc7d4bf4ec628b2a22386b19
2024-04-30 08:04:43 +00:00
Jakob Ackermann
74f8a078ca Merge pull request #18116 from overleaf/jpa-bulk-replace-localhost
[misc] bulk replace localhost with 127.0.0.1

GitOrigin-RevId: d238f3635302e8ff5500d611108c4d1bef216726
2024-04-26 08:04:39 +00:00
Antoine Clausse
46afce9e09 Merge pull request #17925 from overleaf/jpa-ac-20s-timeout-migration-logging-tweaks
[web] timeout 20s split-test teardown cleanups (remove override, remove tests, ...)

GitOrigin-RevId: 8d8c44539cf45d0f5142f84cf8372cecda3bf77a
2024-04-17 08:04:30 +00:00
Antoine Clausse
e2648c28cd Merge pull request #17909 from overleaf/ac-tear-down-compile-timeout-tests-2
[web]  Remove split-tests `compile-backend-class*` and `compile-timeout-20s*` (attempt 2)

GitOrigin-RevId: 5658f2977d3e7089eec5bbe7a33eee81c153e41d
2024-04-16 08:04:08 +00:00
Antoine Clausse
113344b6a9 Merge pull request #17899 from overleaf/revert-17700-ac-tear-down-compile-timeout-tests
Revert "[web] Remove split-tests `compile-backend-class*` and `compile-timeout-20s*`"

GitOrigin-RevId: d5070ced06adbd435e782a44b7ef767e395bd6a0
2024-04-15 08:04:37 +00:00
Antoine Clausse
7cb0b0b6af [web] Remove split-tests compile-backend-class* and compile-timeout-20s* (#17700)
* Remove split-tests of `compile-timeout-20s` and `compile-timeout-20s-existing-users`

* Remove `NEW_COMPILE_TIMEOUT_ENFORCED_CUTOFF` variables

* Revert timeout override `60` -> `20`

* Update settings.overrides.saas.js: `compileTimeout: 20`

* Remove `compile-backend-class-n2d`

* Remove `force_new_compile_timeout`

* Remove `showNewCompileTimeoutUI`

* Remove `compileTimeChanging`

* Simplify code by removing segmentation object

* Remove `CompileTimeoutChangingSoon`

* Remove `user.features.compileTimeout = '20 (with 10s prompt)'`

* Remove `CompileTimeWarning`

* Remove `TimeoutUpgradePrompt` (old)

* Remove `compile-backend-class`

* Remove unused translations

* Update tests

* Fix: Show `CompileTimeout` even if `!window.ExposedSettings.enableSubscriptions`

* Create script to migrate users to 20s compileTimeout

* migration script: exclude `compileTimeout: 20` from the match

* migration script: use `batchedUpdate`

* Remove `showFasterCompilesFeedbackUI` and `FasterCompilesFeedback`

Helped-by: Jakob Ackermann <jakob.ackermann@overleaf.com>

* Remove `_getCompileBackendClassDetails`, simplify definition of `limits` object

* Remove `Settings.apis.clsi.defaultBackendClass`

* Remove unnecessary second scan of the whole user collection in dry mode

* Override `timeout` to 20 for users having `compileGroup === 'standard' && compileTimeout <= 60`

* Remove second `logCount`: re-run the script in dry-mode if you want to see that count

* Use secondary readPreference when counting users

* Fix script setup and exit 0

* Fix: Remove `user.` from query path!

* Add acceptance test on script migration_compile_timeout_60s_to_20s.js

GitOrigin-RevId: 3cb65130e6d7fbd9c54005f4c213066d0473e9d8
2024-04-15 08:04:24 +00:00
Davinder Singh
c45c283ed4 Making getAnalyticsIdFromMongoUser private to UserAnalyticsIdCache (#16125)
* Making getAnalyticsIdFromMongoUser private to UserAnalyticsIdCache

* renaming userAnalyticsIdCache to UserAnalyticsIdCache

* deleting function _getAnalyticsIdFromMongoUser

* renaming userAnalyticsIdCache to UserAnalyticsIdCache part 2

* format:fix

* adding upperCamelCase

* capital case first letter

GitOrigin-RevId: 2e9c18c544b8cffb53838aed56e1ef16979606a5
2024-01-08 09:05:31 +00:00
Thomas
2995b59a4e Gradual rollout of new compile timeout to existing users (#15478)
* Rollout compile timeout to existing users

* Update copy for compile timeout messages

GitOrigin-RevId: bada02efb3fc506ae6e35bdb7871a58ef4295cbe
2023-10-31 09:04:11 +00:00
Jakob Ackermann
e4154cb286 Merge pull request #15247 from overleaf/jpa-submission-backend-class
[web] use n2d instances for compiling submissions

GitOrigin-RevId: 57eb27af230b994c3816c0476673ea558226ac1e
2023-10-25 08:04:03 +00:00
Thomas
a2ad326b30 Merge pull request #15169 from overleaf/tm-compile-timeout-20s-phase-two
Decide new user cutoff using baseline (original) n2d assignment

GitOrigin-RevId: 7ea263cc551f87a4d9fed70450c32d7dea6b1e58
2023-10-11 08:03:33 +00:00
Thomas
5c8de7dd58 Implement test to reduce compile timeout to 20 seconds (#14705)
Compile timeout reduction to 20s for treatment users

Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>
GitOrigin-RevId: 54f70fe4b1fc631cef966deb0c1d28c904dd3a44
2023-09-19 08:03:52 +00:00
Jakob Ackermann
ed99efca27 Merge pull request #14501 from overleaf/jpa-streaming-log-noise
[web] CompileController: ignore noisy spurious error

GitOrigin-RevId: 8885286a5e06bfa38bc70d98b703f52a1269d899
2023-08-28 08:04:57 +00:00
Jakob Ackermann
6982bb4601 Merge pull request #14358 from overleaf/jpa-abort-check
[web] skip streaming clsi response when the request was aborted

GitOrigin-RevId: 8152399f18328eafd3e93143190a5eb8d1fdff1d
2023-08-18 08:04:51 +00:00
Jakob Ackermann
f2262f322b Merge pull request #14023 from overleaf/jpa-web-clsi-fetch
[web] migrate CompileController from request to node-fetch

GitOrigin-RevId: ffb3319319d7f986ec972b4b6c56bd5499ecd9ab
2023-08-04 08:03:39 +00:00
Mathias Jakobsen
ce239a7652 Merge pull request #13980 from overleaf/mj-web-compile-timeout
[web] Add timeout for compile requests to clsi

GitOrigin-RevId: 94d6de60eec1fc9e52971e14f38f2d4aef6b7f5d
2023-07-24 08:03:46 +00:00
Eric Mc Sween
39a396f3a2 Merge pull request #13760 from overleaf/em-fetch-utils-web
Use fetch-utils in web

GitOrigin-RevId: cbd0298200bbe42567c6e94934bfb5114fa9b66f
2023-07-17 11:02:40 +00:00
Eric Mc Sween
9346d1fa14 Merge pull request #13537 from overleaf/em-promisify-clsi-manager-2
Promisify ClsiManager

GitOrigin-RevId: e785cdb1989b433f7baa1a4d974b09ba3a93fcbd
2023-07-17 10:54:40 +00:00
Alf Eaton
a911139deb Merge pull request #13525 from overleaf/jpa-teardown-compile-domain-test
[web] start tear down of split test for new compile domain

GitOrigin-RevId: 405bba924a37f56614b0744f47914306f420a028
2023-07-17 10:43:01 +00:00
Eric Mc Sween
33a8a90134 Merge pull request #13510 from overleaf/em-promisify-clsi-manager
Clean up of ClsiManager and ClsiCookieManager

GitOrigin-RevId: e5047b253613e87fd6cb4f12855b821028fcaf8e
2023-07-17 10:41:13 +00:00
Eric Mc Sween
a153b9ddc4 Merge pull request #13454 from overleaf/jpa-n2d-split-test
[web] add split test for running standard compiles on n2d instances

GitOrigin-RevId: 05b85870c2ff5eb0345e700101225535a4a569f1
2023-07-17 10:34:42 +00:00
Alf Eaton
4842879c57 Allow digits in PDF filenames (#13122)
GitOrigin-RevId: 58c93d299c3bfe79dd4c223b8b060411bad078c7
2023-05-18 08:03:55 +00:00
Alf Eaton
cc1a767f99 Allow all Unicode "letter" characters in PDF filename (#13071)
GitOrigin-RevId: 030d7326b33525a34bccff26e12103e9a8e1a3f3
2023-05-17 08:05:19 +00:00
Alf Eaton
214c72816e Use content-disposition for setting the Content-Disposition header (#13072)
GitOrigin-RevId: f84100beb5e00485be4732c954b5ee553fe18558
2023-05-17 08:05:10 +00:00
Jakob Ackermann
3f7e897320 Merge pull request #13091 from overleaf/jpa-metric-blg
[web] add metric for blg file usage

GitOrigin-RevId: 5a6bd4185d5f236a56994331cfe3b25c1998d0db
2023-05-16 08:05:38 +00:00
Jakob Ackermann
a32622b35e Merge pull request #12198 from overleaf/jpa-force-new-compile-domain
[web] changes for force-new-compile-domain test

GitOrigin-RevId: a4ceaf46fdcebed156d155385cbbc2f06405d31f
2023-03-17 09:04:49 +00:00
Jakob Ackermann
074fa09667 Merge pull request #11940 from overleaf/jpa-fix-assignment-of-initial-clsi-vm
[web] fix assignment of initial compile server

GitOrigin-RevId: 84aea918946a106a5826d7cf39cc0bf47047cb35
2023-03-07 09:04:33 +00:00
Eric Mc Sween
daef90acc0 Merge pull request #11492 from overleaf/em-rate-limiter
Move all remaining rate limiters to rate-limiter-flexible

GitOrigin-RevId: 163ab2aebecb281057e552dc75591dd02028990c
2023-01-31 09:03:44 +00:00
Jakob Ackermann
d2791e9adf Merge pull request #11442 from overleaf/jpa-hybrid-pdf-download
[web] add hybrid download of compile output

GitOrigin-RevId: b9c6dfc86e3af2f7b1728e791f074ec6d5e0badf
2023-01-27 09:04:12 +00:00
Jakob Ackermann
ca3114fbe4 Merge pull request #11246 from overleaf/jpa-user-content-domain-access-check
[misc] prepare migration to user content domain

GitOrigin-RevId: 581ccab6d39ec021fb44a555a09e55441c35d0d1
2023-01-18 09:05:14 +00:00
Brian Gough
3e21bde0cd Merge pull request #10740 from overleaf/jpa-auto-compile-rate-limit-two-buckets
[web] split rate-limit tracking for auto-compiles into two buckets

GitOrigin-RevId: dcc743343010afe7d877fe5d37f020f10a8fd412
2022-12-06 09:04:10 +00:00
Jakob Ackermann
d1dc798c07 Merge pull request #10139 from overleaf/jpa-split-test-min-chunk-size
[misc] add split test for a per request pdfCachingMinChunkSize

GitOrigin-RevId: 6a8a3c6267501789f2047a67b03db6ac6df427c3
2022-10-26 08:03:39 +00:00
Jakob Ackermann
b2d2ff6aa4 Merge pull request #8847 from overleaf/jpa-emit-start-of-xref-table
[clsi] try to emit the start of the xref table

GitOrigin-RevId: 6d8348a349572cc997ac5924664428228c00fed1
2022-07-18 08:04:17 +00:00
Jakob Ackermann
cfa4df0c77 Merge pull request #8730 from overleaf/jpa-refactor-pdf-caching
[web] refactor pdf caching

GitOrigin-RevId: af6b871fad652f757d8c465085b33a79b955cd6b
2022-07-07 08:03:28 +00:00