Commit Graph

169 Commits

Author SHA1 Message Date
Jakob Ackermann 39110d9da9 [clsi-cache] check compiler settings before using compile from cache (#24845)
* [web] provide an actual rootFolder from EditorProviders in tests

- Fixup SocketIOMock and ShareJS mocks to provide the complete interface
- Extend SocketIOMock interface to count event listeners
- Fixup test that did not expect to find a working rootDoc

* [web] expose imageName from ProjectContext

* [clsi-cache] check compiler settings before using compile from cache

* [web] avoid fetching initial compile from clsi-cache in PDF detach tab

GitOrigin-RevId: e3c754a7ceca55f03a317e1bc8ae45ed12cc2f02
2025-04-16 08:05:35 +00:00
Jakob Ackermann 4a17a1e713 [web] gracefully access compile stats for event (#24818)
* [web] gracefully access compile stats for event

* [clsi] always emit stats and timings

GitOrigin-RevId: 959e5fe1508245ffecfab1219fd86e53b210fca1
2025-04-14 08:04:51 +00:00
Jakob Ackermann d99ba08d01 [clsi] run SyncTeX in specific output dir rather than compile dir (#24690)
* [clsi] drop support for docker-in-docker

* [clsi] run SyncTeX in specific output dir rather than compile dir

* [clsi] store output.synctex.gz outside of tar-ball in clsi-cache

* [clsi] add documentation for rewriting of docker bind-mounts

* [server-pro] update env vars for sandboxed compiles in sample config

GitOrigin-RevId: 8debd7102ac612544961f237aa4ff1c530aa3da3
2025-04-10 08:05:26 +00:00
Jakob Ackermann b831a0b3f7 [clsi-cache] frontend (#24389)
* [clsi-lb] forward ?clsiserverid=cache requests to clsi-cache

* [web] use clsi-cache in frontend

* [web] upgrade compile from cache to full compile when triggered inflight

* [web] fix pdf-js-viewer.spec.tsx tests -- add ?clsiserverid=foo to url

* [web] fix renamed reference after merge

* [web] fix download of other output files and use specific build

* [web] consolidate validation of filename into ClsiCacheHandler

* [web] remove unused projectName from getLatestBuildFromCache

* [web] avoid hitting the same clsi-cache instance first all the time

* [web] update documentation

GitOrigin-RevId: d48265a7ba89d6731092640e1492bc9f103f5c33
2025-04-10 08:05:22 +00:00
Jakob Ackermann b538d56591 [clsi-cache] backend (#24388)
* [clsi-cache] initial revision of the clsi-cache service

* [clsi] send output files to clsi-cache and import from clsi-cache

* [web] pass editorId to clsi

* [web] clear clsi-cache when clearing clsi cache

* [web] add split-tests for controlling clsi-cache rollout

* [web] populate clsi-cache when cloning/creating project from template

* [clsi-cache] produce less noise when populating cache hits 404

* [clsi-cache] push docker image to AR

* [clsi-cache] push docker image to AR

* [clsi-cache] allow compileGroup in job payload

* [clsi-cache] set X-Zone header from latest endpoint

* [clsi-cache] use method POST for /enqueue endpoint

* [web] populate clsi-cache in zone b with template data

* [clsi-cache] limit number of editors per project/user folder to 10

* [web] clone: populate the clsi-cache unless the TeXLive release changed

* [clsi-cache] keep user folder when clearing cache as anonymous user

* [clsi] download old output.tar.gz when synctex finds empty compile dir

* [web] fix lint

* [clsi-cache] multi-zonal lookup of single build output

* [clsi-cache] add more validation and limits

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [clsi] do not include clsi-cache tar-ball in output.zip

* [clsi-cache] fix reference after remaining constant

Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>

* [web] consolidate validation of filename into ClsiCacheHandler

* [clsi-cache] extend metrics and event tracking

- break down most of the clsi metrics by label
  - compile=initial - new compile dir without previous output files
  - compile=recompile - recompile in existing compile dir
  - compile=from-cache - compile using previous clsi-cache
- extend segmentation on compile-result-backend event
  - isInitialCompile=true - found new compile dir at start of request
  - restoredClsiCache=true - restored compile dir from clsi-cache

* [clsi] rename metrics labels for download of clsi-cache

This is in preparation for synctex changes.

* [clsi] use constant for limit of entries in output.tar.gz

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>

* [clsi-cache] fix cloning of project cache

---------

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: 4901a65497af13be1549af7f38ceee3188fcf881
2025-04-10 08:05:17 +00:00
Jakob Ackermann 13bf214a3c [web] generate clsi buildId ahead of fetching project content (#24337)
* [web] generate clsi buildId ahead of fetching project content

The buildIds timestamp component will be used for cache invalidation.

* [clsi] strict validation for buildId

* [clsi] validate buildId parameter

GitOrigin-RevId: 88e8b2d48e78fa137b6dca7f2e6b93bbcf88a777
2025-03-24 10:46:02 +00:00
Jakob Ackermann 195a12a3dc [web] fix download of output.pdf when creating Server Pro template (#23988)
GitOrigin-RevId: 899b9bfd54af8d3f345393068663d769ec3a869b
2025-03-03 09:05:26 +00:00
Brian Gough 2ce82fabab Merge pull request #22533 from overleaf/ar-only-use-history-for-blobs-when-enabled
[web] only use history for blobs when enabled

GitOrigin-RevId: 010983e9b29657d4c594e03945dca5700577bf0a
2025-01-10 09:05:32 +00:00
Brian Gough bb1072a3a4 Merge pull request #22219 from overleaf/jpa-handle-missing-hash
[web] gracefully handle missing hash when compiling

GitOrigin-RevId: a9b121716319117ead84b7fed9a67098f63fd9a7
2024-11-29 09:05:43 +00:00
Jakob Ackermann d19c5e236f Merge pull request #22208 from overleaf/jpa-clsi-hash
[misc] clsi: read files from history-v1 with fallback to filestore

GitOrigin-RevId: c54bb128780198c14e7a63818f39fad62ce65d4e
2024-11-29 09:05:39 +00:00
andrew rumble d627a0bee9 Remove files param from output.zip endpoint
GitOrigin-RevId: 300cd870578237ad8e61be4e9e41991f884aa754
2024-09-24 08:04:57 +00:00
Andrew Rumble d1a58e6b77 Merge pull request #18538 from overleaf/ar-add-download-all-link-in-web
[web] add download all link for output files

GitOrigin-RevId: 3d574d75d53e577cb0f8fd3caa4f757d9e1b7889
2024-06-03 08:04:07 +00:00
Christopher Hoskin 3342d672c2 Merge pull request #18397 from overleaf/em-revert-download-all-link
Revert "Merge pull request #18190 from overleaf/ar-add-download-all-l…

GitOrigin-RevId: 681eb2734636d76558e682dc85083bfcaa6b7d2d
2024-05-17 08:05:10 +00:00
Andrew Rumble 105d67bd04 Merge pull request #18190 from overleaf/ar-add-download-all-link
[web] Add download all link for output files

GitOrigin-RevId: a275f17b6acf3130250d91462ba01588b93b62f5
2024-05-16 08:05:04 +00:00
Antoine Clausse eca84b9370 Reintroduce compile sampling for analysis (#18217)
* Revert "Merge pull request #14143 from overleaf/jlm-unsampled-compiles"

This reverts commit 2457977d61756be52db506d7c84531b22a26abe0, reversing
changes made to 3c452b3943c453707aac90d38e7b5b2ea9df0df4.

* Re-add export of `getPercentile` in SplitTestHandler.js

Was previously removed in https://github.com/overleaf/internal/commit/aa8c0aadec90b23ead60a4baeb33d6c3c6a2e536

GitOrigin-RevId: 42c48829e7246a6bdb9dbb2d075638c1eafdf2f0
2024-05-06 08:04:45 +00:00
Andrew Rumble 9c3d9ef590 Merge pull request #17935 from overleaf/ar-refactor-compile-async
[web] make CompileManager async

GitOrigin-RevId: 617bde1f429fa9aafc7d4bf4ec628b2a22386b19
2024-04-30 08:04:43 +00:00
Antoine Clausse 616bd0df16 Merge pull request #17925 from overleaf/jpa-ac-20s-timeout-migration-logging-tweaks
[web] timeout 20s split-test teardown cleanups (remove override, remove tests, ...)

GitOrigin-RevId: 8d8c44539cf45d0f5142f84cf8372cecda3bf77a
2024-04-17 08:04:30 +00:00
Antoine Clausse d35204033f Merge pull request #17909 from overleaf/ac-tear-down-compile-timeout-tests-2
[web]  Remove split-tests `compile-backend-class*` and `compile-timeout-20s*` (attempt 2)

GitOrigin-RevId: 5658f2977d3e7089eec5bbe7a33eee81c153e41d
2024-04-16 08:04:08 +00:00
Antoine Clausse 491bc2628d Merge pull request #17899 from overleaf/revert-17700-ac-tear-down-compile-timeout-tests
Revert "[web] Remove split-tests `compile-backend-class*` and `compile-timeout-20s*`"

GitOrigin-RevId: d5070ced06adbd435e782a44b7ef767e395bd6a0
2024-04-15 08:04:37 +00:00
Antoine Clausse 2dd10c7fee [web] Remove split-tests compile-backend-class* and compile-timeout-20s* (#17700)
* Remove split-tests of `compile-timeout-20s` and `compile-timeout-20s-existing-users`

* Remove `NEW_COMPILE_TIMEOUT_ENFORCED_CUTOFF` variables

* Revert timeout override `60` -> `20`

* Update settings.overrides.saas.js: `compileTimeout: 20`

* Remove `compile-backend-class-n2d`

* Remove `force_new_compile_timeout`

* Remove `showNewCompileTimeoutUI`

* Remove `compileTimeChanging`

* Simplify code by removing segmentation object

* Remove `CompileTimeoutChangingSoon`

* Remove `user.features.compileTimeout = '20 (with 10s prompt)'`

* Remove `CompileTimeWarning`

* Remove `TimeoutUpgradePrompt` (old)

* Remove `compile-backend-class`

* Remove unused translations

* Update tests

* Fix: Show `CompileTimeout` even if `!window.ExposedSettings.enableSubscriptions`

* Create script to migrate users to 20s compileTimeout

* migration script: exclude `compileTimeout: 20` from the match

* migration script: use `batchedUpdate`

* Remove `showFasterCompilesFeedbackUI` and `FasterCompilesFeedback`

Helped-by: Jakob Ackermann <jakob.ackermann@overleaf.com>

* Remove `_getCompileBackendClassDetails`, simplify definition of `limits` object

* Remove `Settings.apis.clsi.defaultBackendClass`

* Remove unnecessary second scan of the whole user collection in dry mode

* Override `timeout` to 20 for users having `compileGroup === 'standard' && compileTimeout <= 60`

* Remove second `logCount`: re-run the script in dry-mode if you want to see that count

* Use secondary readPreference when counting users

* Fix script setup and exit 0

* Fix: Remove `user.` from query path!

* Add acceptance test on script migration_compile_timeout_60s_to_20s.js

GitOrigin-RevId: 3cb65130e6d7fbd9c54005f4c213066d0473e9d8
2024-04-15 08:04:24 +00:00
David 32d2603adb Merge pull request #16731 from overleaf/dp-ip-rate-metrics
Add tracking of rate limit method to metrics

GitOrigin-RevId: 3996c2a0ccb747018571ce402120be46fc52eace
2024-02-13 09:04:09 +00:00
Miguel Serrano 51de84664d Merge pull request #16346 from overleaf/msm-remove-underscore
Remove `underscore` in monorepo

GitOrigin-RevId: fd3b92e3e8a940a7a086669232102320c34a89e2
2024-01-09 09:03:58 +00:00
Davinder Singh 5c5a01777c Making getAnalyticsIdFromMongoUser private to UserAnalyticsIdCache (#16125)
* Making getAnalyticsIdFromMongoUser private to UserAnalyticsIdCache

* renaming userAnalyticsIdCache to UserAnalyticsIdCache

* deleting function _getAnalyticsIdFromMongoUser

* renaming userAnalyticsIdCache to UserAnalyticsIdCache part 2

* format:fix

* adding upperCamelCase

* capital case first letter

GitOrigin-RevId: 2e9c18c544b8cffb53838aed56e1ef16979606a5
2024-01-08 09:05:31 +00:00
Jimmy Domagala-Tang d94ca9679f Merge pull request #16322 from overleaf/dk-ignore-clsi-warning
[web] ignore noisy CLSI 404 warning

GitOrigin-RevId: d821496e2fb91ba184a32449986c128b9b079f3f
2024-01-04 09:03:37 +00:00
Jakob Ackermann 453edbfe0a Merge pull request #15793 from overleaf/jpa-compile-domain-cleanup
[web] tear down split test for new compile domain

GitOrigin-RevId: 7ab417d67e508cd95bd4e78ce7ce2a993c789dca
2023-11-17 09:02:56 +00:00
Thomas 3cd0cefd30 Gradual rollout of new compile timeout to existing users (#15478)
* Rollout compile timeout to existing users

* Update copy for compile timeout messages

GitOrigin-RevId: bada02efb3fc506ae6e35bdb7871a58ef4295cbe
2023-10-31 09:04:11 +00:00
Jakob Ackermann 347da4e6bb Merge pull request #15247 from overleaf/jpa-submission-backend-class
[web] use n2d instances for compiling submissions

GitOrigin-RevId: 57eb27af230b994c3816c0476673ea558226ac1e
2023-10-25 08:04:03 +00:00
Eric Mc Sween 680ebae30b Merge pull request #15172 from overleaf/em-promise-utils
Move util/promises from web into a shared library

GitOrigin-RevId: fe1980dc57b9dc8ce86fa1fad6a8a817e9505b3d
2023-10-20 08:04:05 +00:00
John Lees-Miller 8f1dac6d68 Merge pull request #15195 from overleaf/tm-compile-timeout-20s-phase-two-date-change
Update new/existing user cut off date for compile timeout

GitOrigin-RevId: 382e8bc171f1d3728cfbaefa85d9c2d13779d4ee
2023-10-11 08:03:46 +00:00
Thomas 03aaabd5cb Merge pull request #15169 from overleaf/tm-compile-timeout-20s-phase-two
Decide new user cutoff using baseline (original) n2d assignment

GitOrigin-RevId: 7ea263cc551f87a4d9fed70450c32d7dea6b1e58
2023-10-11 08:03:33 +00:00
Thomas 31cb9e336b Implement test to reduce compile timeout to 20 seconds (#14705)
Compile timeout reduction to 20s for treatment users

Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>
GitOrigin-RevId: 54f70fe4b1fc631cef966deb0c1d28c904dd3a44
2023-09-19 08:03:52 +00:00
Jakob Ackermann 63bd1f6095 Merge pull request #14501 from overleaf/jpa-streaming-log-noise
[web] CompileController: ignore noisy spurious error

GitOrigin-RevId: 8885286a5e06bfa38bc70d98b703f52a1269d899
2023-08-28 08:04:57 +00:00
Jakob Ackermann 33ac9e18f9 Merge pull request #14358 from overleaf/jpa-abort-check
[web] skip streaming clsi response when the request was aborted

GitOrigin-RevId: 8152399f18328eafd3e93143190a5eb8d1fdff1d
2023-08-18 08:04:51 +00:00
Jakob Ackermann bf2e33ec83 Merge pull request #14023 from overleaf/jpa-web-clsi-fetch
[web] migrate CompileController from request to node-fetch

GitOrigin-RevId: ffb3319319d7f986ec972b4b6c56bd5499ecd9ab
2023-08-04 08:03:39 +00:00
John Lees-Miller 7c7a3d7a9c Merge pull request #14143 from overleaf/jlm-unsampled-compiles
Remove compile sampling for analysis

GitOrigin-RevId: 2457977d61756be52db506d7c84531b22a26abe0
2023-08-03 08:04:12 +00:00
Mathias Jakobsen c7006fabd4 Merge pull request #13980 from overleaf/mj-web-compile-timeout
[web] Add timeout for compile requests to clsi

GitOrigin-RevId: 94d6de60eec1fc9e52971e14f38f2d4aef6b7f5d
2023-07-24 08:03:46 +00:00
Eric Mc Sween 28e87a9314 Merge pull request #13760 from overleaf/em-fetch-utils-web
Use fetch-utils in web

GitOrigin-RevId: cbd0298200bbe42567c6e94934bfb5114fa9b66f
2023-07-17 11:02:40 +00:00
Eric Mc Sween 75a86bab87 Merge pull request #13537 from overleaf/em-promisify-clsi-manager-2
Promisify ClsiManager

GitOrigin-RevId: e785cdb1989b433f7baa1a4d974b09ba3a93fcbd
2023-07-17 10:54:40 +00:00
Alf Eaton 29f67a3985 Merge pull request #13525 from overleaf/jpa-teardown-compile-domain-test
[web] start tear down of split test for new compile domain

GitOrigin-RevId: 405bba924a37f56614b0744f47914306f420a028
2023-07-17 10:43:01 +00:00
Eric Mc Sween 1fc13b8cbf Merge pull request #13510 from overleaf/em-promisify-clsi-manager
Clean up of ClsiManager and ClsiCookieManager

GitOrigin-RevId: e5047b253613e87fd6cb4f12855b821028fcaf8e
2023-07-17 10:41:13 +00:00
Eric Mc Sween f038cd4d10 Merge pull request #13454 from overleaf/jpa-n2d-split-test
[web] add split test for running standard compiles on n2d instances

GitOrigin-RevId: 05b85870c2ff5eb0345e700101225535a4a569f1
2023-07-17 10:34:42 +00:00
Alf Eaton 4670c698ee Allow digits in PDF filenames (#13122)
GitOrigin-RevId: 58c93d299c3bfe79dd4c223b8b060411bad078c7
2023-05-18 08:03:55 +00:00
Alf Eaton c8c17bca38 Allow all Unicode "letter" characters in PDF filename (#13071)
GitOrigin-RevId: 030d7326b33525a34bccff26e12103e9a8e1a3f3
2023-05-17 08:05:19 +00:00
Alf Eaton 7f795e4c8b Use content-disposition for setting the Content-Disposition header (#13072)
GitOrigin-RevId: f84100beb5e00485be4732c954b5ee553fe18558
2023-05-17 08:05:10 +00:00
Jakob Ackermann 21950fdb30 Merge pull request #13091 from overleaf/jpa-metric-blg
[web] add metric for blg file usage

GitOrigin-RevId: 5a6bd4185d5f236a56994331cfe3b25c1998d0db
2023-05-16 08:05:38 +00:00
Brian Gough 1da76f0a8d Merge pull request #12848 from overleaf/bg-fix-path-exception
add exception handling for  path.join in ProjectEntityHandler

GitOrigin-RevId: dad305057fd6b2821525ca5b6d1933824989e241
2023-05-02 19:43:50 +00:00
Eric Mc Sween 21971956b7 Merge pull request #12219 from overleaf/em-camel-case-web
Camel case variables in web

GitOrigin-RevId: 28e61b759b27f71265f33ab64f588374dba610e0
2023-03-22 09:05:04 +00:00
Jakob Ackermann df3c7e48ab Merge pull request #12198 from overleaf/jpa-force-new-compile-domain
[web] changes for force-new-compile-domain test

GitOrigin-RevId: a4ceaf46fdcebed156d155385cbbc2f06405d31f
2023-03-17 09:04:49 +00:00
Jakob Ackermann e0c84f08fb Merge pull request #11940 from overleaf/jpa-fix-assignment-of-initial-clsi-vm
[web] fix assignment of initial compile server

GitOrigin-RevId: 84aea918946a106a5826d7cf39cc0bf47047cb35
2023-03-07 09:04:33 +00:00
Eric Mc Sween 75abea72b0 Merge pull request #11492 from overleaf/em-rate-limiter
Move all remaining rate limiters to rate-limiter-flexible

GitOrigin-RevId: 163ab2aebecb281057e552dc75591dd02028990c
2023-01-31 09:03:44 +00:00