Commit Graph

79 Commits

Author SHA1 Message Date
Jakob Ackermann b538d56591 [clsi-cache] backend (#24388)
* [clsi-cache] initial revision of the clsi-cache service

* [clsi] send output files to clsi-cache and import from clsi-cache

* [web] pass editorId to clsi

* [web] clear clsi-cache when clearing clsi cache

* [web] add split-tests for controlling clsi-cache rollout

* [web] populate clsi-cache when cloning/creating project from template

* [clsi-cache] produce less noise when populating cache hits 404

* [clsi-cache] push docker image to AR

* [clsi-cache] push docker image to AR

* [clsi-cache] allow compileGroup in job payload

* [clsi-cache] set X-Zone header from latest endpoint

* [clsi-cache] use method POST for /enqueue endpoint

* [web] populate clsi-cache in zone b with template data

* [clsi-cache] limit number of editors per project/user folder to 10

* [web] clone: populate the clsi-cache unless the TeXLive release changed

* [clsi-cache] keep user folder when clearing cache as anonymous user

* [clsi] download old output.tar.gz when synctex finds empty compile dir

* [web] fix lint

* [clsi-cache] multi-zonal lookup of single build output

* [clsi-cache] add more validation and limits

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [clsi] do not include clsi-cache tar-ball in output.zip

* [clsi-cache] fix reference after remaining constant

Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>

* [web] consolidate validation of filename into ClsiCacheHandler

* [clsi-cache] extend metrics and event tracking

- break down most of the clsi metrics by label
  - compile=initial - new compile dir without previous output files
  - compile=recompile - recompile in existing compile dir
  - compile=from-cache - compile using previous clsi-cache
- extend segmentation on compile-result-backend event
  - isInitialCompile=true - found new compile dir at start of request
  - restoredClsiCache=true - restored compile dir from clsi-cache

* [clsi] rename metrics labels for download of clsi-cache

This is in preparation for synctex changes.

* [clsi] use constant for limit of entries in output.tar.gz

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>

* [clsi-cache] fix cloning of project cache

---------

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: 4901a65497af13be1549af7f38ceee3188fcf881
2025-04-10 08:05:17 +00:00
Jakob Ackermann 13bf214a3c [web] generate clsi buildId ahead of fetching project content (#24337)
* [web] generate clsi buildId ahead of fetching project content

The buildIds timestamp component will be used for cache invalidation.

* [clsi] strict validation for buildId

* [clsi] validate buildId parameter

GitOrigin-RevId: 88e8b2d48e78fa137b6dca7f2e6b93bbcf88a777
2025-03-24 10:46:02 +00:00
Brian Gough 2ce82fabab Merge pull request #22533 from overleaf/ar-only-use-history-for-blobs-when-enabled
[web] only use history for blobs when enabled

GitOrigin-RevId: 010983e9b29657d4c594e03945dca5700577bf0a
2025-01-10 09:05:32 +00:00
Brian Gough bb1072a3a4 Merge pull request #22219 from overleaf/jpa-handle-missing-hash
[web] gracefully handle missing hash when compiling

GitOrigin-RevId: a9b121716319117ead84b7fed9a67098f63fd9a7
2024-11-29 09:05:43 +00:00
Jakob Ackermann d19c5e236f Merge pull request #22208 from overleaf/jpa-clsi-hash
[misc] clsi: read files from history-v1 with fallback to filestore

GitOrigin-RevId: c54bb128780198c14e7a63818f39fad62ce65d4e
2024-11-29 09:05:39 +00:00
Andrew Rumble d1a58e6b77 Merge pull request #18538 from overleaf/ar-add-download-all-link-in-web
[web] add download all link for output files

GitOrigin-RevId: 3d574d75d53e577cb0f8fd3caa4f757d9e1b7889
2024-06-03 08:04:07 +00:00
Miguel Serrano 51de84664d Merge pull request #16346 from overleaf/msm-remove-underscore
Remove `underscore` in monorepo

GitOrigin-RevId: fd3b92e3e8a940a7a086669232102320c34a89e2
2024-01-09 09:03:58 +00:00
Eric Mc Sween 680ebae30b Merge pull request #15172 from overleaf/em-promise-utils
Move util/promises from web into a shared library

GitOrigin-RevId: fe1980dc57b9dc8ce86fa1fad6a8a817e9505b3d
2023-10-20 08:04:05 +00:00
Mathias Jakobsen c7006fabd4 Merge pull request #13980 from overleaf/mj-web-compile-timeout
[web] Add timeout for compile requests to clsi

GitOrigin-RevId: 94d6de60eec1fc9e52971e14f38f2d4aef6b7f5d
2023-07-24 08:03:46 +00:00
Eric Mc Sween 28e87a9314 Merge pull request #13760 from overleaf/em-fetch-utils-web
Use fetch-utils in web

GitOrigin-RevId: cbd0298200bbe42567c6e94934bfb5114fa9b66f
2023-07-17 11:02:40 +00:00
Eric Mc Sween 75a86bab87 Merge pull request #13537 from overleaf/em-promisify-clsi-manager-2
Promisify ClsiManager

GitOrigin-RevId: e785cdb1989b433f7baa1a4d974b09ba3a93fcbd
2023-07-17 10:54:40 +00:00
Eric Mc Sween 1fc13b8cbf Merge pull request #13510 from overleaf/em-promisify-clsi-manager
Clean up of ClsiManager and ClsiCookieManager

GitOrigin-RevId: e5047b253613e87fd6cb4f12855b821028fcaf8e
2023-07-17 10:41:13 +00:00
Jakob Ackermann 21950fdb30 Merge pull request #13091 from overleaf/jpa-metric-blg
[web] add metric for blg file usage

GitOrigin-RevId: 5a6bd4185d5f236a56994331cfe3b25c1998d0db
2023-05-16 08:05:38 +00:00
Brian Gough 1da76f0a8d Merge pull request #12848 from overleaf/bg-fix-path-exception
add exception handling for  path.join in ProjectEntityHandler

GitOrigin-RevId: dad305057fd6b2821525ca5b6d1933824989e241
2023-05-02 19:43:50 +00:00
Jakob Ackermann e0c84f08fb Merge pull request #11940 from overleaf/jpa-fix-assignment-of-initial-clsi-vm
[web] fix assignment of initial compile server

GitOrigin-RevId: 84aea918946a106a5826d7cf39cc0bf47047cb35
2023-03-07 09:04:33 +00:00
Jakob Ackermann 956cacaef7 Merge pull request #10139 from overleaf/jpa-split-test-min-chunk-size
[misc] add split test for a per request pdfCachingMinChunkSize

GitOrigin-RevId: 6a8a3c6267501789f2047a67b03db6ac6df427c3
2022-10-26 08:03:39 +00:00
Jakob Ackermann 97624d0c6c Merge pull request #8847 from overleaf/jpa-emit-start-of-xref-table
[clsi] try to emit the start of the xref table

GitOrigin-RevId: 6d8348a349572cc997ac5924664428228c00fed1
2022-07-18 08:04:17 +00:00
Jakob Ackermann 64184a7f56 Merge pull request #8730 from overleaf/jpa-refactor-pdf-caching
[web] refactor pdf caching

GitOrigin-RevId: af6b871fad652f757d8c465085b33a79b955cd6b
2022-07-07 08:03:28 +00:00
Jakob Ackermann 50dd7a37f7 Merge pull request #8567 from overleaf/jpa-missing-class
[web] fetch compileBackendClass for linked project output downloads

GitOrigin-RevId: fc29aa700df879402ba1734775160799ce8ce61d
2022-06-24 08:05:01 +00:00
Jakob Ackermann b19c56ccaf Merge pull request #8396 from overleaf/jpa-split-test-mongo-user
[web] implement split test assignment based on mongo user

GitOrigin-RevId: d3e2dff6a5e925cfd0426e9ebfeb7b64dc803f42
2022-06-16 08:03:20 +00:00
Eric Mc Sween baaf4d4240 Merge pull request #8286 from overleaf/em-halt-on-error-backend
Stop on first error backend implementation

GitOrigin-RevId: 497b1ed2c13f544760d8ad8d029359db75275389
2022-06-07 08:02:21 +00:00
Jakob Ackermann a78bcee15f Merge pull request #8135 from overleaf/jpa-refactor-zonal-download
[misc] refactor handling of zone prefix in compile response

GitOrigin-RevId: f1f33d7d257854176f383bb5d786710f6b09f737
2022-05-26 08:03:53 +00:00
Jakob Ackermann f0bd6dda23 Merge pull request #7986 from overleaf/jpa-eslint-8
[misc] upgrade eslint packages to the latest version everywhere

GitOrigin-RevId: f1480d4a171acef82fb26c4aa54be3a6088b0ab3
2022-05-17 08:05:59 +00:00
Eric Mc Sween e0d91eaa26 Merge pull request #7906 from overleaf/em-downgrade-logs
Downgrade all INFO logs to DEBUG

GitOrigin-RevId: 05ed582ef0721fcada059f0ad158565f50feca27
2022-05-17 08:05:26 +00:00
Jakob Ackermann 52073a13a9 Merge pull request #7867 from overleaf/jpa-compile-group
[web] pass compileGroup to clsi-lb

GitOrigin-RevId: c15adbff27e702b3e0f29be5b57f7a9520d8d02f
2022-05-16 08:03:36 +00:00
Jakob Ackermann 264b107e39 Merge pull request #6927 from overleaf/jpa-split-clsi-metrics
[misc] split clsi metrics into multiple categories

GitOrigin-RevId: 964ab57b344b92383a2937beff50139bae273a0e
2022-03-02 09:02:52 +00:00
Jakob Ackermann 220daff655 Merge pull request #6384 from overleaf/jpa-fix-proxying-of-json
[web] fix proxying of responses from other services

GitOrigin-RevId: ac1ecc02c06e353f8379fadb47824a46672610e5
2022-01-18 09:03:30 +00:00
Eric Mc Sween b22df1dcba Merge pull request #5984 from overleaf/em-unnecessarily-async-functions
Clean up unnecessarily async functions

GitOrigin-RevId: 59f0f0a76b4436f3b99a09b747670d443bac4582
2021-12-07 09:03:33 +00:00
Eric Mc Sween 5fc6d7dcb3 Merge pull request #5740 from overleaf/em-gcp-logging-web
Improve GCP logging for web

GitOrigin-RevId: b304c87a3fe46c29189f665eb3daf22c23d6eb8f
2021-11-11 09:03:09 +00:00
Eric Mc Sween e5676a9643 Merge pull request #5648 from overleaf/em-revert-gcp-logging-web
Revert "Improve GCP logging for web"

GitOrigin-RevId: 92d446baf62108da1df92146eec12a2fe69d30ee
2021-11-02 09:03:29 +00:00
Eric Mc Sween 641b10cceb Merge pull request #5632 from overleaf/em-gcp-logging-web
Improve GCP logging for web

GitOrigin-RevId: 1198fab2e821a55563058171cfa435605216e337
2021-11-02 09:03:22 +00:00
Jakob Ackermann 358e8b7424 Merge pull request #5349 from overleaf/jpa-no-depreacted-api
[misc] fix eslint violations for node/no-depreacted-api

GitOrigin-RevId: 0f7d64984da9e789c4ab95381db34afb89fa1a94
2021-10-21 08:03:18 +00:00
Brian Gough cc1b73336a Merge pull request #4944 from overleaf/bg-use-file-line-errors
use file line errors in log output

GitOrigin-RevId: 6732b19552fe15431a70fbefbc572253c389c64e
2021-09-22 08:03:25 +00:00
Alexandre Bourdin 1094b0570b Merge pull request #4922 from overleaf/jpa-web-clsi-metrics
[web] add new metrics for tracking initial assign/switch of clsi backend

GitOrigin-RevId: 4fb15ee8727cb397d1e44a86efb7d4833626bc6b
2021-09-01 08:03:11 +00:00
Jakob Ackermann b8f2cc1b91 Merge pull request #4841 from overleaf/jpa-ho-clsi-user-project-persistence-take-2
[web] partition clsi persistence by project and user

GitOrigin-RevId: d42dca11dfbe329974c26d3b31eb9fcba359b785
2021-08-23 11:22:39 +00:00
Jakob Ackermann 8bbf6fb52c Merge pull request #4840 from overleaf/revert-4736-ho-clsi-user-id-mapping-minimal
Revert "Clsi user id mapping (mono repo)"

GitOrigin-RevId: 046bf21f1ab90f375d3945d8baa22600d8cdbf2b
2021-08-23 08:03:00 +00:00
Henry Oswald 2b7c7e84ef Clsi user id mapping (#4736)
* add user-id into clsi project mapping

* add user_id on backend group clsi key

GitOrigin-RevId: ebbf025f5cd88848b44f35a46045d112ea6b4c3b
2021-08-23 08:02:45 +00:00
Jakob Ackermann 63fb7a022a Merge pull request #4342 from overleaf/jpa-fix-clsi-server-id-handling
[misc] get current clsi server id directly from compile response

GitOrigin-RevId: 403e7ca35270a8937a0066fe2c8daf52ed71531d
2021-07-28 02:06:29 +00:00
Jakob Ackermann 5e773ce950 Merge pull request #4101 from overleaf/ae-settings-module
Migrate from `settings-sharelatex` to `@overleaf/settings`

GitOrigin-RevId: 9a298ba26382180c1351683c5fddc9004418c1e6
2021-07-08 02:08:28 +00:00
Jakob Ackermann 103873a013 Merge pull request #4275 from overleaf/jpa-drop-slow-compile-request-bailout
[misc] drop slow compile request bailout

GitOrigin-RevId: b5eec86878fbc7220dfe163241f29a3f5851c19f
2021-06-29 02:06:04 +00:00
Jakob Ackermann 15978e1510 Merge pull request #4262 from overleaf/jpa-bump-sync-overhead
[ClsiManager] bump sync overhead from 60s to 120s

GitOrigin-RevId: ce968a054419a3ad8e6b0bd0ff5897aae3643f8c
2021-06-25 02:06:32 +00:00
Jakob Ackermann e57be19ee3 Merge pull request #4261 from overleaf/jpa-fake-timed-out-response
[ClsiManager] respond w/ status=timedout for timed out compile requests

GitOrigin-RevId: 2937355bfd8d58870509ab40c2ba19f014f82a37
2021-06-25 02:06:28 +00:00
Jakob Ackermann 7b5aa23285 Merge pull request #4251 from overleaf/jpa-compile-request-timeout
[ClsiManager] timeout compile request

GitOrigin-RevId: 0213bb93bbdc9a75c07b27226b780dff5ee6f732
2021-06-24 02:07:05 +00:00
Jakob Ackermann 7db7cd4a49 [misc] merge pdf caching into main (#4033)
* [frontend] WIP: pdf caching using service worker -- squashed

Ref: 920fbaa00b31530f7c457a2d93bad5e553798057
Co-Authored-By: Brian Gough <brian.gough@overleaf.com>
Co-Authored-By: Eric Mc Sween <eric.mcsween@overleaf.com>

* [misc] add contentId into the URL for protecting PDF stream contents

* [misc] gracefully handle missing ranges in serviceWorker

* [misc] support PDF stream caching for anonymous users

* [misc] polish header names and add URL to error message when fetch fails

* [misc] polish event handler registration

* [misc] limit serviceWorker scope to /project/ -- trailing slash

This will block the service worker from intercepting requests on the
 project dashboard.

* [misc] add per-request feature flag for enabling PDF stream caching

* [misc] expose compile stats and timings to the frontend

* [misc] serviceWorker: support clsiServerId and compileGroup url params

* [misc] serviceWorker: polish header maps

* [misc] serviceWorker: drop TODO for p-limit -- the browser has a queue

* [misc] serviceWorker: drop verbose log message on every fetch

* [misc] cut down size of diff in backend code

* [misc] add test case for forwarding of pdf caching and metrics details

* [misc] serviceWorker: drop all the log lines

* [misc] serviceWorker: add boundary guards to the compile request regex

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: 4b291b4a4f2866cf07bccf8ec9068f33bbfdc916
2021-05-18 02:07:57 +00:00
Hugh O'Brien 1f3b01486e Merge pull request #3991 from overleaf/em-oversized-logs
Avoid logging project contents

GitOrigin-RevId: 6255ffebca83728bc69e64286ea40fd5aed244a1
2021-05-08 02:10:01 +00:00
Miguel Serrano f9871103bf Merge pull request #3949 from overleaf/msm-reenable-eslint-const-rule
Reenable eslint `prefer-const` rule

GitOrigin-RevId: 4f3825be8b8dff381095209085a36eaab76260d5
2021-05-06 02:09:14 +00:00
Alf Eaton 1be43911b4 Merge pull request #3942 from overleaf/prettier-trailing-comma
Set Prettier's "trailingComma" setting to "es5"

GitOrigin-RevId: 9f14150511929a855b27467ad17be6ab262fe5d5
2021-04-28 02:10:01 +00:00
Eric Mc Sween 38312fd7d3 Merge pull request #3557 from overleaf/jpa-reland-clear-persistance
[reland] cleanup pdf url composing and clear clsi persistance

GitOrigin-RevId: 59ac281d16564a64852ccc07dac0c99cadfba8f2
2021-01-22 03:04:48 +00:00
Eric Mc Sween db93fa3a8b Merge pull request #3556 from overleaf/jpa-clsi-persistance-query-param
[CompileController] enable clsi node persistence via query parameter

GitOrigin-RevId: 515814d6ad5832e69538ef6d63f81c61c66fd73f
2021-01-22 03:04:42 +00:00
Jakob Ackermann 3e3d683a2f Merge pull request #3545 from overleaf/revert-3517-jpa-clear-clsi-persistance
Revert "[ClsiManager] clear the clsi persistence when clearing the cache"

GitOrigin-RevId: fbaeff527f69526da1bfc32de37bbdf6081bcd23
2021-01-08 03:04:47 +00:00