* [monorepo] avoid corepack network requests
- Download yarn via corepack as the first step in all the docker files
- Turn off networking in corepack
- Do not run things in the upstream node image
Instead, use the monorepo image, or base layer in all the services.
- Always build the base layer when running tests (uses cache)
* [monorepo] install corepack in shared place
* [clsi-lb] remove unrelated changes
* [web] add missing DC_RUN_FLAGS
* [monorepo] only rebuild test images locally
Also remove spurious build config in docker-compose.ci.yml.
* [server-ce] test: make yarn files available to host-admin and e2e
* [monorepo] put the corepack install snippet in a few more places
GitOrigin-RevId: 38005016ae5a708e12295e246269d6c18fece937
Migrates the Overleaf monorepo package manager from npm (v11) to Yarn 4 (v4.9.1) using node-modules linker mode.
GitOrigin-RevId: 50d32ab01955c15e29679eff9e9e9cfb897fab2d
* using CLSI logic for fetching the project contents and skip the .zip export
* Use unique conversion directory for project-to-docx export to avoid corrupting the shared compile
directory when a compile runs concurrently
* Remove X-Accel-Buffering header — not needed as CLSI does not run behind nginx
* moving log before sending the data
* Return CLSI stream directly instead of buffering to disk on web
Previously convertProjectToDocx wrote the CLSI response to a temp file
on disk, then the controller read it back to stream to the client.
Now the stream is returned directly and piped to the response,
avoiding unnecessary disk I/O on the web server.
* Use href redirect for docx export instead of fetching blob into memory
* making functions and files more generic so they can be used in future for other documents exports as well
* adding export-docx split test
* adding unit tests
* adding cypress E2E test
* format:fix
* renaming the route to download from convert
* adding new icon for export docx button
* format:fix
* remove unused showExportDocumentErrorToast export and adding guard against invalid Content-Length header from CLSI
* format:fix
* refactor(clsi): move promisify(parse) into RequestParser
* refactor: generic conversion endpoint with type as route
param
* refactor: use type→extension map for validated conversion types
* refactor(clsi): remove --standalone flag and fix rejection test
* fixing the href in cypress test
* renaming function
* adding type to Metrics.inc
* fix: rename exportProjectDocument, add WithLock wrapper and metrics type label
* format:fix
* fix: hide docx export from anonymous users and add WithLock wrapper
* format fix
* remove redundant Content-Length validation from DocumentConversionManager
* format:fix
* removing trailing icon
GitOrigin-RevId: e9764fefac2c4b625d23be9e942ea4a8b283c70d
* [clsi] handle draft mode and tikzexternalize as part of sync phase
* [clsi] emit empty string from SafeReader on ENOENT
* [clsi] persist history state after clearing dirty state without changes
GitOrigin-RevId: d9dcd2e6887017f7935b5e95bdbdc6e11a3b18f5
* [clsi] remove locking from docker actions
Start:
- We have an in-memory lock on the compile request
Destroy:
- as part of run: see above
- as part of cleanup: we check the last access time now, so it cannot
happen concurrent to compiling anymore.
Co-authored-by: Anna Claire Fields <anna.fields@overleaf.com>
* [clsi] update comment
---------
Co-authored-by: Anna Claire Fields <anna.fields@overleaf.com>
GitOrigin-RevId: a58df45416ae31c0b38d5efec7f9371d747303df
* [clsi] avoid server error when clearing cache while compiling
* [clsi] tweak API around releasing locks
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
---------
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: d3f171467d3bc26941758dd333f30049b37a05c8
* [k8s] clsi-cache: double the number of shards
* [monorepo] add missing clsi-cache env vars to dev-env
* [clsi] flip direction of clsi-cache shard migration
* [clsi] remove upper bound from clsi-cache shard migration
GitOrigin-RevId: a325a11c3ac9e22a12ad2d8ea802b91d2e175e24
* handle old versions of latexmk in run count extraction
the log lines for the run number change from stderr to stdout in TL2022
* extend SimpleLatexFileTest to include TL2017
* reset metrics for each scenario in SimpleLatexFileTests
* fix buildscript merge conflict
GitOrigin-RevId: fb74f2025d21ddf43be6a3b90ac6f7df4d975db6
* [clsi] only download history snapshot from clsi-cache when enabled
* [clsi-perf] migrate to compile from history mode
GitOrigin-RevId: 2dd54e032bd85d6335488741c039a5a1bd60090d
* [saas-e2e] test gallery templates with binary file
* [rails] add make target for fixing rubocop errors
* [rails] migrate compiles of conversions/submissions to history mode
* [rails] forward version to clsi request
* [rails] trim down compile request
* [saas-e2e] source v1 secrets after make install
GitOrigin-RevId: 65269e1df1051c9f3b4f1813d2e9dcf32a01be50
* [clsi] do not overwrite last access during initial scan
* [clsi] cleanup submission cache 5-10min after startup
* [clsi] address review comments
GitOrigin-RevId: e03beec1b3deaee50629ada72b0242a8a2b2ae66
* [clsi] initial implementation of compile from history
* [clsi] copy changes
* [saas-e2e] extend test case with nested folder
* [saas-e2e] add test case for tracked changes
* [web] fix accumulating changes from multiple chunks
* [web] optimize size check for compile request payload
* [clsi] deduplicate globalBlobs
* [clsi] add validation for request body details
* [clsi] add metrics for compile from history
* [clsi] download binary files concurrently
* [clsi] skip download of empty file blob
* [clsi] break down e2e compile time metric by compileFromHistory
GitOrigin-RevId: 0dadef93e89d8a172c35cb130a1042d9d1bec42a
* [monorepo] switch all output file reads to clsi-nginx
* [clsi-lb] allow gallery download requests
* [terraform] clsi: use nginx.conf from clsi service
* [clsi] fix flakey tests
* [clsi] replace alias with rewrite and root in nginx config
* [k8s] clsi-lb: expose download port on internal service
* [web] add explicit endpoint for downloading all output files
Serve the output.zip endpoint from clsi.
* [clsi] fix regex for latexqc submission ids
Previously, we only handled template submission ids.
GitOrigin-RevId: 6c3b21b01ec41ae767530b14aac31fbe3d640dd5
* [monorepo] remove PII and variables from error messages
Exclusions:
- scripts
- tests
- fuzzing
- SplitTestManager (messages are sent to admin frontend)
- Group setup (we may want an error per unique tuple)
- sharejs (unused types; text type errors are shadowed already)
- history-v1 error messages that are used by the ErrorRecorder
- errors that flag issues with configuration/call signatures
I've used these search terms for finding unwanted error messages:
- new Error(`
- new Error\(\n\s+` (regex search)
- new OError(`
- new OError\(\n\s+` (regex search)
* [web] throw NotFoundError from ProjectLocator
* [github-sync] fix OError.tag call in script
Co-authored-by: Jessica Lawshe <jessica.lawshe@overleaf.com>
* [templates] revert changes to test client
---------
Co-authored-by: Jessica Lawshe <jessica.lawshe@overleaf.com>
GitOrigin-RevId: 736857a4fc5d9bfb0f8cb03e0f004eda87e5a220
* [monorepo] move retries from the outside to the per-test level
* [web] increase hookTimeout in CI
* [monorepo] consolidate test retries
- do not retry unit tests
- only retry in ci, optionally locally with "RETRIES=3 make test..."
- add retries for web
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
---------
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: 86e45edcfb087d18e0e957ad6df9a6105dcd5770
* [clsi] tweak logging for clsi-cache
- Use `clsi-cache` identifier on log line
- Add shard to context
- Record nFiles on "too many entries for tar" error
* [clsi] do not trip clsi-cache circuit breaker on ENOENT errors
These can happen when an output/compile-dir is purged while we download
files.
GitOrigin-RevId: ffa73ef312bce5232ef72e3b81966bb6e14d2255
