* [monorepo] consolidate clsi-lb host/ip env-vars
Target env-var is CLSI_LB_HOST. Keep CLSI_LB_IP populated for a week.
* [clsi] initial version of /convert/pdf-to-jpeg
* [rails] use fake-secrets in CI and Codespaces
* [rails] adapt tests for using clsi to convert PDFs to image
* [rails] add rake task for comparing clsi conversion with transloadit
* [clsi] double check that output.jpg is a regular file
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [clsi] fix composing basename
* [monorepo] fix clsi-lb host env-var post merge
* [monorepo] sort dev-environment.env hosts
* [rails] use local pdf file rather than downloading it again
Download from the old renderer code path still. It's dead code.
* [terraform] clsi: enable pdf to jpg conversion
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: 5ecaa8559d299486340bb3961f06b29f7c4dfcca
* [clsi] add request flag for isCompileFromHistory
* [clsi] derive cacheKey for history snapshot from compile dir
* [clsi] migrate convert project to document to compile from history
* [clsi] address review feedback
* [web] determine root doc at the time of converting the project
* [web] wait for flush before starting document conversion
* [saas-e2e] add tests for root doc override when converting project
GitOrigin-RevId: 71c578030949b89f3a74e7f7ab882dfa9c98c17a
* [docstore] add useSecondary flag to projectHasRanges
The rev-check for unarchiving always consults with the primary.
Two extra changes:
- Add a projection argument to peekDoc in order to skip lines download
from projectHasRanges.
- Add one retry to peekDoc to reduce chances of surfacing a rev-check
violation.
* [web] resync_projects: use the secondaries for all reads
* [web] add default value for useSecondary
* [docstore] add default value for useSecondary
* [k8s] docstore: set MONGO_HAS_SECONDARIES=true
GitOrigin-RevId: f15ec4fdc1cabe74c1eab87bec85f28d6f7a587d
* [history-v1] add endpoint for downloading latest zip
* [web] address review feedback
* [web] tests: do not overwrite db.projects.overleaf, extend it
* [web] set includeReferer flag from downloading zip
GitOrigin-RevId: e63e549f004230086f82eccf03b43fd62bde6071
* [web] cleanup archived split-test assignments from user record on login
Co-authored-by: Anna Claire Fields <anna.fields@overleaf.com>
* [migrations] purge archived split tests from all users
Co-authored-by: Anna Claire Fields <anna.fields@overleaf.com>
* [web] add missing mock and update snapshot test
* [web] gracefully access db.users.splitTests
---------
Co-authored-by: Anna Claire Fields <anna.fields@overleaf.com>
GitOrigin-RevId: bd185074a402556d7b7c812208cf834dd52b27a5
* [web] add includeReferer flag to SplitTestHandler.getAssignment
* [web] tests: migrate User.getSplitTestAssignment to async/await
I don't want to fight with callbacks and optional arguments. Just move
it to async/await. New tests should use async/await, so there is no
point in making this work in callback-hell.
* [web] remove unused URL import
GitOrigin-RevId: 6251001e6ba7354f704fa663be8ef365ca0b9d23
* [monorepo] avoid corepack network requests
- Download yarn via corepack as the first step in all the docker files
- Turn off networking in corepack
- Do not run things in the upstream node image
Instead, use the monorepo image, or base layer in all the services.
- Always build the base layer when running tests (uses cache)
* [monorepo] install corepack in shared place
* [clsi-lb] remove unrelated changes
* [web] add missing DC_RUN_FLAGS
* [monorepo] only rebuild test images locally
Also remove spurious build config in docker-compose.ci.yml.
* [server-ce] test: make yarn files available to host-admin and e2e
* [monorepo] put the corepack install snippet in a few more places
GitOrigin-RevId: 38005016ae5a708e12295e246269d6c18fece937
* [web] make double compile test parameters configurable via env vars
* [k8s] web: enable double compile test for free compiles on n4 instances
GitOrigin-RevId: 3a5cb8ed6d044fcf3f4c0d2b9d252326bac48511
* [web] check for missing module tsconfig in CI
* [web] add missing tsconfig into test_unit container
GitOrigin-RevId: 7b861f8e68f961e3455b72b5660cf3825389c656
* [history-v1] fix re-running test_acceptance_run
Preserve the contents of the migrations collection. Deleting the entries
does not "undo" the migrations. On re-run east would try applying all
migrations and fail as things were already applied.
* [history-v1] remove guard migration before running tests
GitOrigin-RevId: e6eeafd58215e5148dd70c37c7a87d84e0a12bf3
* [github] code spaces: add extra-split-tests.json
* [github] address review feedback
* [github] add missing bind mount
* [monorepo] mount split test configs with :ro
* [monorepo] only mount backup/split-tests into web
GitOrigin-RevId: 6334a44599a2c8bb79bf1ad698e656c8bee3992b
* [metrics] mongo: fail when command monitoring is not available
* [metrics] mongo: add optional client label to pool metrics
* [web] collect mongo stats on native client
* [metrics] mongo: record namespace of find commands
* [metrics] mongo: add counter for all the commands with collection label
* [web] add missing mock
GitOrigin-RevId: 9f378d8aa8d7167f56cf512681d63ef115c6dd98
* [monorepo] move building of nginx proxy into Makefile
* [v1] add fake-secrets.env
* [github] add copilot-setup-steps.yml workflow
* [make] debug permissions
* [monorepo] use the host uid/gid for monorepo service
* [web] populate DOCKER_USER for test_frontend_ct
* [github] run npm install outside docker in Copilot sandbox
* [monorepo] add comment for building nginx-proxy image locally
* [monorepo] change uid/gid of node user in cypress image
* [monorepo] low-level rewrite of passwd and groups to match host user
GitOrigin-RevId: d74a5801f0318bab7e7f460374255426706002a3
* [monorepo] run npm install before top-level lint/format
* [monorepo] extend lint_fix for web
* [monorepo] bin/run: silence docker compose
We do not need it to print the newly created containers name.
* [monorepo] run format:fix
* [monorepo] run all the tasks or services/web/lint_fix_other in docker
GitOrigin-RevId: 997b360d1136f466c8e6bf8c5ad0e1189406d1f1
* [web] enable async local storage on all the endpoints
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [web] deduplicate getting the project when loading the editor
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [web] use ProjectAccess state for computing analytics segmentation
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [web] restore ownership of active flag and deferredTpdsFlushCounter
* [web] add missing await
* [web] update unit tests
* [web] add metrics for project access caching
* [web] add missing test mock
* [web] invalidate async local storage when changing project access
* [web] deduplicate project lookup when checking for token access
* [web] add helper function for getting cached ProjectAccess
* [web] add acceptance test for caching of ProjectAccess
* [web] account for saas-only project access in tests
* Revert "[web] enable async local storage on all the endpoints"
This reverts commit 1b82f3b935040e8cfd180d1f6bf4183a655580e2.
* [web] add async local storage to project endpoints in top-50
* [web] invalidate async local storage for project access from modules
Co-authored-by: Jessica Lawshe <jessica.lawshe@overleaf.com>
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Jessica Lawshe <jessica.lawshe@overleaf.com>
GitOrigin-RevId: 3eea7956b24e6f937dc1c17948681063d4dca3ea
* [project-history] add best effort flush when cloning project
* [web] update labels in clone project modal for admins
* [project-history] do not shadow history flush failure
* [web] fix accessible label for 'Add comment' button
* [chat] clone comment threads when cloning project with ranges
GitOrigin-RevId: ef30204c8a94b3d6204d56dcca2f62a46319996b
* [github] code spaces: add more standard users with a common password
* [github] code spaces: automatically share dev-env with Overleaf org
* [github] code spaces: merge split tests rather than replacing them
* [github] code spaces: ts-check the setup script in web
GitOrigin-RevId: 1a86ac7e0304d47e68290352f58a0e193eed1d77
* [web] add consistent aria-label to editing/reviewing toggle
* [docstore] add endpoint for getting all docs with ranges
* [history-v1] fix schema of chunkId when deleting old history chunk
* [web] skip duplicate project lookup for resolving rootDocPath
* [web] ignore new limits for root doc path when making debug copy
* [web] allow admins to clone projects with ranges and entire history
* [web] fix tests
* [history-v1] re-order params for cloning project
* [web] fix duplicate import of logger after merge
* [project-history] re-order params for cloning project history metadata
GitOrigin-RevId: 7fa35b4f90885dd453150a348d491ba0ec8de412
* [web] serve site manifest as application/json
* [monorepo] bail out early from dev-env-setup when inside devcontainer
* [monorepo] move utils for waiting for services into shared bash script
* [monorepo] move domain related env-vars into dedicated .env file
The .env file will be overridden in GitHub Code Spaces.
* [monorepo] remove unused initial_mongo_setup Make target
It was previously used for the mongo replica set setup.
That setup is a docker init script now.
* [monorepo] add flag for silencing docker build output
* [monorepo] add integration for GitHub Code Spaces
* [web] make prettier happy
* [web] restore no use-credentials for manifest outside code spaces
* [monorepo] run make install as part of post-start again
GitOrigin-RevId: 8c2cc3b969b7e3e8f9c5dac778192c9412cbf4df
* [clsi] handle draft mode and tikzexternalize as part of sync phase
* [clsi] emit empty string from SafeReader on ENOENT
* [clsi] persist history state after clearing dirty state without changes
GitOrigin-RevId: d9dcd2e6887017f7935b5e95bdbdc6e11a3b18f5
* [clsi] remove locking from docker actions
Start:
- We have an in-memory lock on the compile request
Destroy:
- as part of run: see above
- as part of cleanup: we check the last access time now, so it cannot
happen concurrent to compiling anymore.
Co-authored-by: Anna Claire Fields <anna.fields@overleaf.com>
* [clsi] update comment
---------
Co-authored-by: Anna Claire Fields <anna.fields@overleaf.com>
GitOrigin-RevId: a58df45416ae31c0b38d5efec7f9371d747303df
* [monorepo] fix prettier formatting
* [monorepo] use a single prettier invocation for monorepo-check
GitOrigin-RevId: cf0cc2293b228be707ff23bdb3c5781c1f8d69f3
The req.body.rootResourcePath has been shipped three weeks ago, so it's
unlikely to trip up stale editor sessions.
For now, only block node-fetch and log the rest.
GitOrigin-RevId: 541189675f68fdcab09f4b409b4143024a29f94a
* [web] migrate compile of templates/example project to history mode
* [saas-e2e] give clsi-cache more time to process the jobs
* [web] add comment on in-memory project update
Co-authored-by: Brian Gough <briangough@users.noreply.github.com>
* [saas-e2e] add retries to clsi-cache check
* [saas-e2e] remove spurious retry trigger
* [saas-e2e] remove broken assertion
---------
Co-authored-by: Brian Gough <briangough@users.noreply.github.com>
GitOrigin-RevId: 8766662cbe0ab5520c5c79d86fa6f774e012b3da
* [web] gracefully handle broken histories when compiling from history
* [web] trim down schema
GitOrigin-RevId: 97d59b31eb25644d7de1194a45281def6982b130
* [clsi] avoid server error when clearing cache while compiling
* [clsi] tweak API around releasing locks
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
---------
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: d3f171467d3bc26941758dd333f30049b37a05c8
* [web] enable clsi-cache for all compiles for 20min during outages
* [web] populate clsi-cache when compile from history is enabled
* [web] hide history snapshot from frontend
GitOrigin-RevId: e46c8b5ab401ebdc2407e44eb95439e4d43400da
* [k8s] clsi-cache: double the number of shards
* [monorepo] add missing clsi-cache env vars to dev-env
* [clsi] flip direction of clsi-cache shard migration
* [clsi] remove upper bound from clsi-cache shard migration
GitOrigin-RevId: a325a11c3ac9e22a12ad2d8ea802b91d2e175e24
* [clsi] only download history snapshot from clsi-cache when enabled
* [clsi-perf] migrate to compile from history mode
GitOrigin-RevId: 2dd54e032bd85d6335488741c039a5a1bd60090d
* [saas-e2e] test gallery templates with binary file
* [rails] add make target for fixing rubocop errors
* [rails] migrate compiles of conversions/submissions to history mode
* [rails] forward version to clsi request
* [rails] trim down compile request
* [saas-e2e] source v1 secrets after make install
GitOrigin-RevId: 65269e1df1051c9f3b4f1813d2e9dcf32a01be50
* [clsi] do not overwrite last access during initial scan
* [clsi] cleanup submission cache 5-10min after startup
* [clsi] address review comments
GitOrigin-RevId: e03beec1b3deaee50629ada72b0242a8a2b2ae66
* [saas-e2e] port history tests to old history
* [web] fix incremental compile from history for old history ids
* [saas-e2e] tweak return type
GitOrigin-RevId: 2c89b570647c292c720cd0d02b6188f66e3e3a69
* [storybook] invalidate babel cache if needed
* [storybook] defer import until after overriding env var
GitOrigin-RevId: 735e6c794cb61f72b7a5ccbe9d6ae310b577b727
* [clsi] initial implementation of compile from history
* [clsi] copy changes
* [saas-e2e] extend test case with nested folder
* [saas-e2e] add test case for tracked changes
* [web] fix accumulating changes from multiple chunks
* [web] optimize size check for compile request payload
* [clsi] deduplicate globalBlobs
* [clsi] add validation for request body details
* [clsi] add metrics for compile from history
* [clsi] download binary files concurrently
* [clsi] skip download of empty file blob
* [clsi] break down e2e compile time metric by compileFromHistory
GitOrigin-RevId: 0dadef93e89d8a172c35cb130a1042d9d1bec42a
* [web] increase timeout for downloads from clsi
* [web] detach request timeout from streaming timeout
* [web] align timeout for clsi-cache downloads with clsi downloads
* [web] destroy stream when bailing out early from aborted requests
GitOrigin-RevId: cc4cad9d684214b2eee0f5fc43513e430ceb0977
* [monorepo] switch all output file reads to clsi-nginx
* [clsi-lb] allow gallery download requests
* [terraform] clsi: use nginx.conf from clsi service
* [clsi] fix flakey tests
* [clsi] replace alias with rewrite and root in nginx config
* [k8s] clsi-lb: expose download port on internal service
* [web] add explicit endpoint for downloading all output files
Serve the output.zip endpoint from clsi.
* [clsi] fix regex for latexqc submission ids
Previously, we only handled template submission ids.
GitOrigin-RevId: 6c3b21b01ec41ae767530b14aac31fbe3d640dd5
* [dependency-management] check eslint and prettier in Jenkins pipeline
* [npm-overrides-helper] check eslint and prettier in Jenkins pipeline
GitOrigin-RevId: 647f105babd536d0962ac2dbb3d6a0235ba73976
* [web] validate and parse email using EmailHelper when ratelimiting
* [web] use a fake email for rate-limiting ldap logins in Server Pro
GitOrigin-RevId: 27ea7724319e06c4d64ac81e1155dcab558da99c
11 years ago, the db.projects collection was storing doc lines in the
file-tree/rootFolder. Any operations on the project that did not need
those lines were benefitting from excluding all those entries from the
file-tree. These days, the verbose exclusions are not useful anymore and
merely add load on mongo.
REF: 9805c6a9ff
GitOrigin-RevId: 89f544688934c1ed1ca98877ffbe8baefe66c126
Jakob Ackermann