* [web] use client-side filtering for split tests in admin panel
Filtering on non-indexed fields is not allowed with notablescan.
* [web] add initial value for DEFAULT_FILTERS.active
* [web] unset error before fetching split tests
GitOrigin-RevId: 0813873c5a90d57c4fdbfd0043fcc325f00408aa
* [migrations] only apply non-blocking migrations in CI
* [migrations] use "yarn run migrations" instead of "npm run migrations"
GitOrigin-RevId: d9f96db398adf2febcd82333295fb8d5eab741bb
* [monorepo] fix handling of quoted MOCHA_GREP
* [agents] refine guidance for running tests
- nudge it towards scoped module targets
- nudge it to really run e2e tests when changed
- nudge it towards a11y cypress matchers
- recommend quoting for MOCHA_GREP
* [web] fix forwarding of mocha args from module Makefile
* [monorepo] use BRANCH_NAME_TAG_SAFE for building the ci image name
* [web] also make BRANCH_NAME_TAG_SAFE available in module Makefile
(It's not really needed, but will make copilot happy.)
GitOrigin-RevId: bfda3c8fe2b4f857ef111ac63f94b35a82ad75f8
* [web] _getUserCompileLimits: use owner.analyticsId directly
* [web] deduplicate project lookups for compile requests
* [web] add test case for prefetched project
GitOrigin-RevId: 0e140619c874accf8673153ed90af2ce70cf9234
* [web] split-tests: add helpers to using an already fetched mongo user
* [web] split-tests: add ignoreOverrides option to getAssignment
* [web] split-tests: reuse already fetched mongo users where possible
* [web] notifications: fetch users in bulk for split-test assignment
* [web] split-tests: use session for assignment where possible
* [web] split-tests: pass req/res down into domain-capture hook
* [web] split-tests: explicitly throw early
* [web] split-tests: refine jsdoc types
* [web] mark analyticsId and labsProgram as required in user model
* [web] split-tests: refine jsdoc types
* Revert "[web] split-tests: pass req/res down into domain-capture hook"
This reverts commit 897a158384cd2b3e416fc34495b20d1e2a0e68f0.
GitOrigin-RevId: ea4afd15c2506f369827b5193fd16d220d3c7d0e
* [web] adopt free/premium compile backend class
* [rails] adopt free compile backend class for pdf conversion
GitOrigin-RevId: e12c99a27363f5654132c2fcdc6c5d8ccc591868
* [web] always provide an analyticsId when registering new users
* [web] add recordEventForMongoUser and use it during login
* [web] use recordEventForMongoUser/recordEventForSession where possible
* [web] use session cache for analyticsId when refreshing features
* [web] add setUserPropertyForMongoUser and use it where possible
* [web] use setUserPropertyForSessionInBackground where possible
* [web] use mongo user for analyticsId when setting utm user properties
* [web] admin-panel: user updated user when sending events
* [web] enforce labsProgram being populated on user object
Co-authored-by: Andrew Rumble <andrew.rumble@overleaf.com>
* [web] revert FeaturesUpdater.refreshFeatures change
* [web] labs: move session maintenance into handler
* [web] analyticsMiddleware: ensure that all users have labsProgram field
* [web] migrate a few more events to recordEventForSession
* [migrations] mark back-fill of db.users.analyticsId as required
* [web] setUserPropertyForSessionInBackground: throw early
---------
Co-authored-by: Andrew Rumble <andrew.rumble@overleaf.com>
GitOrigin-RevId: 19c26bc36abd8c11f0a8f458d25abfb7cbda0bb7
* [web] throw early in UserCreator.createNewUser with missing analyticsId
* [migrations] rerun back-fill of db.users.analyticsId
GitOrigin-RevId: eabf16326192fca4dbea00ad643cee4315c426c6
* [server-ce] test: the first startWith must resetData when using dataDir
* [server-ce] test: fix order of creating user and resetting DB
GitOrigin-RevId: 2a6e6344919fe48d41704e1a3cb39c6e22b5a924
* [web] add migration for back-filling db.users.labsProgram=false
Co-authored-by: Andrew Rumble <andrew.rumble@overleaf.com>
* [web] include user.labsProgram=false in session
---------
Co-authored-by: Andrew Rumble <andrew.rumble@overleaf.com>
GitOrigin-RevId: 0b3322007f6a36d2999d646869dc84f2c6ad4d2d
* [monorepo] provide clsi-sharded dev-env setup
* [monorepo] provide sharded clsi-nginx when using clsi-sharded
* [clsi-perf] run on all instances again
* [monorepo] switch default TL to 2025.1 in clsi (same as web)
* [monorepo] avoid starting base clsi when running clsi-perf with clsi-sharded
* [clsi-lb] use 3s check interval in clsi-sharded mode
GitOrigin-RevId: 3cae8f0c1db69dbd76b94b45dc45b3abb2fa0a83
* [clsi] forward instanceType to web
* [clsi] expose zone in compile response
* [clsi-perf] add new fields for instanceType and zone
* [clsi] expose isPreEmptible flag in compile response
* [clsi-perf] also add isPreEmptible field
* [clsi] rename isPreEmptible -> isSpotInstance
GitOrigin-RevId: 2bc2d772e0b14da83afb51f5f3e4fefd3f60b733
* [editor-core] avoid duplicate non-bmp check on InsertOp.insertion
* [editor-core] remove non-bmp check from StringFileData
Existing content may contain non-bmp characters. New edits with non-bmp
input are rejected via non-bmp check in InsertOp constructor.
GitOrigin-RevId: e9ef3a4a8c4696e8dc751325608a788351791d1c
This was useful when iterating on the pdf-caching feature.
PDF caching has been stable for years. We can remove the logging now.
GitOrigin-RevId: 826c91c1e45d3dae1bb3ce5aeb7b6a3f9283392c
* [web] gracefully handle notifications being down on project dashboard
* [monorepo] remove haproxy from service-to-service communication
Co-authored-by: Anna Claire Fields <acfields11@gmail.com>
* [monorepo] fix OVERLEAF_LATEXQC_API_BASE_URL in dev-env
* [SaaS test fix] update history v1 export base url
---------
Co-authored-by: Anna Claire Fields <acfields11@gmail.com>
GitOrigin-RevId: cf355ef061412a1b57b5383cb60d4be574055af5
* [web] determine rootResourcePath if not set in compile request
This affects the following compile requests:
- compile request from "Download PDF" button on project dashboard
Pre-resolving the rootResourcePath for all projects is too expensive.
Adding an endpoint to resolve it is too slow and doubles project auth
check.
- custom frontends
Can't do much about these. I tried blocking them, requests still come
in after weeks of returning 400s.
The rootDoc maintenance methods now return {rootResourcePath,rootDocId}
rather than just the rootDocId. This avoids an extra project lookup when
the rootDocId was updated.
GitOrigin-RevId: ab7c539b7ffd162bde0903fe1040a48834a245f5
* [monorepo] consolidate clsi-lb host/ip env-vars
Target env-var is CLSI_LB_HOST. Keep CLSI_LB_IP populated for a week.
* [clsi] initial version of /convert/pdf-to-jpeg
* [rails] use fake-secrets in CI and Codespaces
* [rails] adapt tests for using clsi to convert PDFs to image
* [rails] add rake task for comparing clsi conversion with transloadit
* [clsi] double check that output.jpg is a regular file
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [clsi] fix composing basename
* [monorepo] fix clsi-lb host env-var post merge
* [monorepo] sort dev-environment.env hosts
* [rails] use local pdf file rather than downloading it again
Download from the old renderer code path still. It's dead code.
* [terraform] clsi: enable pdf to jpg conversion
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: 5ecaa8559d299486340bb3961f06b29f7c4dfcca
* [clsi] add request flag for isCompileFromHistory
* [clsi] derive cacheKey for history snapshot from compile dir
* [clsi] migrate convert project to document to compile from history
* [clsi] address review feedback
* [web] determine root doc at the time of converting the project
* [web] wait for flush before starting document conversion
* [saas-e2e] add tests for root doc override when converting project
GitOrigin-RevId: 71c578030949b89f3a74e7f7ab882dfa9c98c17a
* [docstore] add useSecondary flag to projectHasRanges
The rev-check for unarchiving always consults with the primary.
Two extra changes:
- Add a projection argument to peekDoc in order to skip lines download
from projectHasRanges.
- Add one retry to peekDoc to reduce chances of surfacing a rev-check
violation.
* [web] resync_projects: use the secondaries for all reads
* [web] add default value for useSecondary
* [docstore] add default value for useSecondary
* [k8s] docstore: set MONGO_HAS_SECONDARIES=true
GitOrigin-RevId: f15ec4fdc1cabe74c1eab87bec85f28d6f7a587d
* [history-v1] add endpoint for downloading latest zip
* [web] address review feedback
* [web] tests: do not overwrite db.projects.overleaf, extend it
* [web] set includeReferer flag from downloading zip
GitOrigin-RevId: e63e549f004230086f82eccf03b43fd62bde6071
* [web] cleanup archived split-test assignments from user record on login
Co-authored-by: Anna Claire Fields <anna.fields@overleaf.com>
* [migrations] purge archived split tests from all users
Co-authored-by: Anna Claire Fields <anna.fields@overleaf.com>
* [web] add missing mock and update snapshot test
* [web] gracefully access db.users.splitTests
---------
Co-authored-by: Anna Claire Fields <anna.fields@overleaf.com>
GitOrigin-RevId: bd185074a402556d7b7c812208cf834dd52b27a5
* [web] add includeReferer flag to SplitTestHandler.getAssignment
* [web] tests: migrate User.getSplitTestAssignment to async/await
I don't want to fight with callbacks and optional arguments. Just move
it to async/await. New tests should use async/await, so there is no
point in making this work in callback-hell.
* [web] remove unused URL import
GitOrigin-RevId: 6251001e6ba7354f704fa663be8ef365ca0b9d23
* [monorepo] avoid corepack network requests
- Download yarn via corepack as the first step in all the docker files
- Turn off networking in corepack
- Do not run things in the upstream node image
Instead, use the monorepo image, or base layer in all the services.
- Always build the base layer when running tests (uses cache)
* [monorepo] install corepack in shared place
* [clsi-lb] remove unrelated changes
* [web] add missing DC_RUN_FLAGS
* [monorepo] only rebuild test images locally
Also remove spurious build config in docker-compose.ci.yml.
* [server-ce] test: make yarn files available to host-admin and e2e
* [monorepo] put the corepack install snippet in a few more places
GitOrigin-RevId: 38005016ae5a708e12295e246269d6c18fece937
* [web] make double compile test parameters configurable via env vars
* [k8s] web: enable double compile test for free compiles on n4 instances
GitOrigin-RevId: 3a5cb8ed6d044fcf3f4c0d2b9d252326bac48511
* [web] check for missing module tsconfig in CI
* [web] add missing tsconfig into test_unit container
GitOrigin-RevId: 7b861f8e68f961e3455b72b5660cf3825389c656
* [history-v1] fix re-running test_acceptance_run
Preserve the contents of the migrations collection. Deleting the entries
does not "undo" the migrations. On re-run east would try applying all
migrations and fail as things were already applied.
* [history-v1] remove guard migration before running tests
GitOrigin-RevId: e6eeafd58215e5148dd70c37c7a87d84e0a12bf3
* [github] code spaces: add extra-split-tests.json
* [github] address review feedback
* [github] add missing bind mount
* [monorepo] mount split test configs with :ro
* [monorepo] only mount backup/split-tests into web
GitOrigin-RevId: 6334a44599a2c8bb79bf1ad698e656c8bee3992b
* [metrics] mongo: fail when command monitoring is not available
* [metrics] mongo: add optional client label to pool metrics
* [web] collect mongo stats on native client
* [metrics] mongo: record namespace of find commands
* [metrics] mongo: add counter for all the commands with collection label
* [web] add missing mock
GitOrigin-RevId: 9f378d8aa8d7167f56cf512681d63ef115c6dd98
* [monorepo] move building of nginx proxy into Makefile
* [v1] add fake-secrets.env
* [github] add copilot-setup-steps.yml workflow
* [make] debug permissions
* [monorepo] use the host uid/gid for monorepo service
* [web] populate DOCKER_USER for test_frontend_ct
* [github] run npm install outside docker in Copilot sandbox
* [monorepo] add comment for building nginx-proxy image locally
* [monorepo] change uid/gid of node user in cypress image
* [monorepo] low-level rewrite of passwd and groups to match host user
GitOrigin-RevId: d74a5801f0318bab7e7f460374255426706002a3
* [monorepo] run npm install before top-level lint/format
* [monorepo] extend lint_fix for web
* [monorepo] bin/run: silence docker compose
We do not need it to print the newly created containers name.
* [monorepo] run format:fix
* [monorepo] run all the tasks or services/web/lint_fix_other in docker
GitOrigin-RevId: 997b360d1136f466c8e6bf8c5ad0e1189406d1f1
* [web] enable async local storage on all the endpoints
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [web] deduplicate getting the project when loading the editor
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [web] use ProjectAccess state for computing analytics segmentation
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [web] restore ownership of active flag and deferredTpdsFlushCounter
* [web] add missing await
* [web] update unit tests
* [web] add metrics for project access caching
* [web] add missing test mock
* [web] invalidate async local storage when changing project access
* [web] deduplicate project lookup when checking for token access
* [web] add helper function for getting cached ProjectAccess
* [web] add acceptance test for caching of ProjectAccess
* [web] account for saas-only project access in tests
* Revert "[web] enable async local storage on all the endpoints"
This reverts commit 1b82f3b935040e8cfd180d1f6bf4183a655580e2.
* [web] add async local storage to project endpoints in top-50
* [web] invalidate async local storage for project access from modules
Co-authored-by: Jessica Lawshe <jessica.lawshe@overleaf.com>
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Jessica Lawshe <jessica.lawshe@overleaf.com>
GitOrigin-RevId: 3eea7956b24e6f937dc1c17948681063d4dca3ea
* [project-history] add best effort flush when cloning project
* [web] update labels in clone project modal for admins
* [project-history] do not shadow history flush failure
* [web] fix accessible label for 'Add comment' button
* [chat] clone comment threads when cloning project with ranges
GitOrigin-RevId: ef30204c8a94b3d6204d56dcca2f62a46319996b
* [github] code spaces: add more standard users with a common password
* [github] code spaces: automatically share dev-env with Overleaf org
* [github] code spaces: merge split tests rather than replacing them
* [github] code spaces: ts-check the setup script in web
GitOrigin-RevId: 1a86ac7e0304d47e68290352f58a0e193eed1d77
* [web] add consistent aria-label to editing/reviewing toggle
* [docstore] add endpoint for getting all docs with ranges
* [history-v1] fix schema of chunkId when deleting old history chunk
* [web] skip duplicate project lookup for resolving rootDocPath
* [web] ignore new limits for root doc path when making debug copy
* [web] allow admins to clone projects with ranges and entire history
* [web] fix tests
* [history-v1] re-order params for cloning project
* [web] fix duplicate import of logger after merge
* [project-history] re-order params for cloning project history metadata
GitOrigin-RevId: 7fa35b4f90885dd453150a348d491ba0ec8de412
* [web] serve site manifest as application/json
* [monorepo] bail out early from dev-env-setup when inside devcontainer
* [monorepo] move utils for waiting for services into shared bash script
* [monorepo] move domain related env-vars into dedicated .env file
The .env file will be overridden in GitHub Code Spaces.
* [monorepo] remove unused initial_mongo_setup Make target
It was previously used for the mongo replica set setup.
That setup is a docker init script now.
* [monorepo] add flag for silencing docker build output
* [monorepo] add integration for GitHub Code Spaces
* [web] make prettier happy
* [web] restore no use-credentials for manifest outside code spaces
* [monorepo] run make install as part of post-start again
GitOrigin-RevId: 8c2cc3b969b7e3e8f9c5dac778192c9412cbf4df
* [clsi] handle draft mode and tikzexternalize as part of sync phase
* [clsi] emit empty string from SafeReader on ENOENT
* [clsi] persist history state after clearing dirty state without changes
GitOrigin-RevId: d9dcd2e6887017f7935b5e95bdbdc6e11a3b18f5
* [clsi] remove locking from docker actions
Start:
- We have an in-memory lock on the compile request
Destroy:
- as part of run: see above
- as part of cleanup: we check the last access time now, so it cannot
happen concurrent to compiling anymore.
Co-authored-by: Anna Claire Fields <anna.fields@overleaf.com>
* [clsi] update comment
---------
Co-authored-by: Anna Claire Fields <anna.fields@overleaf.com>
GitOrigin-RevId: a58df45416ae31c0b38d5efec7f9371d747303df
* [monorepo] fix prettier formatting
* [monorepo] use a single prettier invocation for monorepo-check
GitOrigin-RevId: cf0cc2293b228be707ff23bdb3c5781c1f8d69f3
The req.body.rootResourcePath has been shipped three weeks ago, so it's
unlikely to trip up stale editor sessions.
For now, only block node-fetch and log the rest.
GitOrigin-RevId: 541189675f68fdcab09f4b409b4143024a29f94a
* [web] migrate compile of templates/example project to history mode
* [saas-e2e] give clsi-cache more time to process the jobs
* [web] add comment on in-memory project update
Co-authored-by: Brian Gough <briangough@users.noreply.github.com>
* [saas-e2e] add retries to clsi-cache check
* [saas-e2e] remove spurious retry trigger
* [saas-e2e] remove broken assertion
---------
Co-authored-by: Brian Gough <briangough@users.noreply.github.com>
GitOrigin-RevId: 8766662cbe0ab5520c5c79d86fa6f774e012b3da
* [web] gracefully handle broken histories when compiling from history
* [web] trim down schema
GitOrigin-RevId: 97d59b31eb25644d7de1194a45281def6982b130
* [clsi] avoid server error when clearing cache while compiling
* [clsi] tweak API around releasing locks
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
---------
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: d3f171467d3bc26941758dd333f30049b37a05c8
* [web] enable clsi-cache for all compiles for 20min during outages
* [web] populate clsi-cache when compile from history is enabled
* [web] hide history snapshot from frontend
GitOrigin-RevId: e46c8b5ab401ebdc2407e44eb95439e4d43400da
* [k8s] clsi-cache: double the number of shards
* [monorepo] add missing clsi-cache env vars to dev-env
* [clsi] flip direction of clsi-cache shard migration
* [clsi] remove upper bound from clsi-cache shard migration
GitOrigin-RevId: a325a11c3ac9e22a12ad2d8ea802b91d2e175e24
* [clsi] only download history snapshot from clsi-cache when enabled
* [clsi-perf] migrate to compile from history mode
GitOrigin-RevId: 2dd54e032bd85d6335488741c039a5a1bd60090d
Jakob Ackermann