* [agents] allow running all the test-runner skill commands
* [agents] move linting into AGENTS.md
* [agents] double down on permissions and cwd
* [saas-e2e] add support for MOCHA_GREP
* [saas-e2e] fail fast by default outside CI
* [agents] avoid need for Codespace detection
* [agents] work around missing trust prompt in Codespaces
* [saas-e2e] e2e_test_setup: merge split-tests in dev-env
GitOrigin-RevId: f1c1cd2a732e98b2e6125ef86a2b2cda5d3aabb6
* Add png to pdf converter for optimising image loading in pdftex
* refactor
* move fixtures
* refactor and add unit tests
* Remove gamma correction from PNG processing and update related tests
* Add crc32fast dependency and enhance PNG processing tests
* Add fixture test cases for PNG processing
* fix alpha channel to match pdftex
pdftex always uses an 8 bit alpha channel in the output even for 16 bit images
* Refactor PNG processing to use Option for alpha stream and update related logic
* Enhance PDF writing by adding DecodeParms for alpha stream and optimize PNG encoding process
* Add pixel dimensions support in PNG processing and PDF writing
* [copybara] move clsi excludes
* [clsi] move gitignore rules
* [png2pdf] move into dockerfiles folder
* [png2pdf] use atomic writes
* [clsi] integrate png2pdf
* [saas-e2e] add coverage for png2pdf
* [png2pdf] make compatible with seccomp profile in clsi
* [copybara] remove exclude following move of png2pdf
* [copybara] remove spurious excludes
* fix broken detection of malformed chunk headers
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* [project-history] add build script flag for depending on doc-updater types
* [png2pdf] run tests as part of clsi pipeline
* [png2pdf] fix calculation of PDF page size
* [web] put png2pdf behind feature flag and make it opt-in
* [web] fix ClsiManager.promises.sendRequest invocation after merge
* extend is_copy_safe function to follow pdftex
see the conditions in pdftex's writepng.c
https://svn.tug.org:8369/texlive/trunk/Build/source/texk/web2c/pdftexdir/writepng.c?revision=78081&view=markup#l575
the criteria are not an exact match because the png crate does not support all the
png chunk types
https://github.com/image-rs/image-png/issues/116
if the criteria result in a png being converted or skipped the final result should be the same.
in one case the conversion will be done by pdftex (at runtime)
in the other case the conversion will be done by png2pdf -
both conversions should give the same result.
* include tests in png2pdf Dockerfile
* [clsi] explicitly disable png2pdf conversion for project conversion
* [clsi] png2pdf: count converted files
* [png2pdf] bump date in docker image tag
* [web] make draft-setting option wider when png2pdf is available
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: d0d952ee009bf93c72bba2ffe763ae0160d396e4
* [web] add frontend tests for sharejs transform
* [web] extend tests for sharejs transform to flush eventually
* [web] full coverage for sharejs transform
* [web] add tests for sharejs events
* [web] add tests for sharejs callbacks
* [web] avoid sending empty sharejs op
* [web] add tests for resending after reconnecting
* [web] avoid resending of empty sharejs op after reconnecting
GitOrigin-RevId: 226fe47a7069cb877fd5a60454f86cc5855d9a0e
* [real-time] update tests for more realistic applyOtUpdate messages
* [saas-e2e] add tests for reviewer role
GitOrigin-RevId: f0a5307230beb00be3a1387129e06eb5b2106c09
* [web] use client-side filtering for split tests in admin panel
Filtering on non-indexed fields is not allowed with notablescan.
* [web] add initial value for DEFAULT_FILTERS.active
* [web] unset error before fetching split tests
GitOrigin-RevId: 0813873c5a90d57c4fdbfd0043fcc325f00408aa
* [migrations] only apply non-blocking migrations in CI
* [migrations] use "yarn run migrations" instead of "npm run migrations"
GitOrigin-RevId: d9f96db398adf2febcd82333295fb8d5eab741bb
* [monorepo] fix handling of quoted MOCHA_GREP
* [agents] refine guidance for running tests
- nudge it towards scoped module targets
- nudge it to really run e2e tests when changed
- nudge it towards a11y cypress matchers
- recommend quoting for MOCHA_GREP
* [web] fix forwarding of mocha args from module Makefile
* [monorepo] use BRANCH_NAME_TAG_SAFE for building the ci image name
* [web] also make BRANCH_NAME_TAG_SAFE available in module Makefile
(It's not really needed, but will make copilot happy.)
GitOrigin-RevId: bfda3c8fe2b4f857ef111ac63f94b35a82ad75f8
* [web] _getUserCompileLimits: use owner.analyticsId directly
* [web] deduplicate project lookups for compile requests
* [web] add test case for prefetched project
GitOrigin-RevId: 0e140619c874accf8673153ed90af2ce70cf9234
* [web] split-tests: add helpers to using an already fetched mongo user
* [web] split-tests: add ignoreOverrides option to getAssignment
* [web] split-tests: reuse already fetched mongo users where possible
* [web] notifications: fetch users in bulk for split-test assignment
* [web] split-tests: use session for assignment where possible
* [web] split-tests: pass req/res down into domain-capture hook
* [web] split-tests: explicitly throw early
* [web] split-tests: refine jsdoc types
* [web] mark analyticsId and labsProgram as required in user model
* [web] split-tests: refine jsdoc types
* Revert "[web] split-tests: pass req/res down into domain-capture hook"
This reverts commit 897a158384cd2b3e416fc34495b20d1e2a0e68f0.
GitOrigin-RevId: ea4afd15c2506f369827b5193fd16d220d3c7d0e
* [web] adopt free/premium compile backend class
* [rails] adopt free compile backend class for pdf conversion
GitOrigin-RevId: e12c99a27363f5654132c2fcdc6c5d8ccc591868
* [web] always provide an analyticsId when registering new users
* [web] add recordEventForMongoUser and use it during login
* [web] use recordEventForMongoUser/recordEventForSession where possible
* [web] use session cache for analyticsId when refreshing features
* [web] add setUserPropertyForMongoUser and use it where possible
* [web] use setUserPropertyForSessionInBackground where possible
* [web] use mongo user for analyticsId when setting utm user properties
* [web] admin-panel: user updated user when sending events
* [web] enforce labsProgram being populated on user object
Co-authored-by: Andrew Rumble <andrew.rumble@overleaf.com>
* [web] revert FeaturesUpdater.refreshFeatures change
* [web] labs: move session maintenance into handler
* [web] analyticsMiddleware: ensure that all users have labsProgram field
* [web] migrate a few more events to recordEventForSession
* [migrations] mark back-fill of db.users.analyticsId as required
* [web] setUserPropertyForSessionInBackground: throw early
---------
Co-authored-by: Andrew Rumble <andrew.rumble@overleaf.com>
GitOrigin-RevId: 19c26bc36abd8c11f0a8f458d25abfb7cbda0bb7
* [web] throw early in UserCreator.createNewUser with missing analyticsId
* [migrations] rerun back-fill of db.users.analyticsId
GitOrigin-RevId: eabf16326192fca4dbea00ad643cee4315c426c6
* [server-ce] test: the first startWith must resetData when using dataDir
* [server-ce] test: fix order of creating user and resetting DB
GitOrigin-RevId: 2a6e6344919fe48d41704e1a3cb39c6e22b5a924
* [web] add migration for back-filling db.users.labsProgram=false
Co-authored-by: Andrew Rumble <andrew.rumble@overleaf.com>
* [web] include user.labsProgram=false in session
---------
Co-authored-by: Andrew Rumble <andrew.rumble@overleaf.com>
GitOrigin-RevId: 0b3322007f6a36d2999d646869dc84f2c6ad4d2d
* [monorepo] provide clsi-sharded dev-env setup
* [monorepo] provide sharded clsi-nginx when using clsi-sharded
* [clsi-perf] run on all instances again
* [monorepo] switch default TL to 2025.1 in clsi (same as web)
* [monorepo] avoid starting base clsi when running clsi-perf with clsi-sharded
* [clsi-lb] use 3s check interval in clsi-sharded mode
GitOrigin-RevId: 3cae8f0c1db69dbd76b94b45dc45b3abb2fa0a83
* [clsi] forward instanceType to web
* [clsi] expose zone in compile response
* [clsi-perf] add new fields for instanceType and zone
* [clsi] expose isPreEmptible flag in compile response
* [clsi-perf] also add isPreEmptible field
* [clsi] rename isPreEmptible -> isSpotInstance
GitOrigin-RevId: 2bc2d772e0b14da83afb51f5f3e4fefd3f60b733
* [editor-core] avoid duplicate non-bmp check on InsertOp.insertion
* [editor-core] remove non-bmp check from StringFileData
Existing content may contain non-bmp characters. New edits with non-bmp
input are rejected via non-bmp check in InsertOp constructor.
GitOrigin-RevId: e9ef3a4a8c4696e8dc751325608a788351791d1c
This was useful when iterating on the pdf-caching feature.
PDF caching has been stable for years. We can remove the logging now.
GitOrigin-RevId: 826c91c1e45d3dae1bb3ce5aeb7b6a3f9283392c
* [web] gracefully handle notifications being down on project dashboard
* [monorepo] remove haproxy from service-to-service communication
Co-authored-by: Anna Claire Fields <acfields11@gmail.com>
* [monorepo] fix OVERLEAF_LATEXQC_API_BASE_URL in dev-env
* [SaaS test fix] update history v1 export base url
---------
Co-authored-by: Anna Claire Fields <acfields11@gmail.com>
GitOrigin-RevId: cf355ef061412a1b57b5383cb60d4be574055af5
* [web] determine rootResourcePath if not set in compile request
This affects the following compile requests:
- compile request from "Download PDF" button on project dashboard
Pre-resolving the rootResourcePath for all projects is too expensive.
Adding an endpoint to resolve it is too slow and doubles project auth
check.
- custom frontends
Can't do much about these. I tried blocking them, requests still come
in after weeks of returning 400s.
The rootDoc maintenance methods now return {rootResourcePath,rootDocId}
rather than just the rootDocId. This avoids an extra project lookup when
the rootDocId was updated.
GitOrigin-RevId: ab7c539b7ffd162bde0903fe1040a48834a245f5
* [monorepo] consolidate clsi-lb host/ip env-vars
Target env-var is CLSI_LB_HOST. Keep CLSI_LB_IP populated for a week.
* [clsi] initial version of /convert/pdf-to-jpeg
* [rails] use fake-secrets in CI and Codespaces
* [rails] adapt tests for using clsi to convert PDFs to image
* [rails] add rake task for comparing clsi conversion with transloadit
* [clsi] double check that output.jpg is a regular file
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [clsi] fix composing basename
* [monorepo] fix clsi-lb host env-var post merge
* [monorepo] sort dev-environment.env hosts
* [rails] use local pdf file rather than downloading it again
Download from the old renderer code path still. It's dead code.
* [terraform] clsi: enable pdf to jpg conversion
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: 5ecaa8559d299486340bb3961f06b29f7c4dfcca
* [clsi] add request flag for isCompileFromHistory
* [clsi] derive cacheKey for history snapshot from compile dir
* [clsi] migrate convert project to document to compile from history
* [clsi] address review feedback
* [web] determine root doc at the time of converting the project
* [web] wait for flush before starting document conversion
* [saas-e2e] add tests for root doc override when converting project
GitOrigin-RevId: 71c578030949b89f3a74e7f7ab882dfa9c98c17a
* [docstore] add useSecondary flag to projectHasRanges
The rev-check for unarchiving always consults with the primary.
Two extra changes:
- Add a projection argument to peekDoc in order to skip lines download
from projectHasRanges.
- Add one retry to peekDoc to reduce chances of surfacing a rev-check
violation.
* [web] resync_projects: use the secondaries for all reads
* [web] add default value for useSecondary
* [docstore] add default value for useSecondary
* [k8s] docstore: set MONGO_HAS_SECONDARIES=true
GitOrigin-RevId: f15ec4fdc1cabe74c1eab87bec85f28d6f7a587d
* [history-v1] add endpoint for downloading latest zip
* [web] address review feedback
* [web] tests: do not overwrite db.projects.overleaf, extend it
* [web] set includeReferer flag from downloading zip
GitOrigin-RevId: e63e549f004230086f82eccf03b43fd62bde6071
* [web] cleanup archived split-test assignments from user record on login
Co-authored-by: Anna Claire Fields <anna.fields@overleaf.com>
* [migrations] purge archived split tests from all users
Co-authored-by: Anna Claire Fields <anna.fields@overleaf.com>
* [web] add missing mock and update snapshot test
* [web] gracefully access db.users.splitTests
---------
Co-authored-by: Anna Claire Fields <anna.fields@overleaf.com>
GitOrigin-RevId: bd185074a402556d7b7c812208cf834dd52b27a5
* [web] add includeReferer flag to SplitTestHandler.getAssignment
* [web] tests: migrate User.getSplitTestAssignment to async/await
I don't want to fight with callbacks and optional arguments. Just move
it to async/await. New tests should use async/await, so there is no
point in making this work in callback-hell.
* [web] remove unused URL import
GitOrigin-RevId: 6251001e6ba7354f704fa663be8ef365ca0b9d23
* [monorepo] avoid corepack network requests
- Download yarn via corepack as the first step in all the docker files
- Turn off networking in corepack
- Do not run things in the upstream node image
Instead, use the monorepo image, or base layer in all the services.
- Always build the base layer when running tests (uses cache)
* [monorepo] install corepack in shared place
* [clsi-lb] remove unrelated changes
* [web] add missing DC_RUN_FLAGS
* [monorepo] only rebuild test images locally
Also remove spurious build config in docker-compose.ci.yml.
* [server-ce] test: make yarn files available to host-admin and e2e
* [monorepo] put the corepack install snippet in a few more places
GitOrigin-RevId: 38005016ae5a708e12295e246269d6c18fece937
* [web] make double compile test parameters configurable via env vars
* [k8s] web: enable double compile test for free compiles on n4 instances
GitOrigin-RevId: 3a5cb8ed6d044fcf3f4c0d2b9d252326bac48511
* [web] check for missing module tsconfig in CI
* [web] add missing tsconfig into test_unit container
GitOrigin-RevId: 7b861f8e68f961e3455b72b5660cf3825389c656
* [history-v1] fix re-running test_acceptance_run
Preserve the contents of the migrations collection. Deleting the entries
does not "undo" the migrations. On re-run east would try applying all
migrations and fail as things were already applied.
* [history-v1] remove guard migration before running tests
GitOrigin-RevId: e6eeafd58215e5148dd70c37c7a87d84e0a12bf3
* [github] code spaces: add extra-split-tests.json
* [github] address review feedback
* [github] add missing bind mount
* [monorepo] mount split test configs with :ro
* [monorepo] only mount backup/split-tests into web
GitOrigin-RevId: 6334a44599a2c8bb79bf1ad698e656c8bee3992b
* [metrics] mongo: fail when command monitoring is not available
* [metrics] mongo: add optional client label to pool metrics
* [web] collect mongo stats on native client
* [metrics] mongo: record namespace of find commands
* [metrics] mongo: add counter for all the commands with collection label
* [web] add missing mock
GitOrigin-RevId: 9f378d8aa8d7167f56cf512681d63ef115c6dd98
* [monorepo] move building of nginx proxy into Makefile
* [v1] add fake-secrets.env
* [github] add copilot-setup-steps.yml workflow
* [make] debug permissions
* [monorepo] use the host uid/gid for monorepo service
* [web] populate DOCKER_USER for test_frontend_ct
* [github] run npm install outside docker in Copilot sandbox
* [monorepo] add comment for building nginx-proxy image locally
* [monorepo] change uid/gid of node user in cypress image
* [monorepo] low-level rewrite of passwd and groups to match host user
GitOrigin-RevId: d74a5801f0318bab7e7f460374255426706002a3
* [monorepo] run npm install before top-level lint/format
* [monorepo] extend lint_fix for web
* [monorepo] bin/run: silence docker compose
We do not need it to print the newly created containers name.
* [monorepo] run format:fix
* [monorepo] run all the tasks or services/web/lint_fix_other in docker
GitOrigin-RevId: 997b360d1136f466c8e6bf8c5ad0e1189406d1f1
* [web] enable async local storage on all the endpoints
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [web] deduplicate getting the project when loading the editor
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [web] use ProjectAccess state for computing analytics segmentation
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [web] restore ownership of active flag and deferredTpdsFlushCounter
* [web] add missing await
* [web] update unit tests
* [web] add metrics for project access caching
* [web] add missing test mock
* [web] invalidate async local storage when changing project access
* [web] deduplicate project lookup when checking for token access
* [web] add helper function for getting cached ProjectAccess
* [web] add acceptance test for caching of ProjectAccess
* [web] account for saas-only project access in tests
* Revert "[web] enable async local storage on all the endpoints"
This reverts commit 1b82f3b935040e8cfd180d1f6bf4183a655580e2.
* [web] add async local storage to project endpoints in top-50
* [web] invalidate async local storage for project access from modules
Co-authored-by: Jessica Lawshe <jessica.lawshe@overleaf.com>
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Jessica Lawshe <jessica.lawshe@overleaf.com>
GitOrigin-RevId: 3eea7956b24e6f937dc1c17948681063d4dca3ea
* [project-history] add best effort flush when cloning project
* [web] update labels in clone project modal for admins
* [project-history] do not shadow history flush failure
* [web] fix accessible label for 'Add comment' button
* [chat] clone comment threads when cloning project with ranges
GitOrigin-RevId: ef30204c8a94b3d6204d56dcca2f62a46319996b
* [github] code spaces: add more standard users with a common password
* [github] code spaces: automatically share dev-env with Overleaf org
* [github] code spaces: merge split tests rather than replacing them
* [github] code spaces: ts-check the setup script in web
GitOrigin-RevId: 1a86ac7e0304d47e68290352f58a0e193eed1d77
* [web] add consistent aria-label to editing/reviewing toggle
* [docstore] add endpoint for getting all docs with ranges
* [history-v1] fix schema of chunkId when deleting old history chunk
* [web] skip duplicate project lookup for resolving rootDocPath
* [web] ignore new limits for root doc path when making debug copy
* [web] allow admins to clone projects with ranges and entire history
* [web] fix tests
* [history-v1] re-order params for cloning project
* [web] fix duplicate import of logger after merge
* [project-history] re-order params for cloning project history metadata
GitOrigin-RevId: 7fa35b4f90885dd453150a348d491ba0ec8de412
* [web] serve site manifest as application/json
* [monorepo] bail out early from dev-env-setup when inside devcontainer
* [monorepo] move utils for waiting for services into shared bash script
* [monorepo] move domain related env-vars into dedicated .env file
The .env file will be overridden in GitHub Code Spaces.
* [monorepo] remove unused initial_mongo_setup Make target
It was previously used for the mongo replica set setup.
That setup is a docker init script now.
* [monorepo] add flag for silencing docker build output
* [monorepo] add integration for GitHub Code Spaces
* [web] make prettier happy
* [web] restore no use-credentials for manifest outside code spaces
* [monorepo] run make install as part of post-start again
GitOrigin-RevId: 8c2cc3b969b7e3e8f9c5dac778192c9412cbf4df
* [clsi] handle draft mode and tikzexternalize as part of sync phase
* [clsi] emit empty string from SafeReader on ENOENT
* [clsi] persist history state after clearing dirty state without changes
GitOrigin-RevId: d9dcd2e6887017f7935b5e95bdbdc6e11a3b18f5
* [clsi] remove locking from docker actions
Start:
- We have an in-memory lock on the compile request
Destroy:
- as part of run: see above
- as part of cleanup: we check the last access time now, so it cannot
happen concurrent to compiling anymore.
Co-authored-by: Anna Claire Fields <anna.fields@overleaf.com>
* [clsi] update comment
---------
Co-authored-by: Anna Claire Fields <anna.fields@overleaf.com>
GitOrigin-RevId: a58df45416ae31c0b38d5efec7f9371d747303df
* [monorepo] fix prettier formatting
* [monorepo] use a single prettier invocation for monorepo-check
GitOrigin-RevId: cf0cc2293b228be707ff23bdb3c5781c1f8d69f3
The req.body.rootResourcePath has been shipped three weeks ago, so it's
unlikely to trip up stale editor sessions.
For now, only block node-fetch and log the rest.
GitOrigin-RevId: 541189675f68fdcab09f4b409b4143024a29f94a
* [web] migrate compile of templates/example project to history mode
* [saas-e2e] give clsi-cache more time to process the jobs
* [web] add comment on in-memory project update
Co-authored-by: Brian Gough <briangough@users.noreply.github.com>
* [saas-e2e] add retries to clsi-cache check
* [saas-e2e] remove spurious retry trigger
* [saas-e2e] remove broken assertion
---------
Co-authored-by: Brian Gough <briangough@users.noreply.github.com>
GitOrigin-RevId: 8766662cbe0ab5520c5c79d86fa6f774e012b3da
Jakob Ackermann