* [document-updater] adopt strict zod types and extend test coverage
* [saas-e2e] add test for project restore with binary and linked file
GitOrigin-RevId: a8e45bd1e46b6ac8a293da24c8b8ad7b236fd9bd
* [notifications] adopt strict zod types and extend test coverage
* [notifications] validate expires date via schema
GitOrigin-RevId: 25dee0578ec443447e4c38a50a7e6c8b44831fe7
- Trigger monorepo-check on Jenkins init scripts and fix violations
- Ignore all the json-dashboards in terraform folders
GitOrigin-RevId: f27b46d6fb81f5ac731f56b065951bede7d1a9e4
* [saas-e2e] run all the spec files
* [saas-e2e] add addComment helper
* [saas-e2e] add replyToComment helper
* [saas-e2e] check latexqc report for errors
* [saas-e2e] fail fast when the login request failed
* [monorepo] upsert the minimal v1 db entries in all environments
* [monorepo] serve the CDN assets from nginx to avoid session mix
* [monorepo] avoid loading fonts from Cypress UI via web
* [monorepo] effectively disable login + loadEditor rate-limit in dev-env
* [saas-e2e] project-list: generate unique project name on re-run
* [saas-e2e] add debug logs for latexqc
GitOrigin-RevId: 778003759414a3c993578062a631d083ecb1d2e6
* [editor-core] extend coverage of zod schemas
* [real-time] adopt strict zod types and extend test coverage
* [project-history] make type check happy
* [web] make type check happy
GitOrigin-RevId: e1a65b9b3560b31ad2af817067a4b549865a5bb1
tid does not exist anywhere else in document-updater at
sha 3d76f4b9bf when
the typo was added. The comment id was read from .t only.
Doc-snapshots are copied the ranges literally without any
normalizing of nested structures.
GitOrigin-RevId: f76fbc930fdf02d9367603b5b1d6414de8d9f559
* [saas-e2e] conversion: wait for initial compile before looking at PDF
* [eslint-plugin] s/before/beforeWithReRunOnTestRetry/ in e2e tests
* [server-ce] remove filestore migration scripts and E2E tests
* [server-ce] test: trim down upgrading tests to 5.x -> latest
GitOrigin-RevId: 2426ec4a4f3d67a5e0e7850ea84a942ae4619f94
With the removal of per-doc rooms, there is nothing to race anymore.
The next lower timeout is 0, i.e. useless check.
GitOrigin-RevId: effb6cc0525f5fd785c7dce8df4e279de1f1b13f
* [web] store ObjectIds in addUserToProject acceptance test helper
* [web] optimize project lookup for tpds operations
GitOrigin-RevId: 03db9fc450556072f177d56f640fcc6114b295d4
* [web] add endpoint for resolving a project for writes from tpds
* [web] return historyId from new endpoint for resolving projects
* [web] add zod validation to new endpoint for resolving projects
GitOrigin-RevId: 0e607e56442117a4150a08373169c44b53dd4fdf
* [agents] allow running all the test-runner skill commands
* [agents] move linting into AGENTS.md
* [agents] double down on permissions and cwd
* [saas-e2e] add support for MOCHA_GREP
* [saas-e2e] fail fast by default outside CI
* [agents] avoid need for Codespace detection
* [agents] work around missing trust prompt in Codespaces
* [saas-e2e] e2e_test_setup: merge split-tests in dev-env
GitOrigin-RevId: f1c1cd2a732e98b2e6125ef86a2b2cda5d3aabb6
* Add png to pdf converter for optimising image loading in pdftex
* refactor
* move fixtures
* refactor and add unit tests
* Remove gamma correction from PNG processing and update related tests
* Add crc32fast dependency and enhance PNG processing tests
* Add fixture test cases for PNG processing
* fix alpha channel to match pdftex
pdftex always uses an 8 bit alpha channel in the output even for 16 bit images
* Refactor PNG processing to use Option for alpha stream and update related logic
* Enhance PDF writing by adding DecodeParms for alpha stream and optimize PNG encoding process
* Add pixel dimensions support in PNG processing and PDF writing
* [copybara] move clsi excludes
* [clsi] move gitignore rules
* [png2pdf] move into dockerfiles folder
* [png2pdf] use atomic writes
* [clsi] integrate png2pdf
* [saas-e2e] add coverage for png2pdf
* [png2pdf] make compatible with seccomp profile in clsi
* [copybara] remove exclude following move of png2pdf
* [copybara] remove spurious excludes
* fix broken detection of malformed chunk headers
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* [project-history] add build script flag for depending on doc-updater types
* [png2pdf] run tests as part of clsi pipeline
* [png2pdf] fix calculation of PDF page size
* [web] put png2pdf behind feature flag and make it opt-in
* [web] fix ClsiManager.promises.sendRequest invocation after merge
* extend is_copy_safe function to follow pdftex
see the conditions in pdftex's writepng.c
https://svn.tug.org:8369/texlive/trunk/Build/source/texk/web2c/pdftexdir/writepng.c?revision=78081&view=markup#l575
the criteria are not an exact match because the png crate does not support all the
png chunk types
https://github.com/image-rs/image-png/issues/116
if the criteria result in a png being converted or skipped the final result should be the same.
in one case the conversion will be done by pdftex (at runtime)
in the other case the conversion will be done by png2pdf -
both conversions should give the same result.
* include tests in png2pdf Dockerfile
* [clsi] explicitly disable png2pdf conversion for project conversion
* [clsi] png2pdf: count converted files
* [png2pdf] bump date in docker image tag
* [web] make draft-setting option wider when png2pdf is available
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: d0d952ee009bf93c72bba2ffe763ae0160d396e4
* [web] add frontend tests for sharejs transform
* [web] extend tests for sharejs transform to flush eventually
* [web] full coverage for sharejs transform
* [web] add tests for sharejs events
* [web] add tests for sharejs callbacks
* [web] avoid sending empty sharejs op
* [web] add tests for resending after reconnecting
* [web] avoid resending of empty sharejs op after reconnecting
GitOrigin-RevId: 226fe47a7069cb877fd5a60454f86cc5855d9a0e
* [real-time] update tests for more realistic applyOtUpdate messages
* [saas-e2e] add tests for reviewer role
GitOrigin-RevId: f0a5307230beb00be3a1387129e06eb5b2106c09
* [web] use client-side filtering for split tests in admin panel
Filtering on non-indexed fields is not allowed with notablescan.
* [web] add initial value for DEFAULT_FILTERS.active
* [web] unset error before fetching split tests
GitOrigin-RevId: 0813873c5a90d57c4fdbfd0043fcc325f00408aa
* [migrations] only apply non-blocking migrations in CI
* [migrations] use "yarn run migrations" instead of "npm run migrations"
GitOrigin-RevId: d9f96db398adf2febcd82333295fb8d5eab741bb
* [monorepo] fix handling of quoted MOCHA_GREP
* [agents] refine guidance for running tests
- nudge it towards scoped module targets
- nudge it to really run e2e tests when changed
- nudge it towards a11y cypress matchers
- recommend quoting for MOCHA_GREP
* [web] fix forwarding of mocha args from module Makefile
* [monorepo] use BRANCH_NAME_TAG_SAFE for building the ci image name
* [web] also make BRANCH_NAME_TAG_SAFE available in module Makefile
(It's not really needed, but will make copilot happy.)
GitOrigin-RevId: bfda3c8fe2b4f857ef111ac63f94b35a82ad75f8
* [web] _getUserCompileLimits: use owner.analyticsId directly
* [web] deduplicate project lookups for compile requests
* [web] add test case for prefetched project
GitOrigin-RevId: 0e140619c874accf8673153ed90af2ce70cf9234
* [web] split-tests: add helpers to using an already fetched mongo user
* [web] split-tests: add ignoreOverrides option to getAssignment
* [web] split-tests: reuse already fetched mongo users where possible
* [web] notifications: fetch users in bulk for split-test assignment
* [web] split-tests: use session for assignment where possible
* [web] split-tests: pass req/res down into domain-capture hook
* [web] split-tests: explicitly throw early
* [web] split-tests: refine jsdoc types
* [web] mark analyticsId and labsProgram as required in user model
* [web] split-tests: refine jsdoc types
* Revert "[web] split-tests: pass req/res down into domain-capture hook"
This reverts commit 897a158384cd2b3e416fc34495b20d1e2a0e68f0.
GitOrigin-RevId: ea4afd15c2506f369827b5193fd16d220d3c7d0e
* [web] adopt free/premium compile backend class
* [rails] adopt free compile backend class for pdf conversion
GitOrigin-RevId: e12c99a27363f5654132c2fcdc6c5d8ccc591868
* [web] always provide an analyticsId when registering new users
* [web] add recordEventForMongoUser and use it during login
* [web] use recordEventForMongoUser/recordEventForSession where possible
* [web] use session cache for analyticsId when refreshing features
* [web] add setUserPropertyForMongoUser and use it where possible
* [web] use setUserPropertyForSessionInBackground where possible
* [web] use mongo user for analyticsId when setting utm user properties
* [web] admin-panel: user updated user when sending events
* [web] enforce labsProgram being populated on user object
Co-authored-by: Andrew Rumble <andrew.rumble@overleaf.com>
* [web] revert FeaturesUpdater.refreshFeatures change
* [web] labs: move session maintenance into handler
* [web] analyticsMiddleware: ensure that all users have labsProgram field
* [web] migrate a few more events to recordEventForSession
* [migrations] mark back-fill of db.users.analyticsId as required
* [web] setUserPropertyForSessionInBackground: throw early
---------
Co-authored-by: Andrew Rumble <andrew.rumble@overleaf.com>
GitOrigin-RevId: 19c26bc36abd8c11f0a8f458d25abfb7cbda0bb7
* [web] throw early in UserCreator.createNewUser with missing analyticsId
* [migrations] rerun back-fill of db.users.analyticsId
GitOrigin-RevId: eabf16326192fca4dbea00ad643cee4315c426c6
* [server-ce] test: the first startWith must resetData when using dataDir
* [server-ce] test: fix order of creating user and resetting DB
GitOrigin-RevId: 2a6e6344919fe48d41704e1a3cb39c6e22b5a924
* [web] add migration for back-filling db.users.labsProgram=false
Co-authored-by: Andrew Rumble <andrew.rumble@overleaf.com>
* [web] include user.labsProgram=false in session
---------
Co-authored-by: Andrew Rumble <andrew.rumble@overleaf.com>
GitOrigin-RevId: 0b3322007f6a36d2999d646869dc84f2c6ad4d2d
* [monorepo] provide clsi-sharded dev-env setup
* [monorepo] provide sharded clsi-nginx when using clsi-sharded
* [clsi-perf] run on all instances again
* [monorepo] switch default TL to 2025.1 in clsi (same as web)
* [monorepo] avoid starting base clsi when running clsi-perf with clsi-sharded
* [clsi-lb] use 3s check interval in clsi-sharded mode
GitOrigin-RevId: 3cae8f0c1db69dbd76b94b45dc45b3abb2fa0a83
* [clsi] forward instanceType to web
* [clsi] expose zone in compile response
* [clsi-perf] add new fields for instanceType and zone
* [clsi] expose isPreEmptible flag in compile response
* [clsi-perf] also add isPreEmptible field
* [clsi] rename isPreEmptible -> isSpotInstance
GitOrigin-RevId: 2bc2d772e0b14da83afb51f5f3e4fefd3f60b733
* [editor-core] avoid duplicate non-bmp check on InsertOp.insertion
* [editor-core] remove non-bmp check from StringFileData
Existing content may contain non-bmp characters. New edits with non-bmp
input are rejected via non-bmp check in InsertOp constructor.
GitOrigin-RevId: e9ef3a4a8c4696e8dc751325608a788351791d1c
This was useful when iterating on the pdf-caching feature.
PDF caching has been stable for years. We can remove the logging now.
GitOrigin-RevId: 826c91c1e45d3dae1bb3ce5aeb7b6a3f9283392c
* [web] gracefully handle notifications being down on project dashboard
* [monorepo] remove haproxy from service-to-service communication
Co-authored-by: Anna Claire Fields <acfields11@gmail.com>
* [monorepo] fix OVERLEAF_LATEXQC_API_BASE_URL in dev-env
* [SaaS test fix] update history v1 export base url
---------
Co-authored-by: Anna Claire Fields <acfields11@gmail.com>
GitOrigin-RevId: cf355ef061412a1b57b5383cb60d4be574055af5
* [web] determine rootResourcePath if not set in compile request
This affects the following compile requests:
- compile request from "Download PDF" button on project dashboard
Pre-resolving the rootResourcePath for all projects is too expensive.
Adding an endpoint to resolve it is too slow and doubles project auth
check.
- custom frontends
Can't do much about these. I tried blocking them, requests still come
in after weeks of returning 400s.
The rootDoc maintenance methods now return {rootResourcePath,rootDocId}
rather than just the rootDocId. This avoids an extra project lookup when
the rootDocId was updated.
GitOrigin-RevId: ab7c539b7ffd162bde0903fe1040a48834a245f5
* [monorepo] consolidate clsi-lb host/ip env-vars
Target env-var is CLSI_LB_HOST. Keep CLSI_LB_IP populated for a week.
* [clsi] initial version of /convert/pdf-to-jpeg
* [rails] use fake-secrets in CI and Codespaces
* [rails] adapt tests for using clsi to convert PDFs to image
* [rails] add rake task for comparing clsi conversion with transloadit
* [clsi] double check that output.jpg is a regular file
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [clsi] fix composing basename
* [monorepo] fix clsi-lb host env-var post merge
* [monorepo] sort dev-environment.env hosts
* [rails] use local pdf file rather than downloading it again
Download from the old renderer code path still. It's dead code.
* [terraform] clsi: enable pdf to jpg conversion
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: 5ecaa8559d299486340bb3961f06b29f7c4dfcca
* [clsi] add request flag for isCompileFromHistory
* [clsi] derive cacheKey for history snapshot from compile dir
* [clsi] migrate convert project to document to compile from history
* [clsi] address review feedback
* [web] determine root doc at the time of converting the project
* [web] wait for flush before starting document conversion
* [saas-e2e] add tests for root doc override when converting project
GitOrigin-RevId: 71c578030949b89f3a74e7f7ab882dfa9c98c17a
* [docstore] add useSecondary flag to projectHasRanges
The rev-check for unarchiving always consults with the primary.
Two extra changes:
- Add a projection argument to peekDoc in order to skip lines download
from projectHasRanges.
- Add one retry to peekDoc to reduce chances of surfacing a rev-check
violation.
* [web] resync_projects: use the secondaries for all reads
* [web] add default value for useSecondary
* [docstore] add default value for useSecondary
* [k8s] docstore: set MONGO_HAS_SECONDARIES=true
GitOrigin-RevId: f15ec4fdc1cabe74c1eab87bec85f28d6f7a587d
Jakob Ackermann