Commit Graph

193 Commits

Author SHA1 Message Date
Jimmy Domagala-Tang
994932b8e3 [Web + Doc-Updater] Add track changes accepted notification (#32752)
* feat: update doc manager to return a list of contributors to the accepted change

* feat: add new notification type for accepting a tracked change

* update email with tracked changes accepted

* feat: update tests

* fix: feedback on consistent api and returns

* feat: adding new tests

* feat: self accepted changes shouldnt trigger notification, and using existing changesAccepted hook

* Add better subject and activity list for track change accepted (#33094)

* feat: add better activity list entry and subject header for accepted changes, to match other notifications

* feat: updating tests

* feat: updating accepting_user_id to just user_id

* fix: adding users in emailBuilder test to userCache

GitOrigin-RevId: 6114f77916b5f503b7bbbb5ca8fed99e58edc31b
2026-04-30 08:06:19 +00:00
Brian Gough
15a24db3c9 Merge pull request #31650 from overleaf/bg-handle-timeouts-in-persistence-manager
add missing handling of timeouts in PersistenceManager

GitOrigin-RevId: 7fe74068f3ea647b27103393c3f0b243b4b25fe3
2026-02-20 09:05:50 +00:00
Brian Gough
a9c94c4184 Merge pull request #31444 from overleaf/bg-jpa-use-fetch-in-persistence-manager
remove requestretry from PersistenceManager

GitOrigin-RevId: 1fc9ffdfa7879d7ab4f0f4683544d09fe8526f3d
2026-02-19 09:05:45 +00:00
Jakob Ackermann
7c70b749d4 [monorepo] remove PII and variables from error messages (#31508)
* [monorepo] remove PII and variables from error messages

Exclusions:
- scripts
- tests
- fuzzing
- SplitTestManager (messages are sent to admin frontend)
- Group setup (we may want an error per unique tuple)
- sharejs (unused types; text type errors are shadowed already)
- history-v1 error messages that are used by the ErrorRecorder
- errors that flag issues with configuration/call signatures

I've used these search terms for finding unwanted error messages:
- new Error(`
- new Error\(\n\s+` (regex search)
- new OError(`
- new OError\(\n\s+` (regex search)

* [web] throw NotFoundError from ProjectLocator

* [github-sync] fix OError.tag call in script

Co-authored-by: Jessica Lawshe <jessica.lawshe@overleaf.com>

* [templates] revert changes to test client

---------

Co-authored-by: Jessica Lawshe <jessica.lawshe@overleaf.com>
GitOrigin-RevId: 736857a4fc5d9bfb0f8cb03e0f004eda87e5a220
2026-02-17 09:05:04 +00:00
Rebeka Dekany
a648015db8 Centralize prettier configuration to root level (#30501)
* Merge all .prettierignore files into top-level config

* Merge all .prettierrc files into top-level config

* Replace service-specific glob patterns in package.json format scripts with `prettier .`

* Add template files with Jinja2, Go template, envsubst, and Handlebars syntax to .prettierignore

* Ignore GitHub templates

* Ignore PUG templates to format them separately with `format:pug`

* Encourage double quotes for YAML, YML files

* Move prettier for PUG source format script to the root

* Move prettier for styles source format script to the root

* Remove prettier for jenkins files from web

* Remove prettier source format script from all services

* Make .prettierrc more readable

* Update format scripts by file type

* Organise `.prettierignore`

* Add `--cache` flag to prettier scripts for faster runs

* Format all files with prettier

* Format all or format services

* Remove `format`/`format:fix` scripts from services since now it runs from root `package.json`

* Avoid conlficts with yamllint configuration

* Remove `--cache` flag from prettier scripts

* Update all service Makefiles to use root-level prettier configuration

* Update all Jenkinsfile to use root-level prettier configuration

* Ignore auto-generated files by build_scripts

* Update package-lock.json

* Update root Makefile format targets

* Update SP Jenkinsfile format target

* Update E2E Makefile format script

* Udpate `format_js` to work in both local and CI env

* Add docker-mailtrap to .prettierignore

docker-mailtrap is a third-party git-ignored directory used for testing

* Added Docker env detection to prevent nested Docker spawning

* Ignore handlebars templates

* Add cryptographic files and test output to `.prettierignore`

* Add terraform modules to `.gitignore`

* Remove prettier-plugin-groovy

* Use npx directly instead of Docker for local formatting for faster formatting

* Auto-generate Makefiles

* Revert "Remove prettier-plugin-groovy"

This reverts commit 194a33589a2e1e4d2225d10c67e9f025e4222025.

* Mount monorepo root in RUN_LINT_FORMAT for prettier config access

* Prettier ignores all `node_modules` by default regardless of location

* Show only changed files in format output

* Ignore LICENSE files

* Enable prettier on rendered build_scripts outputs

* Ignoring all the template folders by prettier

* Remove the public/minjs entry since it does not exist

* Remove all non-existent paths

* Sync `.prettierignore` with ignored files by `.gitignore` and `.dockerignore` files

* Revert "Auto-generate Makefiles"

This reverts commit c0233e490de1bc95fe437219d65e0b66e0331ec9.

* Revert "Use npx directly instead of Docker for local formatting for faster formatting"

This reverts commit 1d2b2cf1a6c6974c76885852a90dd55e84167e41.

* Ignore dashboard JSON files

* Ignore files generated by bin/update_build_scripts

* Remove unsupported file types from `.prettierignore`

* Ignore test fixture generated files

* Ignore README file types by prettier

* Ignore generate snapshots by prettier

* Allow to format generated bin/update_build_scripts by prettier

* Ensure build script outputs prettier-compatible tsconfig.json

* Fix build script output to match prettier formatting
- Fix Jinja2 whitespace in docker-compose templates
- Change YAML quotes from single to double

* Don't read cryptographic files by prettier

* Ignore google verification files by prettier

* Revert npx prettier formatting

* Ignore domain verification files

* Show only changed files in format output

* Make `.github` prettier

* Allow all files to be formatted in jobs by prettier

* Allow server-ce/server-pro files to be formatted by prettier

* Ignore more folders in clsi, filestory, git-bridge by prettier

* Update build script with `RUN_LINTING_CI_MONOREPO`

* Ignore docker-mailtrap and downloads in server-ce by prettier

* Restore prettier configs and prettierignore for V1 since it has its own prettier (an older version)

* Source format

GitOrigin-RevId: 637adc3cc422d1f20c86d6ebc8ec514d60758287
2026-02-04 09:08:22 +00:00
Jakob Ackermann
86b29819c2 [document-updater] add missing await (#31034)
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: f0602801ce80ca617027004ad36f71c1e6cdcdd2
2026-01-27 09:06:59 +00:00
Andrew Rumble
2cdbdc1291 Merge pull request #30785 from overleaf/ar-add-logs-for-no-projectHistoryId-from-web
[document-updater] add log for missing projectHistoryId from web

GitOrigin-RevId: 750150312e341ef09b6c82b9df467900beb4b784
2026-01-15 14:19:37 +00:00
Andrew Rumble
a071cd74bc Merge pull request #30784 from overleaf/revert-30675-ar-jpa-log-missing-history-id-docupdater
Revert "[document-updater] add additional logging when doc is missing projectHistoryId"

GitOrigin-RevId: 725508a8f88ad3354b8b4644f4ee1e91efbecde2
2026-01-15 14:19:24 +00:00
Andrew Rumble
258c6224f4 Merge pull request #30675 from overleaf/ar-jpa-log-missing-history-id-docupdater
[document-updater] add additional logging when doc is missing projectHistoryId

GitOrigin-RevId: fba311cb10e0eeeb0a8165ce4e519c7704a7458f
2026-01-15 14:19:11 +00:00
Domagoj Kriskovic
0a84cefc96 Revert "Record project notification timestamp in Redis on applyUpdate (#29509)"
This reverts commit faa7a97fa929941cade5d62d2d680c6c3f34cdc8.

GitOrigin-RevId: 31c88ee836fb5e1ab3950da590c28e24b1397edb
2026-01-12 09:06:50 +00:00
Domagoj Kriskovic
8bca2545ee Record project notification timestamp in Redis on applyUpdate (#29509)
* Record project notification timestamp in Redis on applyUpdate

* Refactor project notification timestamp recording in Redis

* Remove tests which are catching errors that are no longer used

GitOrigin-RevId: faa7a97fa929941cade5d62d2d680c6c3f34cdc8
2026-01-12 09:06:07 +00:00
Brian Gough
a45c39389c Merge pull request #29891 from overleaf/bg-docupdater-make-max-unflushed-age-configurable
Make document-updater max unflushed age configurable via settings

GitOrigin-RevId: 3421f2467b7d8459ba8cdb1859e1b0695b0a01f3
2025-12-08 09:05:08 +00:00
Domagoj Kriskovic
c84cfc815a Document updater script for notifying web on project updates (#29490)
* Document updater script for notifying web on project updates

* use lua script for deleting keys

* define jobId when adding to queue

* removeOnFail age

* mongo check if collaborator exists

GitOrigin-RevId: f67a79c890a03ccf44fd84243c9a3f380e7afb43
2025-11-11 09:06:04 +00:00
Jakob Ackermann
d648c96603 [document-updater] migrate HistoryManager to async/await (#28789)
shouldFlushHistoryOps has a default value for 'threshold', which keeps
the exports simpler and still lets the unit tests override it.

GitOrigin-RevId: 1c6d4a2778052b5af40e2e338589a230ac2f4646
2025-10-09 08:07:36 +00:00
Jakob Ackermann
b0b9733a42 [document-updater] migrate ProjectFlusher to async/await (#28796)
GitOrigin-RevId: 24f61d6c0fab5d65b962cc7031ce0b8c84d5a915
2025-10-09 08:07:31 +00:00
Tim Down
c104aa454e Merge pull request #28845 from overleaf/td-async-await-doc-updater-client
Convert DocUpdateClient in document-updater acceptance tests to async/await

GitOrigin-RevId: 8f2352119f8f1175c2703ed90dbbc483ed039e86
2025-10-08 08:05:26 +00:00
Eric Mc Sween
543f3fafcc Merge pull request #28431 from overleaf/em-docupdater-get-ranges
Look in docupdater when obtaining comments for DSMP

GitOrigin-RevId: acaead280f725da8e0c438d5b429fa983c3cad86
2025-09-15 08:05:29 +00:00
Eric Mc Sween
97b857ceb8 Merge pull request #28311 from overleaf/em-promisify-redis-manager
Promisify RedisManager in document-updater

GitOrigin-RevId: 87c86d3b8904e17e76756789d2ea3e09fab26d02
2025-09-10 08:05:57 +00:00
Eric Mc Sween
2995b29829 Merge pull request #28205 from overleaf/em-promisify-project-manager
Promisify ProjectManager

GitOrigin-RevId: d10807a9a7f11428dd6bfe41210bfd0ce32c37ba
2025-09-03 08:06:42 +00:00
Eric Mc Sween
ca9455d7f2 Merge pull request #28182 from overleaf/em-promisify-http-controller
Promisify HttpController in document-updater

GitOrigin-RevId: fb87a38be856d64781401f7391b7f2a2c35c89fa
2025-09-01 08:05:12 +00:00
Eric Mc Sween
8f0913fafe Merge pull request #27936 from overleaf/em-unit-tests-mongo
Make Mongo available to unit tests in all services

GitOrigin-RevId: b65bbb69883d5bba31d09802b89f35bdc523fe4d
2025-08-25 08:05:19 +00:00
Domagoj Kriskovic
704e577c23 Add endpoint to retrieve document with history ranges and use it in dsmp API (#27564)
GitOrigin-RevId: 3d2ac33cdc903a07b8ec67f7fb6f723ae9c81a26
2025-08-15 08:05:17 +00:00
Domagoj Kriskovic
49bcfdd575 [ds-mobile-app] Implement reject changes in document updater (#27483)
* [ds-mobile-app] Implement reject changes in document updater

* Return rejected change ids

GitOrigin-RevId: 7915170af9164852ce39d783776106e722aaa925
2025-08-11 08:05:29 +00:00
Jakob Ackermann
adf399fb95 [project-history] add support for resync of history-ot ranges (#26475)
* [project-history] add support for resync of history-ot ranges

* [project-history] avoid compressing sharejs and history-ot upgrades

* [document-updater] improve error message of some assertions

... by migrating the assertions like this:
```diff
-stub.calledWith().should.equal(true)
+stub.should.have.been.calledWith()
```
```diff
-stub.called.should.equal(false)
+stub.should.not.have.been.called
```

* [document-updater] move content field in resyncDocContent

* [document-updater] add support for resync of history-ot ranges

GitOrigin-RevId: e6104686a26934a5f25a8f095cbe00c163fbbaa7
2025-06-19 08:05:18 +00:00
Jakob Ackermann
842f6c289f [document-updater] make setDoc aware of tracked deletes in history-ot (#26126)
GitOrigin-RevId: efa1a94f2f435058b553f639e43832454c58591d
2025-06-06 08:05:23 +00:00
Jakob Ackermann
64984ee86a [history-ot] flush history for projects with short queues ASAP (#25776)
* [document-updater] flush history for projects with short queues ASAP

* [k8s] document-updater: enable short history queue for history-ot demo

* [project-history] flush history for projects with short queues ASAP

* [project-history] wait for mongo before running acceptance tests

* [k8s] project-history: enable short history queue for history-ot demo

* [project-history] change wait-for-mongo step in tests

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>

---------

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: 3e989c409e8e9887655b35f2659ce0829e61b357
2025-05-23 08:05:35 +00:00
Jakob Ackermann
1e6b13f9d5 [history-ot] rename remaining history-v1-ot references to history-ot (#25428)
* [history-ot] rename remaining history-v1-ot references to history-ot

* [web] rename History-v1 OT -> History OT in admin panel

* [web] rename OT Migration -> History OT Migration in admin panel

GitOrigin-RevId: 103ce816d5320d6379d51009cdc08b8a71aa48e6
2025-05-20 08:06:46 +00:00
Antoine Clausse
fee5ea8411 Remove returns of promises within functions with callbacks (address DeprecationWarning) (#25603)
* [document-updater] Don't return promises within functions with callbacks

Remove the errors:
DeprecationWarning: Calling promisify on a function that returns a Promise is likely a mistake
https://cloudlogging.app.goo.gl/YHDhoarvLEw2w9rXA

* Remove some more unnecessary returns in functions with callbacks, for consistency

* Add `sendCanaryAppliedOp` to excluded methods for promisification

GitOrigin-RevId: fa6d3e47c4e6561dc29d4c15e57c3289fc1f3dfa
2025-05-16 08:05:33 +00:00
Jakob Ackermann
e8b5ee2ff9 [history-ot] initial implementation of using doc-level history-ot (#25054)
* [history-v1-ot] initial implementation of using doc-level history-v1-ot

* [web] fix advancing of the otMigrationStage

Use 'nextStage' for the user provided, desired stage when advancing.

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [document-updater] document size check in editor-core

* [history-ot] rename history-v1-ot to history-ot and add types

* [history-ot] apply review feedback

- remove extra !!
- merge variable assignment when processing diff-match-match output
- add helper function for getting docstore lines view of StringFileData

Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>

* Revert "[document-updater] add safe rollback point for history-ot (#25283)"

This reverts commit d7230dd14a379a27d2c6ab03a006463a18979d06

Signed-off-by: Jakob Ackermann <jakob.ackermann@overleaf.com>

---------

Signed-off-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
GitOrigin-RevId: 89c497782adb0427635d50d02263d6f535b12481
2025-05-08 08:05:44 +00:00
Jakob Ackermann
a5e2708eae [document-updater] add safe rollback point for history-ot (#25283)
GitOrigin-RevId: d7230dd14a379a27d2c6ab03a006463a18979d06
2025-05-06 08:05:28 +00:00
Jakob Ackermann
dd3956f5f4 [document-updater] delete dead code for handling JSON documents (#25036)
GitOrigin-RevId: 59a2c5f0174cd041ebda1ea4898114e6fb6d41a5
2025-04-24 08:05:35 +00:00
Jakob Ackermann
ff78f687d8 [document-updater] add endpoint for project wide last updated timestamp (#24352)
* [document-updater] fix acceptance test for sending two updates

The Object.create() invocation yields an empty object. The following v
assignment works as expected. The effective update is { v: 43 }.
Processing that fails as no operations were included.

* [document-updater] add endpoint project wide last updated timestamp

* [document-updater] apply review feedback

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>

---------

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: 81397537bfd85c2077f19669860b1391c15b34a3
2025-03-24 10:46:38 +00:00
Jakob Ackermann
b0c9176634 [project-history] add resync for project structure only (#23271)
* [project-history] add resync for project structure only

* [project-history] block resyncProjectStructureOnly that update docs

* [project-history] tweak test description

GitOrigin-RevId: ce2749566c36e04dab21c26dd60dd75d93c0d4c0
2025-02-03 09:05:39 +00:00
Eric Mc Sween
76dd6d1e20 Merge pull request #23212 from overleaf/em-docupdater-send-doc-hash
Send doc hash with history updates from docupdater

GitOrigin-RevId: 119475d4198c6603cecd4fd579a64ff4448261ce
2025-01-30 09:05:21 +00:00
Eric Mc Sween
9f1b6d480b Merge pull request #22869 from overleaf/em-remove-sanity-check
Remove RangesManager sanity check

GitOrigin-RevId: 376c2a197aa68cbde9259ec8c2cea1e9d43c8f69
2025-01-16 09:05:53 +00:00
Eric Mc Sween
e9c1c0f9c8 Merge pull request #22650 from overleaf/em-tracked-deletes-at-same-position
Handle multiple tracked deletes at same position

GitOrigin-RevId: 3cbf1c418bcd50cf08e1b90ce6ba3bc480236079
2025-01-14 09:05:34 +00:00
Domagoj Kriskovic
30ebad91b7 Allow reviewers to resolve their own comments (#22582)
* Allow reviewers to resolve their own comments

* check if reviewer is comment author

* add missing translation

* add CommentsController tests

* added DocumentManagerTests

* added HttpControllerTests

* Add AuthorizationManagerTests

* added AuthorizationMiddlewareTests

* added DocumentUpdaterHandler test

* fix test descriptions

* remove returns from CommentsControllerTests

* use ensureUserCanResolveThread in authorizationMiddleware

* move canResolveThread to AuthorizationManager

* commentId as param in NotFoundError

* refactor canUserResolveThread

GitOrigin-RevId: 131c3d1eb9ac916eaaa9221d351a92bc07b80cdc
2025-01-14 09:05:11 +00:00
Eric Mc Sween
a323a5c915 Merge pull request #22527 from overleaf/revert-22471-em-tracked-deletes-at-same-position
Revert "Improve handling of tracked delete rejections"

GitOrigin-RevId: 444a5a73aa2b93162036dfc463cd76d9c463aadb
2024-12-16 09:05:10 +00:00
Eric Mc Sween
6737cfa38d Merge pull request #22471 from overleaf/em-tracked-deletes-at-same-position
Improve handling of tracked delete rejections

GitOrigin-RevId: 35857d0a3f739c0531223737b2b649c9e8033157
2024-12-16 09:04:58 +00:00
Mathias Jakobsen
1063dabf33 [web+document-updater] Allow appending to documents (#20745)
Co-authored-by: David Powell <david.powell@overleaf.com>
GitOrigin-RevId: f66283926e7da3edf83ada9316c3a001287e1b42
2024-12-12 09:05:31 +00:00
Andrew Rumble
ad8026038c Merge pull request #22397 from overleaf/revert-22320-revert-22213-ar-avoid-duplicate-blob-writes
Revert "Revert "Avoid duplicate blob writes""

GitOrigin-RevId: f6a0083c0edf9ede14dfb52dbf1809a843aa5ff4
2024-12-10 09:05:07 +00:00
Andrew Rumble
ffa28b1a58 Merge pull request #22320 from overleaf/revert-22213-ar-avoid-duplicate-blob-writes
Revert "Avoid duplicate blob writes"

GitOrigin-RevId: 9f86bcea654cd3fa5f66fbdf42080e7f6e2861a7
2024-12-05 09:04:48 +00:00
Andrew Rumble
40b781eb0a Merge pull request #22213 from overleaf/ar-avoid-duplicate-blob-writes
Avoid duplicate blob writes

GitOrigin-RevId: 5e7a472b235bae5ef84389dbbe791189e951908a
2024-12-04 09:05:08 +00:00
Antoine Clausse
7f48c67512 Add prefer-node-protocol ESLint rule (#21532)
* Add `unicorn/prefer-node-protocol`

* Fix `unicorn/prefer-node-protocol` ESLint errors

* Run `npm run format:fix`

* Add sandboxed-module sourceTransformers in mocha setups

Fix `no such file or directory, open 'node:fs'` in `sandboxed-module`

* Remove `node:` in the SandboxedModule requires

* Fix new linting errors with `node:`

GitOrigin-RevId: 68f6e31e2191fcff4cb8058dd0a6914c14f59926
2024-11-11 09:04:51 +00:00
Eric Mc Sween
7444026cc3 Merge pull request #21310 from overleaf/em-validate-tracked-changes
Reapply "Sanity check for tracked changes in document-updater"

GitOrigin-RevId: e7b38d192f5202006f61bd015bba81d751af5413
2024-10-30 09:04:46 +00:00
Brian Gough
4920af44b0 Merge pull request #20679 from overleaf/bg-issue19022
Move flush_all_projects endpoint in document-updater into script

GitOrigin-RevId: cb774d860b5928b7fece1a8e21b0b76aecae73ff
2024-10-14 11:10:42 +00:00
Eric Mc Sween
731ba80b47 Merge pull request #20888 from overleaf/revert-20834-em-validate-tracked-deletes
Revert "Sanity check for tracked changes in document-updater"

GitOrigin-RevId: 1b7592b1f2d60374ce9ba54d9db5e34c9636ea9a
2024-10-14 11:05:01 +00:00
Eric Mc Sween
dc445121fe Merge pull request #20834 from overleaf/em-validate-tracked-deletes
Sanity check for tracked changes in document-updater

GitOrigin-RevId: 5f4c5cb931a6dae5257fed2f21e40777cb466309
2024-10-14 11:04:04 +00:00
Eric Mc Sween
a93223c70b Merge pull request #20826 from overleaf/revert-20799-em-ranges-tracker-sanity-checks
Revert "Sanity check for tracked changes in document-updater"

GitOrigin-RevId: 7876d57298d0f5dbd54929fdf69bce2976f16a9f
2024-10-14 11:01:33 +00:00
Eric Mc Sween
b881a96b84 Merge pull request #20799 from overleaf/em-ranges-tracker-sanity-checks
Sanity check for tracked changes in document-updater

GitOrigin-RevId: 5094eee8c279eb194114ac6f7fa36f86c9e16ca7
2024-10-14 11:01:24 +00:00