* [clsi-cache] shard per zone into three instances
Keep the old instance as read fallback. We can remove it in 4 days.
Disk size: 2Ti gives us the maximum write throughput of 240MiB/s on a
N2D instance with fewer than 8 vCPUs.
* [clsi] fix format
* [k8s] clsi-cache: bring back storage-classes
* [k8s] clsi-cache: fix reference to zonal storage-classes
* [k8s] clsi-cache: add logging configs
* [clsi] improve sharding
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [clsi] fix sharding
Index needs to be positive.
* [clsi] fix sharding
The random part is static per machine/process.
* [clsi] restrict clsi-cache to user projects
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [k8s] clsi-cache: align CLSI_CACHE_NGINX_HOST with service LB
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: 1efb1b3245c8194c305420b25e774ea735251fb3
- Remove settings ignore, they are inconsistent and break local prettier
- Remove .dockerignore files, only root ignore file is used
- Move .idea/.run/*.swp/coverage to root
- Remove .npmrc entries, we are no longer writing the rc file
- Remove node_modules/.DS_Store, is contained in root
- Remove cruft
GitOrigin-RevId: 3025fd5acaef343312f55149466c638e759a6e1f
* [clsi] merge sandboxed compiles config from Server Pro and SaaS
* [clsi] reorder fallback env vars
Co-authored-by: Mathew Evans <matt.evans@overleaf.com>
* [server-pro] bump version of expected release with these changes
---------
Co-authored-by: Mathew Evans <matt.evans@overleaf.com>
GitOrigin-RevId: bada93fec89bcc3f2bab85b6e60b2e27de88b9c2
* Replace `diskusage` by `fs` in clsi
* Replace `diskusage` by `fs` in clsi-cache
* Update disk space calculations to include block size in bytes
Co-authored-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
* Add warning comments about Docker-for-Mac fs stats being off by a factor
---------
Co-authored-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
GitOrigin-RevId: 02ea07e531b89bb3d10ddfe780348b19cbddad1f
* [web] provide an actual rootFolder from EditorProviders in tests
- Fixup SocketIOMock and ShareJS mocks to provide the complete interface
- Extend SocketIOMock interface to count event listeners
- Fixup test that did not expect to find a working rootDoc
* [web] expose imageName from ProjectContext
* [clsi-cache] check compiler settings before using compile from cache
* [web] avoid fetching initial compile from clsi-cache in PDF detach tab
GitOrigin-RevId: e3c754a7ceca55f03a317e1bc8ae45ed12cc2f02
* [clsi] drop support for docker-in-docker
* [clsi] run SyncTeX in specific output dir rather than compile dir
* [clsi] store output.synctex.gz outside of tar-ball in clsi-cache
* [clsi] add documentation for rewriting of docker bind-mounts
* [server-pro] update env vars for sandboxed compiles in sample config
GitOrigin-RevId: 8debd7102ac612544961f237aa4ff1c530aa3da3
* [clsi-cache] initial revision of the clsi-cache service
* [clsi] send output files to clsi-cache and import from clsi-cache
* [web] pass editorId to clsi
* [web] clear clsi-cache when clearing clsi cache
* [web] add split-tests for controlling clsi-cache rollout
* [web] populate clsi-cache when cloning/creating project from template
* [clsi-cache] produce less noise when populating cache hits 404
* [clsi-cache] push docker image to AR
* [clsi-cache] push docker image to AR
* [clsi-cache] allow compileGroup in job payload
* [clsi-cache] set X-Zone header from latest endpoint
* [clsi-cache] use method POST for /enqueue endpoint
* [web] populate clsi-cache in zone b with template data
* [clsi-cache] limit number of editors per project/user folder to 10
* [web] clone: populate the clsi-cache unless the TeXLive release changed
* [clsi-cache] keep user folder when clearing cache as anonymous user
* [clsi] download old output.tar.gz when synctex finds empty compile dir
* [web] fix lint
* [clsi-cache] multi-zonal lookup of single build output
* [clsi-cache] add more validation and limits
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [clsi] do not include clsi-cache tar-ball in output.zip
* [clsi-cache] fix reference after remaining constant
Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
* [web] consolidate validation of filename into ClsiCacheHandler
* [clsi-cache] extend metrics and event tracking
- break down most of the clsi metrics by label
- compile=initial - new compile dir without previous output files
- compile=recompile - recompile in existing compile dir
- compile=from-cache - compile using previous clsi-cache
- extend segmentation on compile-result-backend event
- isInitialCompile=true - found new compile dir at start of request
- restoredClsiCache=true - restored compile dir from clsi-cache
* [clsi] rename metrics labels for download of clsi-cache
This is in preparation for synctex changes.
* [clsi] use constant for limit of entries in output.tar.gz
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
* [clsi-cache] fix cloning of project cache
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: 4901a65497af13be1549af7f38ceee3188fcf881
* [web] convert file-list to typescript
* [web] add type annotation for clsiServerId without providing default
Co-authored-by: David Powell <david.powell@overleaf.com>
---------
Co-authored-by: David Powell <david.powell@overleaf.com>
GitOrigin-RevId: 5ecb79c2540a3e46e296c6bf7f8573fb65febc3f
* [clsi] add helper function for detecting extraneous files
* [clsi] output.pdfxref and output.synctex.gz are extraneous as well
* [clsi] minor code cleanup
Co-authored-by: Mathias Jakobsen <mathias.jakobsen@overleaf.com>
---------
Co-authored-by: Mathias Jakobsen <mathias.jakobsen@overleaf.com>
GitOrigin-RevId: 0d6c96c99b6be3adaee1558b1f958da885e6307a
* [web] generate clsi buildId ahead of fetching project content
The buildIds timestamp component will be used for cache invalidation.
* [clsi] strict validation for buildId
* [clsi] validate buildId parameter
GitOrigin-RevId: 88e8b2d48e78fa137b6dca7f2e6b93bbcf88a777
* [clsi] make error copying output files a fatal compile error
Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>
* [clsi] remove unused endpoints for downloading files from compileDir
Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>
* [clsi] avoid useless clone of output files
* [clsi] add test for output files when compile did not produce a PDF
---------
Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>
GitOrigin-RevId: cb998b99b4d96cb48ddd70987958f614ad3b40fc
* Update shellcheck files to include non ".sh" files having the sh shebang
* Add shellcheck to root Makefile
* `make shellcheck_fix`
* Add shellcheck to server-ce and server-pro
* Exclude SC1091 (Not following) in SP/CE
* Fix errors in SP
* Fix errors in CE
* Update root shellcheck to ignore all failing rules. We can later remove rules one by one.
* Add shellcheck to web
* Add shellcheck step for server-ce and server-pro in cloudbuild.yaml
* Revert "Add shellcheck to root Makefile"
This reverts commit e0fa56f2
* Revert "`make shellcheck_fix`"
This reverts commit eb179245c109a9e742a7fdeeb75a4bdd03963587.
* `make shellcheck_fix` in server-ce
* Fix: Use $(...) notation instead of legacy backticked
```
In init_scripts/100_set_docker_host_ipaddress.sh line 5:
echo "`route -n | awk '/UG[ \t]/{print $2}'` dockerhost" >> /etc/hosts
^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.
Did you mean:
echo "$(route -n | awk '/UG[ \t]/{print $2}') dockerhost" >> /etc/hosts
For more information:
https://www.shellcheck.net/wiki/SC2006 -- Use $(...) notation instead of le...
```
* `make shellcheck_fix` in web
* Manual corrections on autofix
* Update SHELLCHECK_FILES to Perl regex so it matches at start of file
GitOrigin-RevId: 815d53f319a3792efa22703beb75570de5977450
* [misc] silence logger when running tests
* [misc] re-enable logging in some tests for scripts
* [misc] make it easy to turn on verbose logging for tests
```
LOG_LEVEL=debug make test_unit
LOG_LEVEL=debug make test_acceptance
```
GitOrigin-RevId: 219bc6d1f9cbdb89ddd7d94742920913ddde4514
* Add `unicorn/prefer-node-protocol`
* Fix `unicorn/prefer-node-protocol` ESLint errors
* Run `npm run format:fix`
* Add sandboxed-module sourceTransformers in mocha setups
Fix `no such file or directory, open 'node:fs'` in `sandboxed-module`
* Remove `node:` in the SandboxedModule requires
* Fix new linting errors with `node:`
GitOrigin-RevId: 68f6e31e2191fcff4cb8058dd0a6914c14f59926
Finalize promise will only resolve when the archive is closed. We want
to stream as soon as we have the data so this does not suit us. We want
to log errors that are thrown due to finalize, these should be
propogated by archiver to the response already.
GitOrigin-RevId: 4f9d727a00ead1be3caee62e1e0adba069a545c7
This allows us to:
* Get files that are in subdirectories (not known to happen but might as
well support)
* Removes need to sanitise missing_files output
GitOrigin-RevId: 4b27b7f1b506578a6d713ea23979aa9370dfdcb0
Jakob Ackermann