Commit Graph

132 Commits

Author SHA1 Message Date
Jakob Ackermann 6c6e8d9a97 [monorepo] switch all output file reads to clsi-nginx (#31691)
* [monorepo] switch all output file reads to clsi-nginx

* [clsi-lb] allow gallery download requests

* [terraform] clsi: use nginx.conf from clsi service

* [clsi] fix flakey tests

* [clsi] replace alias with rewrite and root in nginx config

* [k8s] clsi-lb: expose download port on internal service

* [web] add explicit endpoint for downloading all output files

Serve the output.zip endpoint from clsi.

* [clsi] fix regex for latexqc submission ids

Previously, we only handled template submission ids.

GitOrigin-RevId: 6c3b21b01ec41ae767530b14aac31fbe3d640dd5
2026-02-24 09:07:12 +00:00
Brian Gough f3e8601cba fix caching of minted output files in TL2025 (#31455)
GitOrigin-RevId: b82df4d9c7898332b310fd956c5f002bf5b20e39
2026-02-11 09:06:14 +00:00
Rebeka Dekany a648015db8 Centralize prettier configuration to root level (#30501)
* Merge all .prettierignore files into top-level config

* Merge all .prettierrc files into top-level config

* Replace service-specific glob patterns in package.json format scripts with `prettier .`

* Add template files with Jinja2, Go template, envsubst, and Handlebars syntax to .prettierignore

* Ignore GitHub templates

* Ignore PUG templates to format them separately with `format:pug`

* Encourage double quotes for YAML, YML files

* Move prettier for PUG source format script to the root

* Move prettier for styles source format script to the root

* Remove prettier for jenkins files from web

* Remove prettier source format script from all services

* Make .prettierrc more readable

* Update format scripts by file type

* Organise `.prettierignore`

* Add `--cache` flag to prettier scripts for faster runs

* Format all files with prettier

* Format all or format services

* Remove `format`/`format:fix` scripts from services since now it runs from root `package.json`

* Avoid conlficts with yamllint configuration

* Remove `--cache` flag from prettier scripts

* Update all service Makefiles to use root-level prettier configuration

* Update all Jenkinsfile to use root-level prettier configuration

* Ignore auto-generated files by build_scripts

* Update package-lock.json

* Update root Makefile format targets

* Update SP Jenkinsfile format target

* Update E2E Makefile format script

* Udpate `format_js` to work in both local and CI env

* Add docker-mailtrap to .prettierignore

docker-mailtrap is a third-party git-ignored directory used for testing

* Added Docker env detection to prevent nested Docker spawning

* Ignore handlebars templates

* Add cryptographic files and test output to `.prettierignore`

* Add terraform modules to `.gitignore`

* Remove prettier-plugin-groovy

* Use npx directly instead of Docker for local formatting for faster formatting

* Auto-generate Makefiles

* Revert "Remove prettier-plugin-groovy"

This reverts commit 194a33589a2e1e4d2225d10c67e9f025e4222025.

* Mount monorepo root in RUN_LINT_FORMAT for prettier config access

* Prettier ignores all `node_modules` by default regardless of location

* Show only changed files in format output

* Ignore LICENSE files

* Enable prettier on rendered build_scripts outputs

* Ignoring all the template folders by prettier

* Remove the public/minjs entry since it does not exist

* Remove all non-existent paths

* Sync `.prettierignore` with ignored files by `.gitignore` and `.dockerignore` files

* Revert "Auto-generate Makefiles"

This reverts commit c0233e490de1bc95fe437219d65e0b66e0331ec9.

* Revert "Use npx directly instead of Docker for local formatting for faster formatting"

This reverts commit 1d2b2cf1a6c6974c76885852a90dd55e84167e41.

* Ignore dashboard JSON files

* Ignore files generated by bin/update_build_scripts

* Remove unsupported file types from `.prettierignore`

* Ignore test fixture generated files

* Ignore README file types by prettier

* Ignore generate snapshots by prettier

* Allow to format generated bin/update_build_scripts by prettier

* Ensure build script outputs prettier-compatible tsconfig.json

* Fix build script output to match prettier formatting
- Fix Jinja2 whitespace in docker-compose templates
- Change YAML quotes from single to double

* Don't read cryptographic files by prettier

* Ignore google verification files by prettier

* Revert npx prettier formatting

* Ignore domain verification files

* Show only changed files in format output

* Make `.github` prettier

* Allow all files to be formatted in jobs by prettier

* Allow server-ce/server-pro files to be formatted by prettier

* Ignore more folders in clsi, filestory, git-bridge by prettier

* Update build script with `RUN_LINTING_CI_MONOREPO`

* Ignore docker-mailtrap and downloads in server-ce by prettier

* Restore prettier configs and prettierignore for V1 since it has its own prettier (an older version)

* Source format

GitOrigin-RevId: 637adc3cc422d1f20c86d6ebc8ec514d60758287
2026-02-04 09:08:22 +00:00
Andrew Rumble cd7da983d1 Merge pull request #30232 from overleaf/ar/convert-clsi-to-es-modules
[clsi] convert to ES modules

GitOrigin-RevId: fb7fa52cc8f678ee31be352e62a5dff95e88008b
2026-01-22 09:06:23 +00:00
Jakob Ackermann 3f9a7cf463 [clsi] consolidate metrics for clsi-perf (#30746)
* [clsi] remove all clsi-perf/health-check metrics

* [clsi] always emit E2E compile time metric

* [clsi] do not collect metrics for clsi-cache-template compiles

* [clsi] fix unit tests: request.metricsOpts always exists

* [clsi] use a gauge for the e2e compile time metric of clsi-perf

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>

* [clsi] remove metrics for binary file downloads from clsi-perf

---------

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: 7995512e57c802086350e3d1a0ec5213ecdf0a05
2026-01-19 09:06:34 +00:00
Brian Gough 67aa42a57a Merge pull request #29650 from overleaf/bg-update-clsi-tests-to-2025
update clsi acceptance tests to use texlive 2025.1 by default

GitOrigin-RevId: d69e97132e87873a8b91c39494c545250298d935
2025-11-13 09:06:23 +00:00
Brian Gough 4ca1407ab9 Merge pull request #29638 from overleaf/bg-add-shell-escape-tests-to-clsi
add shell escape tests to clsi

GitOrigin-RevId: 6cd3ab24fa76f74dccfec43bf6a3d06c0fe9ec6a
2025-11-13 09:06:18 +00:00
Eric Mc Sween 74524db293 Merge pull request #28909 from overleaf/em-compile-metrics
Use histograms to track CLSI compile times

GitOrigin-RevId: cf25f1e6d2094186f419acc70748f0c71b6c3240
2025-10-13 08:07:02 +00:00
Brian Gough d24f37d3a4 Merge pull request #28880 from overleaf/bg-add-time-option-to-clsi
add latexmk `-time` option to clsi and record performance logs

GitOrigin-RevId: 467473859359913da73f83e10b63b45603ea175c
2025-10-09 08:06:12 +00:00
Miguel Serrano e8bc186ca0 Merge pull request #28752 from overleaf/msm-clsi-acceptance-async-await
[clsi] async/await migration in acceptance tests

GitOrigin-RevId: d614fabb6d568dc5c955603fb923fb40b871a703
2025-10-02 08:06:04 +00:00
Brian Gough 2f2862ecd7 Merge pull request #26637 from overleaf/bg-clsi-fix-process-group-for-local-compiles
fix "stop compile" option for local command runner in CE/SP

GitOrigin-RevId: 7986b505362aaf33ac6e161b3b54458baba1e2e6
2025-07-11 08:07:11 +00:00
Jakob Ackermann d489e35782 [web] emit event when synctex mapping was downloaded from clsi-cache (#25424)
* [clsi] tell frontend when synctex mapping was downloaded from clsi-cache

* [web] emit event when synctex mapping was downloaded from clsi-cache

GitOrigin-RevId: 1f6b7e0faaa7dd76449aad566802da971a4cf9ed
2025-05-09 08:06:00 +00:00
Jakob Ackermann 4a17a1e713 [web] gracefully access compile stats for event (#24818)
* [web] gracefully access compile stats for event

* [clsi] always emit stats and timings

GitOrigin-RevId: 959e5fe1508245ffecfab1219fd86e53b210fca1
2025-04-14 08:04:51 +00:00
Jakob Ackermann 1117ea1b3e [clsi] add helper function for detecting extraneous files (#24376)
* [clsi] add helper function for detecting extraneous files

* [clsi] output.pdfxref and output.synctex.gz are extraneous as well

* [clsi] minor code cleanup

Co-authored-by: Mathias Jakobsen <mathias.jakobsen@overleaf.com>

---------

Co-authored-by: Mathias Jakobsen <mathias.jakobsen@overleaf.com>
GitOrigin-RevId: 0d6c96c99b6be3adaee1558b1f958da885e6307a
2025-03-24 10:46:43 +00:00
Jakob Ackermann d8b0ab9436 [clsi] avoid downloads from compile directory (#23975)
* [clsi] make error copying output files a fatal compile error

Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>

* [clsi] remove unused endpoints for downloading files from compileDir

Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>

* [clsi] avoid useless clone of output files

* [clsi] add test for output files when compile did not produce a PDF

---------

Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>
GitOrigin-RevId: cb998b99b4d96cb48ddd70987958f614ad3b40fc
2025-03-03 09:05:04 +00:00
Jakob Ackermann 0a7b2004d2 [misc] silence logger when running tests (#22243)
* [misc] silence logger when running tests

* [misc] re-enable logging in some tests for scripts

* [misc] make it easy to turn on verbose logging for tests

```
LOG_LEVEL=debug make test_unit
LOG_LEVEL=debug make test_acceptance
```
GitOrigin-RevId: 219bc6d1f9cbdb89ddd7d94742920913ddde4514
2025-02-10 09:06:02 +00:00
Jakob Ackermann d19c5e236f Merge pull request #22208 from overleaf/jpa-clsi-hash
[misc] clsi: read files from history-v1 with fallback to filestore

GitOrigin-RevId: c54bb128780198c14e7a63818f39fad62ce65d4e
2024-11-29 09:05:39 +00:00
Antoine Clausse 7f48c67512 Add prefer-node-protocol ESLint rule (#21532)
* Add `unicorn/prefer-node-protocol`

* Fix `unicorn/prefer-node-protocol` ESLint errors

* Run `npm run format:fix`

* Add sandboxed-module sourceTransformers in mocha setups

Fix `no such file or directory, open 'node:fs'` in `sandboxed-module`

* Remove `node:` in the SandboxedModule requires

* Fix new linting errors with `node:`

GitOrigin-RevId: 68f6e31e2191fcff4cb8058dd0a6914c14f59926
2024-11-11 09:04:51 +00:00
Jakob Ackermann a540754f6e Merge pull request #18116 from overleaf/jpa-bulk-replace-localhost
[misc] bulk replace localhost with 127.0.0.1

GitOrigin-RevId: d238f3635302e8ff5500d611108c4d1bef216726
2024-04-26 08:04:39 +00:00
Jakob Ackermann a6fb983afc Merge pull request #17871 from overleaf/jpa-bind-v4-v6
[misc] align the host for binding and requesting in tests

GitOrigin-RevId: 916521c56928329ff2cbf2817f3a0a27aeaf8c3d
2024-04-12 08:04:51 +00:00
Jakob Ackermann b4b369fea5 Merge pull request #16428 from overleaf/jpa-clsi-replace-fs-extra
[clsi] replace fs-extra to avoid excess syscalls

GitOrigin-RevId: cbc8ec01b8ac9fd973e86a6d763c8f599323db94
2024-01-09 09:03:36 +00:00
Tyna William 36db21ed56 Merge pull request #13475 from overleaf/revert-13473-revert-13439-tw-support-rnw-files
Revert "Revert "support for Rnw files""

GitOrigin-RevId: 2bdaf154592f0b3c4bd3762d266b89a159a0b091
2023-07-17 10:55:08 +00:00
Tyna William ea59a98386 Merge pull request #13473 from overleaf/revert-13439-tw-support-rnw-files
Revert "support for Rnw files"

GitOrigin-RevId: 1116814958cabed4c741bd0d3cce7670a6ca5ed4
2023-07-17 10:33:54 +00:00
Tyna William 4100ee81a1 Merge pull request #13439 from overleaf/tw-support-rnw-files
support for Rnw files

GitOrigin-RevId: 407ab1b6b131408fa0e4256d50781e9f70b0c3b8
2023-07-17 10:33:37 +00:00
Jakob Ackermann a437dcf537 Merge pull request #12823 from overleaf/jpa-clsi-download-tests
[clsi] modernise tests for file downloads

GitOrigin-RevId: b33749b9100812de8ae2a26bd84e933f0282457b
2023-04-27 08:06:54 +00:00
Jakob Ackermann 5380999459 Merge pull request #12815 from overleaf/jpa-pipe-tests
[misc] tests: migrate to stream.pipeline or request({ body: stream })

GitOrigin-RevId: 5de077c9ba9c7951d6fec7b1d6da6fb694394610
2023-04-27 08:06:20 +00:00
Eric Mc Sween 16fee6d7d2 Merge pull request #12203 from overleaf/em-camel-case-clsi
Camel case variables in clsi

GitOrigin-RevId: 1542a6b7f8886e5fef496ce31acd2bba537befa9
2023-03-23 09:04:46 +00:00
Alf Eaton c41c14c697 Remove deprecated grunt code (#9506)
GitOrigin-RevId: 2b19d73c593545aaa9f6eb88143d08c5df39a1be
2022-09-16 08:05:37 +00:00
Jakob Ackermann 18768d3f1b Merge pull request #8950 from overleaf/bg-use-qpdf-xref-table
[clsi] parse xref table from qpdf output

GitOrigin-RevId: cc400c65082c7c8c28c21037b97b4fde66a85835
2022-08-03 08:04:10 +00:00
Brian Gough 9ea1226460 Merge pull request #8969 from overleaf/bg-issue8967
Apply draft mode to graphics package as well (in addition to graphicx)

GitOrigin-RevId: 6137f90ef9d579f0e6554801e8f2e0f5dee2aa83
2022-07-25 08:03:57 +00:00
Jakob Ackermann b4402c41e8 Merge pull request #8808 from overleaf/jpa-clsi-linearize-examples
[clsi] linearize example pdfs and update xref table snapshots

GitOrigin-RevId: 5cfad6fdc4f6aef3366b62ca1c171f92e5f392fd
2022-07-12 08:03:30 +00:00
Eric Mc Sween f0a3eeeb32 Merge pull request #7968 from overleaf/em-remove-logger-log
Remove logger.log in favour of logger.info

GitOrigin-RevId: e305ae8adf19ecf144cee123b6837f35d5d45bed
2022-05-17 08:05:39 +00:00
Jakob Ackermann 264b107e39 Merge pull request #6927 from overleaf/jpa-split-clsi-metrics
[misc] split clsi metrics into multiple categories

GitOrigin-RevId: 964ab57b344b92383a2937beff50139bae273a0e
2022-03-02 09:02:52 +00:00
Jakob Ackermann 958216d578 Merge pull request #6926 from overleaf/jpa-clsi-bump-logger-metrics
[clsi] upgrade logger and metrics module

GitOrigin-RevId: 85c346b5eed683672a77d86d6a434dc313b7824b
2022-03-02 09:02:50 +00:00
Jakob Ackermann f6795a324a Merge pull request #5572 from overleaf/jpa-no-path-concat
[misc] fix eslint violations for node/no-path-concat

GitOrigin-RevId: 5122826fb9ae23c373e8c5a6802ebb35eb20314f
2021-10-28 08:03:37 +00:00
Jakob Ackermann 7f9fd00bda Merge pull request #5367 from overleaf/jpa-node-handle-callback-err
[misc] fix eslint violations for node/handle-callback-err

GitOrigin-RevId: 83a4900e8861010df1917bff49382bd9c93375bd
2021-10-28 08:03:26 +00:00
Jakob Ackermann a6f05109a3 Merge pull request #5352 from overleaf/jpa-no-var
[misc] fix eslint violations for `no-var`

GitOrigin-RevId: c52e82f3a8a993b8662cc5aa56e7b95ca3c55832
2021-10-27 08:03:00 +00:00
Jakob Ackermann 94c208311c Merge pull request #5351 from overleaf/jpa-clsi-drop-sqlite
[clsi] goodbye sqlite db

GitOrigin-RevId: e7f16920c44e74a425b92884b48a60272dfb015b
2021-10-07 08:03:50 +00:00
Jakob Ackermann 02918e7483 Merge pull request #4649 from overleaf/jpa-fs-based-caching
[perf] UrlCache: pure fs based cache state for downloads

GitOrigin-RevId: d19afc396324d4c3318b31620c8ad0c04e0544ce
2021-10-07 08:03:18 +00:00
Eric Mc Sween 9ee92daea3 Merge pull request #4893 from overleaf/em-synctex
Use the synctex distributed with TeX Live

GitOrigin-RevId: 5a133f21f48fd1e217ab463e8cb2a5cdec8be1af
2021-09-07 16:26:17 +00:00
Jakob Ackermann f285e503b4 [misc] run format_fix and lint:fix 2021-07-13 12:04:48 +01:00
Jakob Ackermann 64551f0198 [misc] switch from settings-sharelatex to @overleaf/settings 2021-07-12 17:47:21 +01:00
Jakob Ackermann 294088fb27 [ContentCacheManager] use PDF.js Xref table instead of stream detection (#242)
* make the content cache manager tests configurable

* extend stream content in unit tests

* [ContentCacheManagerTests] prepare for full object caching

* filesystem stream for pdfjs

* working??

* cleaning up

* handle overflow

* [misc] install pdfjs-dist

* [misc] move pdfjs code into app/lib/ and scripts/, also use CamelCase

* [misc] abstract the file loading and parsing of xRef tables into helper

* [misc] pdfjsTests: add snapshot based tests for the Xref table parser

* [misc] FSStream: throw proper error and drop commented code

* [misc] FSStream: integrate throwing of MissingDataException into getter

* [misc] pdfjs: fix eslint errors

* [misc] pdfjs: run format_fix

* [misc] pdfjs: allocate very small non empty dummy buffers explicitly

* [misc] install @overleaf/o-error

* [ContentCacheManager] use PDF.js Xref table instead of stream detection

Co-Authored-By: Brian Gough <brian.gough@overleaf.com>

* [pdfjs] parseXrefTable: handle empty PDF files gracefully

Co-authored-by: Brian Gough <brian.gough@overleaf.com>
2021-05-31 09:20:25 +01:00
Jakob Ackermann d56a490121 [misc] AllowedImageNamesTests: add missing done callbacks 2021-05-25 10:24:49 +01:00
Alf Eaton 6507d668ae Update references to .coffee files 2021-05-19 18:57:35 +01:00
Alf Eaton 1b1c4aa4a0 Fix formatting 2021-05-19 18:47:51 +01:00
Alf Eaton f31abe7cb3 Tidy and format 2021-05-19 18:36:27 +01:00
decaffeinate 85920b3120 decaffeinate: Run post-processing cleanups on settings.test.coffee 2021-05-19 16:50:46 +01:00
decaffeinate 8c984a77c4 decaffeinate: Convert settings.test.coffee to JS 2021-05-19 16:50:45 +01:00
decaffeinate 5af8f57dd0 decaffeinate: Rename settings.test.coffee from .coffee to .js 2021-05-19 16:50:45 +01:00