Commit Graph

29 Commits

Author SHA1 Message Date
Brian Gough
7a57ef00cb Merge pull request #30045 from overleaf/bg-object-persistor-make-list-directory-safer
Improve safety of object persistor

GitOrigin-RevId: bced9814de6613b388ca288a5f72cd42cff6c1d3
2025-12-05 09:06:42 +00:00
Miguel Serrano
3a35b8680e Merge pull request #28554 from overleaf/msm-force-s3-lib-storage-uploads
[object-persistor] Use `@aws-sdk/lib-storage` for all uploads

GitOrigin-RevId: ab8e54a7bae843f9e6b05ed9cf936130a36b8c2f
2025-10-08 08:05:55 +00:00
Miguel Serrano
176b1a562f Merge pull request #28543 from overleaf/msm-disable-retries-s3
[object-persistor] fix disable retries on PutObjectCommand

GitOrigin-RevId: 7a812385e70781939f21325572c0eedb856e734f
2025-09-18 08:06:07 +00:00
Miguel Serrano
f1dc6ec155 Upgrade aws-sdk to v3 (re-attempt) (#28410)
* Upgrade `aws-sdk` to v3

Also vendored nodemailer-ses-transport

* Moved default region to CE settings

* Ensure timeout is added to `requestHandler` and `region` is populated

* fix unsigned header in signed URL request

The x-amz-acl header is not needed when using the signed request,
as ACL are already defined when creating the signed URL in
PutObjectCommand constructor.

* Add default AWS region for latexqc service

* remove unnecessary region in compose files

* Use AWS_REGION=us-west-2 for history-v1

* prevent retries uploading streams with PutObjectCommand

* Remove AWS SDK JS message suppression

GitOrigin-RevId: 6fda6f02160023ffed76143397bbd965f86a9509
2025-09-18 08:05:22 +00:00
Miguel Serrano
8948e40b22 Revert "Merge pull request #27679 from overleaf/msm-aws-sdk-upgrade" (#28151)
This reverts commit 4989ae920d8b7fd9e79623947b7c40bcc2e56d92.

GitOrigin-RevId: 541d95bfeaa7ce820e8af67f646f013fe4fe5d21
2025-08-28 08:05:52 +00:00
Miguel Serrano
39381d808a Merge pull request #27679 from overleaf/msm-aws-sdk-upgrade
Upgrade `aws-sdk` to v3

GitOrigin-RevId: 4989ae920d8b7fd9e79623947b7c40bcc2e56d92
2025-08-28 08:05:36 +00:00
Andrew Rumble
4157f8ca00 Add a list directory method to the S3 persistor
GitOrigin-RevId: 6ecff3eb457dc2168ca49ff9409bb09fa932781c
2025-06-25 08:06:05 +00:00
Jakob Ackermann
8e74d3c58c Merge pull request #21947 from overleaf/bg-jpa-back-fill-script-tweaks
[history-v1] back_fill_file_hash: performance tweaks

GitOrigin-RevId: c3d0c7906707fc902addcde64eaf41c24ceeece7
2024-11-20 09:04:51 +00:00
Jakob Ackermann
97b39ef98f Merge pull request #21681 from overleaf/jpa-back-fill-file-hash
[history-v1] add script for back filling hash into project file-tree

GitOrigin-RevId: 70ab8c72f3eea1062708e4471b4ad1c60317ad26
2024-11-18 09:05:39 +00:00
Jakob Ackermann
d6e94d2586 Merge pull request #21908 from overleaf/jpa-storage-class
[object-persistor] s3: add support for setting storage class on upload

GitOrigin-RevId: dc5480a95ec7f1551f08848f7fa45f760d97ab22
2024-11-18 09:05:10 +00:00
Jakob Ackermann
2be894c18a Merge pull request #21892 from overleaf/jpa-auto-gunzip
[object-persistor] add autoGunzip option to getObjectStream

GitOrigin-RevId: 21cd6c9ab349017ddb28d165238371d967ab9a59
2024-11-15 09:06:32 +00:00
Antoine Clausse
7f48c67512 Add prefer-node-protocol ESLint rule (#21532)
* Add `unicorn/prefer-node-protocol`

* Fix `unicorn/prefer-node-protocol` ESLint errors

* Run `npm run format:fix`

* Add sandboxed-module sourceTransformers in mocha setups

Fix `no such file or directory, open 'node:fs'` in `sandboxed-module`

* Remove `node:` in the SandboxedModule requires

* Fix new linting errors with `node:`

GitOrigin-RevId: 68f6e31e2191fcff4cb8058dd0a6914c14f59926
2024-11-11 09:04:51 +00:00
Jakob Ackermann
122d89a831 Merge pull request #21660 from overleaf/jpa-s3-https
[object-persistor] s3: simplify using a custom CA for HTTPS endpoints

GitOrigin-RevId: 2c6a5312a842582e5e40e917ccc586392087cb7a
2024-11-08 09:07:00 +00:00
Jakob Ackermann
65dc6bf940 Merge pull request #21631 from overleaf/jpa-live-s3
[object-persistor] s3SSEC: make compatible with AWS S3

GitOrigin-RevId: 3cd1f5ec43141f15defe081fc916d2ca2d78ca01
2024-11-08 09:06:55 +00:00
Jakob Ackermann
bec73ddfae Merge pull request #21543 from overleaf/jpa-s3-seec-kek-rotation
[object-persistor] s3SSEC: add support for (automatic) KEK rotation

GitOrigin-RevId: 315082e894c74e276a8efbc46b41ec7e102f9010
2024-11-08 09:06:42 +00:00
Jakob Ackermann
859901ac0c Merge pull request #21380 from overleaf/jpa-s3-ssec-backend
[object-persistor] add backend for SSE-C with S3 using KEK and DEK

GitOrigin-RevId: 9676f5cd5e08107c8c284b68b8d450a1c05bf1b1
2024-11-08 09:06:37 +00:00
Jakob Ackermann
7b3e39f63f Merge pull request #21375 from overleaf/jpa-if-none-match
[object-persistor] add support for ifNoneMatch=* in sendStream

GitOrigin-RevId: 268f054ac1b6452105b02757cdec32bad00702fd
2024-11-01 09:05:11 +00:00
Jakob Ackermann
baaac44172 Merge pull request #20302 from overleaf/jpa-object-persistor-metrics
[object-persistor] add more metrics to getObjectStream and sendStream

GitOrigin-RevId: 9fe6b9d205de6ad27838f91d92d2b3a3d6c2f129
2024-09-24 08:05:54 +00:00
Jakob Ackermann
8fa676082e Merge pull request #20299 from overleaf/jpa-object-persistor-metrics-dep
[object-persistor] depend on @overleaf/metrics directly

GitOrigin-RevId: eb0c07af8101d44def14154abb552bc77254e074
2024-09-24 08:05:50 +00:00
Andrew Rumble
71187a51ba Merge pull request #18289 from overleaf/ac-ar-eslint-return-await
Add ESLint rule @typescript-eslint/return-await to backend services

GitOrigin-RevId: 75e3e32597827fcc852e69d479515fc72e8f45e4
2024-05-27 10:22:49 +00:00
Jakob Ackermann
06c839a230 Merge pull request #17237 from overleaf/jpa-s3-iam
[object-persistor] add support for IAM credentials in S3Persistor

GitOrigin-RevId: 4ff34082ca557b17d8a7803f700ee704a3d184c6
2024-02-22 09:04:52 +00:00
Jakob Ackermann
2760ad35fc Merge pull request #12899 from overleaf/jpa-object-persistor-pipe
[object-persistor] move away from manual .pipe()

GitOrigin-RevId: 5407d48fbbb026ba81f43c891499bd3a5ff59518
2023-05-23 08:05:12 +00:00
Jakob Ackermann
9cf1373777 Merge pull request #11769 from overleaf/jpa-s3-config
[server-ce] add support for s3 backend in filestore

GitOrigin-RevId: 20998547326e75acf22562d2964db78ae261b6bf
2023-02-10 09:12:23 +00:00
Jakob Ackermann
4b308553be Merge pull request #6120 from overleaf/jpa-same-linting-packages
[misc] move the linting setup to the root of the monorepo

GitOrigin-RevId: 1633e2a58598add0b727738cd3bfba0ab7bae781
2021-12-17 09:03:06 +00:00
Eric Mc Sween
6521837993 Add S3 options: httpOptions, maxRetries 2020-07-09 17:50:05 -04:00
Eric Mc Sween
3e7e4369af Implement getRedirectUrl() for the S3 backend 2020-07-08 17:13:20 -04:00
Eric Mc Sween
523ff9c4cd Support metadata when uploading objects
Add contentType and contentEncoding options to sendStream(). These
options will set the corresponding metadata on the object.

This changes the API. The fourth argument to sendStream() used to be the
source md5. Now, it's an options object and the source md5 is a property
of that object.
2020-07-08 08:13:53 -04:00
Simon Detheridge
990bacf3c2 Update to OError3 2020-07-07 10:40:37 +01:00
Simon Detheridge
e92b75a2f8 Create new module from overleaf/filestore persistors (#1)
* Create new module from overleaf/filestore persistors

* Convert persistors to ES6 classes with local settings

* Update README.md

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>

* Update README.md

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>

* Update .gitignore

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>

* Switch to AGPL license

* Paginate S3 list-object results

* Remove S3 client caching

* Clean up S3 md5-verification mechanism

* Update README for recent changes

* Update README.md

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>

* Remove package-lock

* Remove comment about FileHandler

* Add directory marker to FSPersistor.deleteDirectory

* Don't copy opts in GcsPersistor.getObjectStream

* Use Date.now instead of getTime

* Catch errors in migration persistor

* Check that settings.buckets exists

* Don't mutate options in ObserverStream constructor

* Update src/PersistorHelper.js

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>

* Lint and format fixes

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
2020-07-02 14:19:45 +01:00