agw/overleaf-cep
1
0
Fork 0
mirror of https://github.com/yu-i-i/overleaf-cep.git synced 2026-11-14 14:13:43 +01:00
Code Issues Packages Projects Releases Wiki Activity
592 Commits 13 Branches 23 Tags
dceef85ccb20197ccca4368f53455fa4cd829b2a
Commit Graph
250 Commits
Author SHA1 Message Date
Simon DetheridgeandJakob Ackermann dceef85ccb Update app/js/FileHandler.js 
Co-Authored-By: Jakob Ackermann <das7pad@outlook.com>
 2020-03-31 16:03:38 +01:00
Simon Detheridge 4489670497 Only delete the converted-cache folder if conversions are enabled 2020-03-31 15:38:42 +01:00
Simon Detheridge f440cec8ec Check if streams are already destroyed before destroying 2020-03-31 11:07:31 +01:00
Simon Detheridge 5fadafaf96 Destroy streams handled by getReadyPipeline on error 2020-03-30 17:14:46 +01:00
Simon Detheridge 87b8f8e194 Destroy file-download stream on error 2020-03-30 17:13:40 +01:00
Simon Detheridge 4cb4d450be Call end() when handing ERR_STREAM_PREMATURE_CLOSE 2020-03-30 15:03:09 +01:00
Simon Detheridge a1ae68f6b5 Tidy up 'getReadyPipeline' 2020-03-26 16:57:55 +00:00
Simon Detheridge 1f037ef653 Catch errors from pipeline and stream when waiting for streams to be readable 2020-03-26 16:57:55 +00:00
Simon Detheridge 7e45a82c35 Use autodestroy on Transform stream 2020-03-26 15:09:56 +00:00
Simon Detheridge d073fe75ca Use single pipeline when calculating md5 2020-03-25 16:59:51 +00:00
Simon Detheridge 65f19346f0 Fix fallback to custom s3 buckets 2020-03-19 11:10:31 +00:00
Simon Detheridge cb4bdd99f4 Use an ISODate for deleted file names 2020-03-18 20:19:12 +00:00
Simon Detheridge 9d32d4ec16 Don't modify 'opts' parameter 2020-03-16 15:57:37 +00:00
Simon Detheridge b37c52fc3a Make GCS delete concurrency configurable 2020-03-16 15:54:05 +00:00
Simon Detheridge 9b658dda18 Copy-on-delete before unlocking 2020-03-16 15:53:45 +00:00
Simon Detheridge 06c4c0f74f Fix incorrect key when deleting projects 2020-03-16 11:35:01 +00:00
Simon Detheridge ce52f8aa60 Update FSPersistor deleteDirectory behaviour to match S3 and GCS 2020-03-16 11:33:51 +00:00
Simon Detheridge 47e96a4d94 Add endpoint to delete a project 
Needs acceptance tests
 2020-03-14 14:56:29 +00:00
Simon Detheridge fc80aa3954 Move directory key validation into FileHandler 2020-03-14 14:31:30 +00:00
Simon Detheridge 58db14456a Add timestamp to files in deleted bucket 2020-03-14 14:11:17 +00:00
Simon Detheridge edf1ce1f7e Delete files from a directory in parallel 2020-03-14 14:02:58 +00:00
Simon Detheridge 183cb0179a Add support to GCS persistor for unlocking files and copying on delete 2020-03-13 16:18:45 +00:00
Simon Detheridge 28c3fe4a56 Validate key names when deleting directory with a configurable regex 2020-03-10 17:54:09 +00:00
Simon Detheridge eb93ae4b10 Use Bucket.deleteFiles to delete directory contents, instead of iterating 2020-03-05 14:12:15 +00:00
Simon Detheridge 460dd96b17 Cosmetic clean-up of GCS Persistor & tests 2020-03-05 13:45:46 +00:00
Simon Detheridge 3bb956b38e Use http for the fake GCS server 2020-03-04 17:01:20 +00:00
Simon Detheridge 30114cd79b Remove unnecessary try/catch around 'createReadStream' 2020-03-04 16:38:05 +00:00
Simon Detheridge a7198764cb Improve/add some comments for clarity 2020-03-04 16:25:11 +00:00
Simon Detheridge def383574e Simplify getMeteredStream to record metric directly 2020-03-04 16:17:36 +00:00
Simon Detheridge 76243fd75a 'bucket' -> 'bucketName' in GCS Persistor 2020-03-04 16:04:26 +00:00
Simon Detheridge e58284aefe Move base64/hex methods to PersistorHelper 
Also add some null-safety checks
 2020-03-04 15:38:18 +00:00
Simon Detheridge 366ce97169 Add GCS Persistor 2020-03-04 15:09:14 +00:00
Simon Detheridge b5e4345855 Merge pull request #96 from overleaf/jpa-dependencies-cleanup 
[misc] dependencies cleanup
 2020-03-04 10:00:56 +00:00
Jakob Ackermann 2b15729658 [misc] promisify FileHandler and remove dependency on async 
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-03-02 18:23:17 +01:00
Jakob Ackermann 32557ab1d7 [SafeExec] replace _.once with lodash.once 2020-03-02 17:08:42 +01:00
Jakob Ackermann 0db4a17a14 [HealthCheckController] use fs.copyFile instead of fs-extra.copy 
the fs-extra method has a HUGE overhead of JS code and also syscalls for
 no particular benefit in this case: just copy the tiny.pdf file.

Here is an overview of the major operations: paths are relative to
  https://github.com/jprichardson/node-fs-extra/blob/1.0.0

We start in /lib/copy/copy.js
- sys: check that the source file exists
- sys: check that the source file has an existing parent directory?!
Continue in /lib/copy/ncp.js
- sys: more stat calls on both source and dest to determine permissions
- read/write streams to pipe the file content through the process
- sys: chmod on the destination to match the source permissions

What we actually need is a call to the binding and let node/the os
 figure out the best way to copy the contents.

Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-03-02 17:02:35 +01:00
Simon Detheridge 8acc9ef0ae Make S3 partSize a setting 2020-02-17 14:06:15 +00:00
Simon Detheridge 7c4bf97a7f Avoid fetching file a second time when source md5 can't be fetched 2020-02-17 14:06:05 +00:00
Simon Detheridge 2f2a819b74 Re-fetch file to calculate md5 if etag is not in correct format 2020-02-17 14:05:43 +00:00
Simon Detheridge 5adfb3e2c0 Use large upload chunks 2020-02-17 14:05:31 +00:00
Simon Detheridge 4e507197fc Revert "Merge pull request #91 from overleaf/revert-78-spd-migration-persistor" 
This reverts commit 0cce792604c258538cf012486e7329fa1be7da0f, reversing
changes made to 08b606e22f855121b2d8a2832a5d7e2f45836c3f.
 2020-02-17 14:04:42 +00:00
Simon Detheridge df2ddbe0e1 Revert "Add Migration Persistor, to send missing file requests to a fallback persistor" 2020-02-14 14:26:33 +00:00
Simon Detheridge e4926e09a7 Move error handler to be the last middleware 2020-02-14 10:58:46 +00:00
Simon Detheridge 44fbc6a733 Merge pull request #82 from overleaf/spd-error-handler-error 
Fix log info assignment in error handler
 2020-02-14 09:18:16 +00:00
Simon Detheridge 3b011258d2 Tidy up md5 hash generation 2020-02-12 13:27:12 +00:00
Simon Detheridge 49ad408b30 Remove unused imports and format correctly 2020-02-12 13:27:12 +00:00
Simon Detheridge 93cd55fb79 Refactor persistors to use a helper for common things 2020-02-12 13:27:12 +00:00
Simon Detheridge 9e0b378948 Remove minipass as dependency and refactor to make things clearer 2020-02-12 13:21:52 +00:00
Simon Detheridge 5d5d325691 Preserve all error information when cleanup of copied file fails 2020-02-12 13:21:52 +00:00
Simon Detheridge b4b7fd226e Add mechanisms to transfer files with md5-based integrity checks 
Fix error in settings and tidy up tests

Remove unused variable declaration

Remove .only from tests and update eslint rules to catch it in future

Use  to catch errors more safely getting md5 hash

Avoid unnecessary call to S3 to get md5 response
 2020-02-12 13:21:52 +00:00