agw/overleaf-cep
1
0
Fork 0
mirror of https://github.com/yu-i-i/overleaf-cep.git synced 2026-11-12 13:13:45 +01:00
Code Issues Packages Projects Releases Wiki Activity
28,693 Commits 13 Branches 23 Tags
6a74e50ba1f7b2d23a515a25a3703cfadd96ab3b
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Jakob Ackermann b62b96cb1c Revert "very basic mvp for shedding load" 2021-07-01 11:57:03 +01:00
Jakob Ackermann 34fc349646 [benchmarks] add benchmark for multi vs mget/mset 2021-04-13 11:47:10 +01:00
Jakob Ackermann 6e551f9b34 [perf] use MGET/MSET/DEL for manipulating multiple keys in one operation 
In some cases we can get rid of MULTI/EXEC operations too.
- putDocInMemory: from 10 down to 2 operations
- removeDocFromMemory: from 14+4 down to 4+4 operations
- updateDoc: from 13 down to 8 operations
 2021-04-13 11:47:10 +01:00
Jakob Ackermann 178440395f [perf] switch write sequence for doc contents and doc tracking 
Doc contents are added only after the tracking has been setup.
All read paths on the tracking have been checked to gracefully handle
 the case of existing doc_id but missing doc contents.

- getDoc: -1 operation

REF: 0a2b47c660c60b95e360d8f3b3e30b862ceb6d79
 2021-04-13 11:46:44 +01:00
Jakob Ackermann 50b24043b7 [perf] use MGET for fetching multiple keys in one operation 
- getDoc: from 13 down to 2 operations
 2021-04-09 08:42:35 +01:00
Jakob Ackermann b7b13a90d1 [misc] ApplyUpdateTests: process all the pending-updates-lists 2021-03-01 15:26:18 +00:00
Jakob Ackermann e1650616b8 [misc] version bump to 3.5.1 2021-02-16 14:21:36 +00:00
Jakob Ackermann 20223d865f [misc] do not register a noop sweeping interval 2021-01-12 14:20:51 +00:00
Jakob Ackermann 29742d6a87 [misc] register the metrics sweeping interval handle for later cleanup 2021-01-12 14:20:31 +00:00
Jakob Ackermann dddb431034 [misc] drop unused async 
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-11-09 16:02:30 +00:00
Jakob Ackermann a53eef5e3d [healthCheck] bring back support for redis cluster 
Assume `cluster-require-full-coverage=yes` is set for the redis cluster,
 which is the default value.

We are writing into a single hash slot, resulting in a partial health
 check only.
The 'cluster-require-full-coverage=yes' blocks writes as soon as any
 hash slot is not covered by an active primary.
We are leveraging the health check build into the redis cluster topology
 to bring the scope of the single read/write/delete to the full cluster.

REF: 1b3566d6c830289dc826da1d13b75d68205d8dec
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-11-09 16:02:29 +00:00
Jakob Ackermann 2d1c84ea2e [misc] fix collection of open sockets 
```
> require('http').globalAgent.sockets
{ 'HOST:PORT:': [{ ... }, { ... }] }
```

- `new URL('http://HOST:PORT:')` throws
  -> test suite `open_sockets`
- gauges were not cleaned up after no agents were present for a given
   key. This lead to continuous emission of the previous connection
   count for a given hostname, following the completion of all requests.
  -> test suite `when all requests complete`
 2020-11-06 15:16:14 +00:00
Jakob Ackermann 6f4829e817 [misc] fix module importing 2020-08-17 14:43:08 +01:00
Jakob Ackermann 37fda043d4 [http] use public node api for getting the response content-length 2020-08-17 11:59:18 +01:00
Jakob Ackermann a5c3bad7f1 [ExampleDocumentTests] drop out in case of an error during compilation 
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-07-03 11:38:12 +01:00
Jakob Ackermann 5cfd92cf85 [ASpell] error handling of cache dump: fix an async fs function call 
fs.unlink requires a callback as second parameter.

Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-07-03 11:13:01 +01:00
Jakob Ackermann ce4f9148c3 [perf] WebsocketLoadBalancer: move back to a sync loop for msg fan out 2020-06-22 17:04:15 +01:00
Jakob Ackermann a70c1e1fa2 [perf] WebsocketLoadBalancer: check is_restricted_message once 
...and filter clients early on.
 2020-06-22 17:04:15 +01:00
Jakob Ackermann b2e4448992 [misc] test/acceptance: ReceiveUpdateTests: test remotely sent update 2020-06-10 09:53:48 +01:00
Jakob Ackermann 83e3ff0ed7 [misc] test/acceptance: ReceiveUpdateTests: add 2nd project/3rd client 
...and check for cross project leakage.
 2020-06-10 09:53:29 +01:00
Jakob Ackermann 56fda1f9b0 [misc] test/acceptance: use the correct redis instances 2020-06-10 09:43:15 +01:00
Jakob Ackermann bc44494466 [HttpController] return 404 in case of a missing client and add tests 
Add acceptance tests for the client view.
 2020-06-10 09:41:36 +01:00
Jakob Ackermann de35fc5ecf [HttpApiController] implement the disconnection of a single client 
The http route returns as soon as the client has fully disconnected.
 2020-06-09 18:01:08 +01:00
Jakob Ackermann 91e296533f [misc] test/acceptance: add tests for the draining of connections 2020-06-09 18:00:44 +01:00
Jakob Ackermann 5937a20c05 Merge pull request #141 from das7pad/hotfix-container-deletion-locking 
[DockerRunner] destroyOldContainers: normalize the container name
 2020-05-29 12:31:50 +02:00
Jakob Ackermann 563a973388 [DockerRunner] destroyOldContainers: normalize the container name 
The docker api returns each name with a `/` prefix.

In order to not interfere with pending compiles, the deletion process
 has to acquire an internal lock on the container. The LockManager uses
 the plain container name without the slash: `project-xxx`.

Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-05-29 11:28:26 +01:00
Jakob Ackermann 419100d167 Merge pull request #2860 from overleaf/jpa-socket-io-v2-percentage-rollout 
[misc] prepare for a percentage roll-out of socket.io v2

GitOrigin-RevId: 792eddeb4a9b41aded67fee13faa949122cd0faf
 2020-05-27 03:15:52 +00:00
Jakob Ackermann 2a8e2efe62 Merge pull request #2845 from overleaf/jpa-faster-web-ci 
[misc] faster web ci

GitOrigin-RevId: bbd6b12040956f6b397c6ea4d8144e733299400b
 2020-05-23 03:18:08 +00:00
Jakob Ackermann 79ea042959 Merge pull request #137 from overleaf/jpa-backport-strict-sequence 
[misc] backport strict sequence of redis (un)subscribe
 2020-05-18 12:31:31 +02:00
Jakob Ackermann 4e400dc7da Merge pull request #136 from overleaf/jpa-backport-sinon-v2 
[misc] backport sinon v2
 2020-05-18 11:18:06 +02:00
Jakob Ackermann b713beb7f0 Merge pull request #135 from overleaf/jpa-skip-leave-project-for-invalid-clients 
[WebsocketController] skip leaveProject when joinProject didn't complete
 2020-05-13 15:34:41 +02:00
Jakob Ackermann df9d8ccbbe Merge pull request #134 from overleaf/jpa-handle-redis-subscribe-error-on-join-project 
[WebsocketController] handle redis subscribe error on joinProject
 2020-05-13 15:34:17 +02:00
Jakob Ackermann d3240b8f9c [perf] speedup login times for acceptance tests: use weak bcrypt params (#2801) 
The overhead of password checks goes down from about 200ms to
 sub-millisecond times on a modern i7 processor (3.8GHz).
In my testing on server CPUs (2.4GHz) this goes down from 400ms to
 single digit ms.

GitOrigin-RevId: 895580fcbf1cde0bae5213ed6d8aae30c67fcfa3
 2020-05-05 03:22:22 +00:00
Jakob Ackermann a3bc59fba5 Merge pull request #56 from overleaf/jpa-fix-install_deps 
[misc] fix install_deps
 2020-04-23 17:33:43 +02:00
Jakob Ackermann 35400de80a Merge pull request #2766 from overleaf/jpa-make-cleanup 
[make] cleanup dc networks and fix frontend target

GitOrigin-RevId: abcd0d061bd924ff0ddf885d1a18b66ebcc7e8e5
 2020-04-23 03:29:34 +00:00
Jakob Ackermann 8cf600089f Merge pull request #726 from overleaf/jpa-temporary-fix-for-lualatex 
[docker-compose] add a temporary fix for LuaLatex compiles
 2020-04-22 11:02:40 +02:00
Jakob Ackermann a0359009f9 Merge pull request #2677 from overleaf/jpa-socket-io-v2-shim 
[socket.io] add support for v2 using a compatibility shim

GitOrigin-RevId: ef52a3d8e4fdf191f6a8c1a530dfce25f3a11466
 2020-04-21 03:23:06 +00:00
Jakob Ackermann 9a1a9f153b Merge pull request #57 from overleaf/jpa-node-12-fix 
[misc] test/unit: fix process usage for node v12
 2020-04-20 17:06:26 +02:00
Jakob Ackermann 80095785b8 [misc] test/unit: fix process usage for node v12 
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-04-20 16:50:51 +02:00
Jakob Ackermann f6f20fdcb6 Merge pull request #43 from overleaf/ew-jpa-fix-express-deprecations 
[misc] fix express deprecations
 2020-04-17 11:21:04 +02:00
Jakob Ackermann 9f15b062b0 Merge pull request #19 from overleaf/ew-jpa-fix-express-deprecations 
[misc] fix express deprecations
 2020-04-17 11:20:48 +02:00
Jakob Ackermann cf8533bee1 [LocalCommandRunner] run: block a double call of the callback 
The subprocess event handler fires the "error" and "close" event in case
 of a failure.
Both events would call the given callback, resulting in double
 processing of the subprocess result downstream.

Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-04-16 15:55:55 +02:00
Jakob Ackermann 825d0701e8 Merge pull request #2742 from overleaf/jpa-hotfix-user-lookup 
[UserPagesController] block arbitrary user lookups

GitOrigin-RevId: 4d88abbd0ad14289a73e7f502c0686f206617459
 2020-04-14 03:19:24 +00:00
Jakob Ackermann c700a0ab9d Merge pull request #159 from overleaf/jpa-fix-docker-modem-error-scheme 
[misc] keep up with the error signature of dockerode/docker-modem
 2020-04-03 14:04:06 +02:00
Jakob Ackermann bb9e8bb6f5 Merge pull request #157 from overleaf/jpa-download-failed-metric 
[misc] add a metric for failing downloads
 2020-04-03 14:03:16 +02:00
Jakob Ackermann 3b10f80adb Merge pull request #156 from overleaf/jpa-logger-1-9-1 
[misc] bump logger-sharelatex to 1.9.1
 2020-04-03 14:03:04 +02:00
Jakob Ackermann 101c4ea4dc [misc] use a directory in /tmp for temporary data 2020-04-03 12:36:32 +02:00
Jakob Ackermann b9deec1095 [misc] narrow down write access/ownership for the run-time user 2020-04-03 12:36:32 +02:00
Jakob Ackermann 3309adf2ad [misc] move the sqlite database into a db/ directory 2020-04-03 12:36:31 +02:00
Jakob Ackermann 5b1656b884 [misc] drop debug output and log docker version on stderr 2020-04-03 12:36:31 +02:00
Jakob Ackermann 6777432569 [misc] add a metric for failing downloads 2020-04-03 12:35:40 +02:00
Jakob Ackermann 697e1466dc Merge pull request #160 from overleaf/jpa-build-scripts-2-1-0 
[misc] bump the build-scripts to version 2.1.0
 2020-04-03 12:33:06 +02:00
Jakob Ackermann 1a09965503 Merge pull request #2719 from overleaf/jpa-custom-wsurl-for-beta-users 
[frontend] ConnectionManager: add support for ie11

GitOrigin-RevId: ab1280f8da68fa87c266463e617b0a84a48065ce
 2020-04-03 03:20:58 +00:00
Jakob Ackermann d6fa1b1ca3 Merge pull request #2718 from overleaf/jpa-custom-wsurl-for-beta-users 
[frontend] ConnectionManager: fix dumb string concat in socket.io v0

GitOrigin-RevId: a1c472623e07b020e498d621e9f3241f8afc44f6
 2020-04-03 03:20:53 +00:00
Jakob Ackermann 155b22caf9 Merge pull request #2713 from overleaf/jpa-custom-wsurl-for-beta-users 
[misc] custom wsUrl for beta users

GitOrigin-RevId: e731ae7245e9c5586ae82cbc4c6716a74e56f2c9
 2020-04-03 03:20:47 +00:00
Jakob Ackermann ea435c88ea Merge pull request #31 from overleaf/jpa-fix-unit-tests 
[misc] fix unit tests
 2020-04-01 16:58:29 +02:00
Jakob Ackermann bdbbe1d596 Merge pull request #2687 from overleaf/jpa-fix-failing-reconnect-caused-by-backend 
[frontend] ConnectionManager: fix failing reconnects (caused by backend)

GitOrigin-RevId: f66e45fcb22849c2057b24eabde319ae7781ec51
 2020-03-31 03:18:36 +00:00
Jakob Ackermann 4d67acc7f5 Merge pull request #2686 from overleaf/jpa-fix-parallel-reconnect 
[frontend] ConnectionManager: fix parallel reconnect

GitOrigin-RevId: 38619b30dc38fd2732b77c0a785c8ab2857c81ff
 2020-03-31 03:18:32 +00:00
Jakob Ackermann 2ce244d662 Merge pull request #2685 from overleaf/jpa-fix-out-of-sync-open-doc 
[frontend] EditorManager: out-of-sync: wait for joinProject to complete

GitOrigin-RevId: d3006f2ad36c6094815aa892e65d2cf8c8243472
 2020-03-31 03:18:27 +00:00
Jakob Ackermann 5dabba11c6 Merge pull request #130 from overleaf/jpa-pub-sub-metrics 
[misc] track redis pub/sub payload sizes on publish
 2020-03-30 12:26:28 +02:00
Jakob Ackermann 56628a16c6 [misc] track redis pub/sub payload sizes on publish 2020-03-30 11:31:44 +02:00
Jakob Ackermann 17c2add0cf [misc] track redis pub/sub payload sizes on publish 2020-03-30 11:31:43 +02:00
Jakob Ackermann a9b8e9be3b [misc] upgrade metrics-sharelatex to 2.6.2 2020-03-30 10:47:01 +02:00
Jakob Ackermann 4a19f022fc [misc] keep up with the error signature of dockerode/docker-modem 
https://github.com/apocas/docker-modem/blob/v2.1.1/lib/modem.js#L296
 2020-03-27 10:39:45 +01:00
Jakob Ackermann f7c8d60c74 Merge pull request #126 from overleaf/jpa-logger-1-9-1 
[misc] bump logger-sharelatex to 1.9.1
 2020-03-26 13:03:48 +01:00
Jakob Ackermann eb909a4a21 Merge pull request #42 from overleaf/jpa-logger-1-9-1 
[misc] bump logger-sharelatex to 1.9.1
 2020-03-26 13:03:19 +01:00
Jakob Ackermann c72633a162 Merge pull request #120 from overleaf/jpa-limit-update-size 
[misc] limit update size
 2020-03-24 17:04:05 +01:00
Jakob Ackermann 69569e3571 [misc] config: add headroom for JSON serialization in maxUpdateSize 2020-03-24 16:21:29 +01:00
Jakob Ackermann af53d3b603 [misc] skip duplicate JSON serialization for size check 2020-03-24 11:22:28 +01:00
Jakob Ackermann cb675d38c2 [misc] SafeJsonParse: align the size limit with the frontend->rt limit 
frontend -> real-time and doc-updater -> real-time should be in sync.
Otherwise we can send a payload to doc-updater, but can not receive the
 confirmation of it -- and the client will send it again in a loop.

Also log the size of the payload.
 2020-03-24 09:14:15 +01:00
Jakob Ackermann 15244a54be [misc] WebsocketController: limit the update size to 7mb 
bail out early on -- especially do not push the update into redis for
 doc-updater to discard it.

Confirm the update silently, otherwise the frontend will send it again.
Broadcast a 'otUpdateError' message and disconnect the client, like
 doc-updater would do.
 2020-03-24 09:12:12 +01:00
Jakob Ackermann ac5feb4a77 [misc] bump logger-sharelatex to 1.9.1 2020-03-23 16:18:07 +01:00
Jakob Ackermann db30170228 [misc] bump logger-sharelatex to 1.9.1 2020-03-23 16:18:07 +01:00
Jakob Ackermann 2ce3c8f97e [misc] bump logger-sharelatex to 1.9.1 2020-03-23 16:18:07 +01:00
Jakob Ackermann b2d1718a2e [misc] bump logger-sharelatex to 1.9.1 2020-03-23 16:18:05 +01:00
Jakob Ackermann 14522a1f4b Merge pull request #17 from overleaf/jpa-fix-decaff-cleanup 
[misc] fix decaff cleanup for reportedToSentry
 2020-03-23 13:10:00 +01:00
Jakob Ackermann 2137c7a42c [misc] version bump to 1.9.1 2020-03-23 13:06:32 +01:00
Jakob Ackermann 7321e80b1e Merge pull request #119 from overleaf/jpa-delay-shutdown 
[misc] let proxys observe an upcoming shutdown before starting to drain
 2020-03-23 12:56:05 +01:00
Jakob Ackermann b049331777 [misc] fix decaff cleanup for reportedToSentry 
Previously it would bailout of the loop after processing ANY attribute.
REF: 6fe4dcbf32720282821d42140ab92593866e0772
 2020-03-23 10:48:55 +01:00
Jakob Ackermann d320c2d5f3 [misc] let proxys observe an upcoming shutdown before starting to drain 
Otherwise clients may be routed to the same pod upon reconnecting.
 2020-03-17 16:41:48 +01:00
Jakob Ackermann cb3508223f Merge pull request #78 from overleaf/jpa-skip-tests 
[misc] make: skip tests for the build and package target
 2020-03-11 11:55:56 +01:00
Jakob Ackermann 090e58a953 [misc] add documentation on how to run commands from the dev-environment 2020-03-11 11:46:46 +01:00
Jakob Ackermann f98212e96b [misc] make: skip tests for the build and package target 
http://maven.apache.org/plugins-archives/maven-surefire-plugin-2.12.4/examples/skipping-test.html
 2020-03-10 15:35:50 +01:00
Jakob Ackermann 2b15729658 [misc] promisify FileHandler and remove dependency on async 
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-03-02 18:23:17 +01:00
Jakob Ackermann ada6ac0764 [misc] revert eslint rules 2020-03-02 18:12:55 +01:00
Jakob Ackermann 1276fb426a [misc] bump buildscript version to 1.3.6 2020-03-02 17:50:45 +01:00
Jakob Ackermann 4b7c6fafd2 [misc] drop unused dependency pngcrush 2020-03-02 17:11:56 +01:00
Jakob Ackermann 3c61e53918 [misc] move mocha to the devDependencies 2020-03-02 17:11:39 +01:00
Jakob Ackermann 6f27f7a1bf [misc] drop the /heapdump route and related heapdump package 2020-03-02 17:09:35 +01:00
Jakob Ackermann 32557ab1d7 [SafeExec] replace _.once with lodash.once 2020-03-02 17:08:42 +01:00
Jakob Ackermann 0db4a17a14 [HealthCheckController] use fs.copyFile instead of fs-extra.copy 
the fs-extra method has a HUGE overhead of JS code and also syscalls for
 no particular benefit in this case: just copy the tiny.pdf file.

Here is an overview of the major operations: paths are relative to
  https://github.com/jprichardson/node-fs-extra/blob/1.0.0

We start in /lib/copy/copy.js
- sys: check that the source file exists
- sys: check that the source file has an existing parent directory?!
Continue in /lib/copy/ncp.js
- sys: more stat calls on both source and dest to determine permissions
- read/write streams to pipe the file content through the process
- sys: chmod on the destination to match the source permissions

What we actually need is a call to the binding and let node/the os
 figure out the best way to copy the contents.

Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-03-02 17:02:35 +01:00
Jakob Ackermann b4e3d7de62 [config] add support for third party s3 which uses path-style buckets 
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-02-28 12:48:52 +01:00
Jakob Ackermann 2b9d0868c5 [misc] test/acceptance: retrieve ingress metrics just before using it 
The upload request can bump the ingress metric.
The content hash validation might require a full download
in case the ETag field of the upload response is not a md5 sum.
 2020-02-28 12:27:06 +01:00
Jakob Ackermann 516102e6fb [misc] test/acceptance: do not hard code fake credentials 2020-02-28 12:26:46 +01:00
Jakob Ackermann 847f124d7b [misc] test/acceptance: skip the shutdown in case we did not start yet 
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-02-28 12:26:24 +01:00
Jakob Ackermann 6589aa6ae0 [misc] test/acceptance: harden the startup check for s3 
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-02-28 12:26:24 +01:00
Jakob Ackermann 94b8a7f891 [misc] test/unit: KeybuilderTests: use a unique suite label 2020-02-28 12:26:23 +01:00
Jakob Ackermann 54e3b57772 [misc] test/unit: add missing globals that are lazy loaded 2020-02-28 12:26:23 +01:00
Jakob Ackermann 7920570dd8 [misc] test/unit: add missing require stubs for metrics and settings 2020-02-28 12:26:22 +01:00
Jakob Ackermann 6155c8a097 [misc] cleanup unused dependency on mongo and redis 2020-02-25 14:45:09 +01:00