1d356f93ec
added random up to 30 mins delay to the ttl of mongo objects
2017-01-26 13:07:54 +00:00
dadc548f81
Ignore comment updates
2017-01-12 10:04:50 +01:00
694be95e66
Swap start_ts and end_ts comparison to correctly break chunks after 5 minutes
2016-10-11 11:03:52 +01:00
a197c0d148
Merge pull request #18 from sharelatex/ja-split-on-deletes
...
Split update summary on big deletes
2016-10-10 08:46:46 +01:00
40ed6fee46
Split update summary on big deletes
2016-10-04 15:13:04 +01:00
92e36023e5
Merge pull request #17 from sharelatex/ja-consistency-tweaks
...
Retry rewind if doc and update versions don't match
2016-09-30 15:14:27 +01:00
f8865e616d
Update ensureIndices to reflect reality
2016-09-30 14:34:38 +01:00
3c8aeb1262
Log number of retries
2016-09-30 13:38:47 +01:00
a7f44bcd01
Add missing return on callback
2016-09-30 13:36:31 +01:00
bddd1fda7d
Retry rewind if doc and update versions don't match
2016-09-30 11:36:47 +01:00
cce93b95a5
Fetch updates before doc content when building a diff
2016-09-22 11:19:05 +01:00
05a048db9a
Add in a little more logging
2016-09-22 11:19:05 +01:00
dfe26262ec
Return a No-op if diff returns nothing
2016-09-16 11:50:44 +01:00
80375ae2dd
Run a diff against big delete - insert changes which are likely copy-pastes
2016-09-16 11:33:36 +01:00
6e5eadfa86
include a timeout on WebApiManager requests
2016-04-13 16:42:36 +01:00
a55b72871f
don't let s3 errors stop archive worker
2016-04-13 14:39:11 +01:00
b343be844e
added metrics to pack worker for archiving
2016-04-08 10:29:04 +01:00
76fe194815
add a metric for append-pack
2016-04-07 15:56:11 +01:00
78100e40c8
add missing metrics file
2016-04-07 15:33:20 +01:00
1a1fa8798d
log attempted update when throwing error in DiffGenerator
2016-04-07 15:16:50 +01:00
6db310bf6b
add insert/archive/unarchive metrics
2016-04-07 15:16:50 +01:00
d0e08039da
don't modify expiry for temporary packs
2016-04-07 15:16:50 +01:00
fd49601716
preserve existing history when user upgrades
2016-04-07 15:16:38 +01:00
e292de5eb0
fix to avoid ever appending permanent changes to expiring packs
2016-04-06 17:00:16 +01:00
8b7bdd345b
consider all packs for archiving
2016-04-06 15:17:31 +01:00
ef47337c78
remove additional fields
2016-04-06 15:17:20 +01:00
0b9a0730c0
mark temporary packs with a last_checked date in the far future
...
they do not need to be checked for archiving
2016-04-06 14:29:49 +01:00
08fc151eee
avoid unnecessary call to insert packs into index
2016-04-06 14:29:21 +01:00
719e0291aa
consider all packs for processing
...
to allow finalisation of old head packs
2016-04-06 14:27:44 +01:00
79baa99634
clean up logging
2016-04-06 14:26:54 +01:00
6ab75795a2
archive head packs after sufficient time
2016-04-06 13:30:09 +01:00
6e18d49736
support archiving from list of project_ids/doc_ids
2016-04-04 17:00:19 +01:00
31348141d8
increase logging for discarded updates and version mismatch
2016-03-24 11:55:29 +00:00
181cebecef
avoid call to fetch packs unnecessarily
2016-03-24 11:55:29 +00:00
98683de3ae
temporarily disable ttl behaviour
...
allow existing packs without temporary flag to expire
2016-03-24 11:38:59 +00:00
3f388fb0ac
only change ttl on cached packs, not temporary ones
...
temporary = without versioning feature enabled
cached = permanent versioned retrieved from s3
2016-03-24 11:38:09 +00:00
8d900013d9
record whether a pack is temporary in the pack itself
...
using the expiresAt field no longer determines if the pack is
temporary because archived packs have an expiresAt field added when
they are retrieved from s3
2016-03-24 11:02:58 +00:00
98738d1344
fix for acceptance test
2016-03-10 15:15:29 +00:00
f01bf99682
acceptance tests - work in progress
2016-03-09 16:56:49 +00:00
f6367e21b8
give separate error for archive in progress vs completed
2016-03-09 14:44:59 +00:00
7350ab531d
exclude already cached packs from archival
2016-03-09 14:44:59 +00:00
28b184e0ca
fix incorrect use of _.union (argument must be array)
2016-03-09 14:44:59 +00:00
8922b97bd7
avoid duplicate filling of UserInfo in getDocUpdates
2016-03-09 14:44:59 +00:00
7e6ea2793b
remove startup dependency on s3 settings
2016-03-09 13:28:02 +00:00
1419d20b1f
fix indentation
2016-03-04 15:43:32 +00:00
3175f6d3a6
handle case where index does not exist
2016-03-03 14:36:16 +00:00
e8b3fb5be6
added more logging to failed health checks
2016-03-03 10:50:55 +00:00
795f717bab
added index definitions
2016-03-01 11:38:23 +00:00
3d9dfeccc3
remove pack worker
...
remove the op-specific code
remove tests for ops, now only packing
remove unused packing code
work in progress
store index for completed packs only
support archiving and unarchiving of individual packs
remove support for archiving whole document history
split out ArchiveManager, IndexManager
remove old DocArchive code
remove docHistoryStats collection
comment about archiving
added method to look at index when last pack has been archived
added start of iterator for project results
use a proper iterator
added heap module
getting it working
increase pack size since bulk operations no longer needed
remove unused MongoAWSexternal
cleanup
added doc iterator
remove old query code
added missing files
cleanup
clean upclean up
started adding pack worker for archiving
work in progress
work in progress
getting pack worker working
updating worker
getting packworker working
added lock
use correct key name for track changes aws access
use correct key name for track changes aws access
always send back users array
fix up comparison of retrieved objects
handle op ids inside packs
log when s3 download completes
comments
cleanup, remove finalisation ideacleanup, remove finalisation idea
remove logging
2016-03-01 10:10:02 +00:00
a23ddf31c0
allow packing of temporary ops
2016-01-29 12:36:03 +00:00
Henry Oswald