mirror of
https://github.com/yu-i-i/overleaf-cep.git
synced 2026-11-12 13:13:45 +01:00
[editor-core] extract more shared utils (#36954)
* [editor-core] share the blob hash and the doc-vs-file rule The git blob hash and the rule that decides whether a file is a doc or a binary file both have to reach the same answer in every service that writes to history, so they move into overleaf-editor-core rather than being spelled out again on each side. blob_utils takes over lib/blob_string_length.js: hashing and the string-length rule answered two halves of the same question -- what history needs to know about content before it can store it -- and each carried the same warning about staying off index.js, which is bundled for the browser. One module states it once, and blobForFile joins them, describing content on disk as the blob it amounts to. The hashing functions are flat rather than a namespace of their own: in a module that also holds the string-length rule, a bare fromStream says nothing about what it makes. history-v1's blob store and recover_zip follow the module to its new name. file_type_detector holds the doc-vs-file rule, Node-only and deep-imported for the same reason. It takes textExtensions, editableFilenames and maxDocLength as configuration rather than reading settings: the limit is web's 2 MiB and not editor-core's own 3 MiB, and the surrogate check covers both halves of a pair where editor-core's covers the high half. buildSetContentOperations' test spelled the blob header out for itself, with the separator as a literal NUL byte, which had git reading the file as binary. It hashes through blobHashFromString now, so it does not spell the header at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [editor-core] name the client a change came from on the origin itself `EditorOrigin` carried an `editorId` that only an editor change could have. What the id is for -- recognising a change again, so history does not apply a resend twice and the client does not apply its own change back -- is a property of the client that submitted it rather than of what it did, so it moves onto `Origin` as an optional `historyClientId` and `EditorOrigin` goes. - `Origin` takes `(kind, historyClientId)` and answers `getHistoryClientId()`; `dropHistoryClientId()` is what the chunk store uses to strip the id from all but a client's latest change, which now leaves the rest of the origin alone -- a restore that a client submitted stays a restore. - `toRaw` is built a layer at a time: the base writes the kind and the id, a subclass spreads that and adds the fields it owns. - The named kinds are their own schemas and discriminate on `kind`, so a payload claiming one is checked against that shape and the error says what it got wrong. The catch-all cannot join them -- its kind is any string -- so it sits alongside and is refined to refuse the named kinds, which it used to swallow: `{kind: 'restore'}` validated as a kind and nothing else, then read back as a restore that recorded nothing about what it restored. - `lib/origin/*.js` gains `// @ts-check`. Without it the annotations in there were decorative: the lazily required subclasses were `any`, so nothing handed to them was checked, and `fromRaw` narrows with `in` to drop the catch-all variant, which a kind check cannot. The callers that stamp the id -- real-time's handshake flag, web's applyChange body -- come with the rest of the branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [editor-core] recognise a change a writer that is not an editor placed The counterpart of `isSameEditorChange` for writers that are not an editor client: an integration that commits a batch on a user's behalf has to recognise that batch coming back, because a lost response does not say whether the commit landed and resubmitting one that did applies its removes twice. It compares the same three fields, and for the same reason -- a rebase rewrites a change's operations but leaves its origin, authors and timestamp alone -- with the timestamp compared as a point in time rather than as a string, so that two spellings of the same instant cannot read as different changes. github-sync is the first caller, and comes with the rest of the branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [editor-core] load an editable file's content, or say it has none Only an editable file has content: any other kind is bytes addressed by hash, and hollow data records the size of content it does not have. Both answer `getContent()` with null, so every caller that wants the content of a doc loaded it, checked the data's class, and reached for its own error when the check failed. `File#loadEager` does that once and answers the data that always has the content. `buildSetContentOperations` takes a `ReadWriteBlobStore` -- everything but looking a blob up by hash, which is all it takes to build operations over content -- so a caller holding a store that cannot answer `getBlob` can still use it. Its metadata argument is `FileMetadata` rather than `Object`, which is what the schema already says a file's metadata is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [editor-core] say that a pair of operations that do not fit is unprocessable compose and transform threw a bare Error when the two operations do not describe the same content. Everything that answers a client for these -- history-v1's import routes, the git-bridge push, project-history's resync -- decides on UnprocessableError, so a bare Error reached them as an internal failure: a 500, which reads as worth retrying, for a pair that fails the same way every time. A client that resubmits on a 500 resubmits forever. The guards that cannot be reached through the API keep their bare Error: a TextOperation composed with something that is not one, and the branch whose message says it should not happen. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * [history-v1] hash blobs with the rule editor-core holds storage/lib/blob_hash.js was a bluebird-era implementation of the same git blob hash editor-core holds, exported from the storage barrel and used across the storage layer, the api and the tests. All of it reads the shared rule now, and the test that existed only to check the copy is gone -- editor-core's suite covers the same non-ASCII case. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> GitOrigin-RevId: 5a0c1e9a06170a0164c01a26a7bb2d205726d531
This commit is contained in:
Jakob Ackermann
committed by
Copybot
Copybot
co-authored by
Claude Opus 5
Claude Opus 5
parent
3f171d500f
commit
04871d6049
@@ -13,10 +13,10 @@ const { parseReq } = require('@overleaf/validation-tools')
|
||||
|
||||
const logger = require('@overleaf/logger')
|
||||
const { Chunk, ChunkResponse, Blob } = require('overleaf-editor-core')
|
||||
const { blobHashFromFile } = require('overleaf-editor-core/lib/blob_utils')
|
||||
const {
|
||||
BlobStore,
|
||||
BatchBlobStore,
|
||||
blobHash,
|
||||
chunkStore,
|
||||
redisBuffer,
|
||||
HashCheckBlobStore,
|
||||
@@ -392,7 +392,7 @@ async function createProjectBlob(req, res, next) {
|
||||
)
|
||||
return render.requestEntityTooLarge(res)
|
||||
}
|
||||
const hash = await blobHash.fromFile(tmpPath)
|
||||
const hash = await blobHashFromFile(tmpPath)
|
||||
if (hash !== expectedHash) {
|
||||
logger.warn({ projectId, hash, expectedHash }, 'Hash mismatch')
|
||||
return render.conflict(res, 'File hash mismatch')
|
||||
|
||||
@@ -1,5 +1,4 @@
|
||||
exports.BatchBlobStore = require('./lib/batch_blob_store')
|
||||
exports.blobHash = require('./lib/blob_hash')
|
||||
exports.HashCheckBlobStore = require('./lib/hash_check_blob_store')
|
||||
exports.chunkStore = require('./lib/chunk_store')
|
||||
exports.redisBuffer = require('./lib/chunk_store/redis')
|
||||
|
||||
@@ -8,7 +8,7 @@ import {
|
||||
} from './backupPersistor.mjs'
|
||||
import { Blob, Chunk, History } from 'overleaf-editor-core'
|
||||
import { BlobStore, GLOBAL_BLOBS, makeProjectKey } from './blob_store/index.js'
|
||||
import blobHash from './blob_hash.js'
|
||||
import { blobHashFromStream } from 'overleaf-editor-core/lib/blob_utils.js'
|
||||
import { NotFoundError } from '@overleaf/object-persistor/src/Errors.js'
|
||||
import logger from '@overleaf/logger'
|
||||
import path from 'node:path'
|
||||
@@ -78,7 +78,7 @@ export async function verifyBlobs(historyId, hashes, projectCache) {
|
||||
}
|
||||
throw err
|
||||
}
|
||||
const backupHash = await blobHash.fromStream(blob.getByteLength(), stream)
|
||||
const backupHash = await blobHashFromStream(blob.getByteLength(), stream)
|
||||
if (backupHash !== hash) {
|
||||
throw new BackupCorruptedInvalidBlobError(
|
||||
'hash mismatch for backed up blob',
|
||||
|
||||
@@ -1,80 +0,0 @@
|
||||
/** @module */
|
||||
'use strict'
|
||||
|
||||
const BPromise = require('bluebird')
|
||||
const fs = BPromise.promisifyAll(require('node:fs'))
|
||||
const crypto = require('node:crypto')
|
||||
const { pipeline } = require('node:stream')
|
||||
const assert = require('./assert')
|
||||
|
||||
function getGitBlobHeader(byteLength) {
|
||||
return 'blob ' + byteLength + '\x00'
|
||||
}
|
||||
|
||||
function getBlobHash(byteLength) {
|
||||
const hash = crypto.createHash('sha1')
|
||||
hash.setEncoding('hex')
|
||||
hash.update(getGitBlobHeader(byteLength))
|
||||
return hash
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute the git blob hash for a blob from a readable stream of its content.
|
||||
*
|
||||
* @function
|
||||
* @param {number} byteLength
|
||||
* @param {stream.Readable} stream
|
||||
* @return {Promise.<string>} hexadecimal SHA-1 hash
|
||||
*/
|
||||
exports.fromStream = BPromise.method(
|
||||
function blobHashFromStream(byteLength, stream) {
|
||||
assert.integer(byteLength, 'blobHash: bad byteLength')
|
||||
assert.object(stream, 'blobHash: bad stream')
|
||||
|
||||
const hash = getBlobHash(byteLength)
|
||||
return new BPromise(function (resolve, reject) {
|
||||
pipeline(stream, hash, function (err) {
|
||||
if (err) {
|
||||
reject(err)
|
||||
} else {
|
||||
hash.end()
|
||||
resolve(hash.read())
|
||||
}
|
||||
})
|
||||
})
|
||||
}
|
||||
)
|
||||
|
||||
/**
|
||||
* Compute the git blob hash for a blob with the given string content.
|
||||
*
|
||||
* @param {string} string
|
||||
* @return {string} hexadecimal SHA-1 hash
|
||||
*/
|
||||
exports.fromString = function blobHashFromString(string) {
|
||||
assert.string(string, 'blobHash: bad string')
|
||||
const hash = getBlobHash(Buffer.byteLength(string))
|
||||
hash.update(string, 'utf8')
|
||||
hash.end()
|
||||
return hash.read()
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute the git blob hash for the content of a file
|
||||
*
|
||||
* @param {string} filePath
|
||||
* @return {Promise<string>} hexadecimal SHA-1 hash
|
||||
*/
|
||||
exports.fromFile = function blobHashFromFile(pathname) {
|
||||
assert.string(pathname, 'blobHash: bad pathname')
|
||||
|
||||
function getByteLengthOfFile() {
|
||||
return fs.statAsync(pathname).then(stat => stat.size)
|
||||
}
|
||||
|
||||
const fromStream = this.fromStream
|
||||
return getByteLengthOfFile(pathname).then(function (byteLength) {
|
||||
const stream = fs.createReadStream(pathname)
|
||||
return fromStream(byteLength, stream)
|
||||
})
|
||||
}
|
||||
@@ -6,14 +6,15 @@ const { ReadableString } = require('@overleaf/stream-utils')
|
||||
|
||||
const core = require('overleaf-editor-core')
|
||||
const {
|
||||
blobHashFromStream,
|
||||
blobHashFromString,
|
||||
getStringLengthOfFile,
|
||||
} = require('overleaf-editor-core/lib/blob_string_length')
|
||||
} = require('overleaf-editor-core/lib/blob_utils')
|
||||
const objectPersistor = require('@overleaf/object-persistor')
|
||||
const OError = require('@overleaf/o-error')
|
||||
const Blob = core.Blob
|
||||
|
||||
const assert = require('../assert')
|
||||
const blobHash = require('../blob_hash')
|
||||
const mongodb = require('../mongodb')
|
||||
const persistor = require('../persistor')
|
||||
const projectKey = require('@overleaf/object-persistor/src/ProjectKey.js')
|
||||
@@ -116,7 +117,7 @@ function getBackend(projectId) {
|
||||
|
||||
async function makeBlobForFile(pathname) {
|
||||
const { size: byteLength } = await fs.promises.stat(pathname)
|
||||
const hash = await blobHash.fromStream(
|
||||
const hash = await blobHashFromStream(
|
||||
byteLength,
|
||||
fs.createReadStream(pathname)
|
||||
)
|
||||
@@ -221,7 +222,7 @@ class BlobStore extends core.BlobStoreBase {
|
||||
*/
|
||||
async putString(string) {
|
||||
assert.string(string, 'bad string')
|
||||
const hash = blobHash.fromString(string)
|
||||
const hash = blobHashFromString(string)
|
||||
|
||||
const existingBlob = await this._findBlobBeforeInsert(hash)
|
||||
if (existingBlob != null) {
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
const { Blob, BlobStoreBase } = require('overleaf-editor-core')
|
||||
const blobHash = require('./blob_hash')
|
||||
const { blobHashFromString } = require('overleaf-editor-core/lib/blob_utils')
|
||||
|
||||
// We want to simulate applying all of the operations so we can return the
|
||||
// resulting hashes to the caller for them to check. To do this, we need to be
|
||||
@@ -34,7 +34,7 @@ class HashCheckBlobStore extends BlobStoreBase {
|
||||
*/
|
||||
async putString(string) {
|
||||
return new Blob(
|
||||
blobHash.fromString(string),
|
||||
blobHashFromString(string),
|
||||
Buffer.byteLength(string),
|
||||
string.length
|
||||
)
|
||||
|
||||
@@ -51,7 +51,7 @@ import {
|
||||
} from '@overleaf/mongo-utils/batchedUpdate.js'
|
||||
import { createGunzip } from 'node:zlib'
|
||||
import { text } from 'node:stream/consumers'
|
||||
import { fromStream as blobHashFromStream } from '../lib/blob_hash.js'
|
||||
import { blobHashFromStream } from 'overleaf-editor-core/lib/blob_utils.js'
|
||||
import { NotFoundError } from '@overleaf/object-persistor/src/Errors.js'
|
||||
|
||||
// Create a singleton promise that loads global blobs once
|
||||
|
||||
@@ -42,9 +42,7 @@ function createStorage() {
|
||||
}
|
||||
|
||||
const core = require('overleaf-editor-core')
|
||||
const {
|
||||
getStringLengthOfFile,
|
||||
} = require('overleaf-editor-core/lib/blob_string_length')
|
||||
const { getStringLengthOfFile } = require('overleaf-editor-core/lib/blob_utils')
|
||||
const projectKey = require('@overleaf/object-persistor/src/ProjectKey.js')
|
||||
const streams = require('../lib/streams')
|
||||
|
||||
|
||||
@@ -20,7 +20,7 @@ const Operation = core.Operation
|
||||
const Snapshot = core.Snapshot
|
||||
const TextOperation = core.TextOperation
|
||||
|
||||
const blobHash = require('../../../../storage').blobHash
|
||||
const { blobHashFromString } = require('overleaf-editor-core/lib/blob_utils')
|
||||
|
||||
describe('overleaf ot', function () {
|
||||
beforeEach(cleanup.everything)
|
||||
@@ -236,7 +236,7 @@ describe('overleaf ot', function () {
|
||||
testFiles.GRAPH_PNG_HASH
|
||||
)
|
||||
expect(snapshot.getFile('main.tex').getHash()).to.equal(
|
||||
blobHash.fromString('hello')
|
||||
blobHashFromString('hello')
|
||||
)
|
||||
return projectId
|
||||
})
|
||||
@@ -283,7 +283,7 @@ describe('overleaf ot', function () {
|
||||
const snapshot = Snapshot.fromRaw(response.obj)
|
||||
expect(snapshot.countFiles()).to.equal(3)
|
||||
expect(snapshot.getFile('main.tex').getHash()).to.equal(
|
||||
blobHash.fromString('hi world')
|
||||
blobHashFromString('hi world')
|
||||
)
|
||||
return projectId
|
||||
})
|
||||
@@ -321,7 +321,7 @@ describe('overleaf ot', function () {
|
||||
const snapshot = Snapshot.fromRaw(response.obj)
|
||||
expect(snapshot.countFiles()).to.equal(3)
|
||||
expect(snapshot.getFile('intro.tex').getHash()).to.equal(
|
||||
blobHash.fromString('hi world')
|
||||
blobHashFromString('hi world')
|
||||
)
|
||||
return projectId
|
||||
})
|
||||
|
||||
@@ -12,7 +12,7 @@ const Change = core.Change
|
||||
const File = core.File
|
||||
const Operation = core.Operation
|
||||
|
||||
const blobHash = require('../../../../storage/lib/blob_hash')
|
||||
const { blobHashFromString } = require('overleaf-editor-core/lib/blob_utils')
|
||||
const persistChanges = require('../../../../storage/lib/persist_changes')
|
||||
|
||||
describe('project hashed content', function () {
|
||||
@@ -72,7 +72,7 @@ describe('project hashed content', function () {
|
||||
const file = response.obj.files['tracked.tex']
|
||||
// The content is hashed as it stands, tracked delete included, and the ranges
|
||||
// are hashed separately.
|
||||
expect(file.hash).to.equal(blobHash.fromString(content))
|
||||
expect(file.hash).to.equal(blobHashFromString(content))
|
||||
expect(file.rangesHash).to.match(/^[0-9a-f]{40}$/)
|
||||
})
|
||||
|
||||
@@ -92,7 +92,7 @@ describe('project hashed content', function () {
|
||||
|
||||
expect(response.status).to.equal(200)
|
||||
const file = response.obj.files['main.tex']
|
||||
expect(file.hash).to.equal(blobHash.fromString(content))
|
||||
expect(file.hash).to.equal(blobHashFromString(content))
|
||||
expect(file.rangesHash).to.be.undefined
|
||||
})
|
||||
})
|
||||
|
||||
@@ -22,7 +22,7 @@ const TextOperation = core.TextOperation
|
||||
const V2DocVersions = core.V2DocVersions
|
||||
|
||||
const knex = require('../../../../storage').knex
|
||||
const blobHash = require('../../../../storage/lib/blob_hash')
|
||||
const { blobHashFromString } = require('overleaf-editor-core/lib/blob_utils')
|
||||
|
||||
describe('history import', function () {
|
||||
beforeEach(cleanup.everything)
|
||||
@@ -537,7 +537,7 @@ describe('history import', function () {
|
||||
},
|
||||
],
|
||||
})
|
||||
const testRangesHash = blobHash.fromString(rangesContent)
|
||||
const testRangesHash = blobHashFromString(rangesContent)
|
||||
const testFile = File.fromHash(testFiles.HELLO_TXT_HASH, testRangesHash)
|
||||
|
||||
const [contentResponse, rangesResponse] = await Promise.all([
|
||||
|
||||
@@ -16,8 +16,11 @@ const {
|
||||
BlobStore,
|
||||
persistChanges,
|
||||
redisBuffer,
|
||||
blobHash,
|
||||
} = require('../../../../storage')
|
||||
const {
|
||||
blobHashFromStream,
|
||||
blobHashFromString,
|
||||
} = require('overleaf-editor-core/lib/blob_utils')
|
||||
|
||||
const { expectHttpError } = require('./support/expect_response')
|
||||
const testServer = require('./support/test_server')
|
||||
@@ -453,7 +456,7 @@ describe('project controller', function () {
|
||||
|
||||
for (let i = 0; i < nTextBlobs; i++) {
|
||||
const content = `text blob ${i}`
|
||||
const hash = blobHash.fromString(content)
|
||||
const hash = blobHashFromString(content)
|
||||
blobHashes.push(hash)
|
||||
expectedTextBytes += content.length
|
||||
const res = await fetch(
|
||||
@@ -469,7 +472,7 @@ describe('project controller', function () {
|
||||
|
||||
for (let i = 0; i < nBinaryBlobs; i++) {
|
||||
const content = Buffer.from([0, i, i + 1, i + 2])
|
||||
const hash = await blobHash.fromStream(
|
||||
const hash = await blobHashFromStream(
|
||||
content.length,
|
||||
Readable.from(content)
|
||||
)
|
||||
|
||||
@@ -13,7 +13,7 @@ import { ObjectId } from 'mongodb'
|
||||
import testFiles from './support/test_files.js'
|
||||
import { BlobStore } from '../../../../storage/lib/blob_store/index.js'
|
||||
import fs from 'node:fs'
|
||||
import blobHash from '../../../../storage/lib/blob_hash.js'
|
||||
import { blobHashFromString } from 'overleaf-editor-core/lib/blob_utils.js'
|
||||
|
||||
const scenarios = [
|
||||
{
|
||||
@@ -197,7 +197,7 @@ for (const scenario of scenarios) {
|
||||
expect(results[1].chunkRecord.endVersion).to.equal(20)
|
||||
expect(results[1].blobsToBackup).to.have.deep.members([
|
||||
{
|
||||
hash: blobHash.fromString('a'.repeat(7)),
|
||||
hash: blobHashFromString('a'.repeat(7)),
|
||||
byteLength: 7,
|
||||
stringLength: 7,
|
||||
},
|
||||
@@ -213,7 +213,7 @@ for (const scenario of scenarios) {
|
||||
expect(results[2].chunkRecord.endVersion).to.equal(24)
|
||||
expect(results[2].blobsToBackup).to.have.deep.members([
|
||||
{
|
||||
hash: blobHash.fromString('a'.repeat(16)),
|
||||
hash: blobHashFromString('a'.repeat(16)),
|
||||
byteLength: 16,
|
||||
stringLength: 16,
|
||||
},
|
||||
@@ -270,7 +270,7 @@ for (const scenario of scenarios) {
|
||||
{
|
||||
version: 11,
|
||||
blob: {
|
||||
hash: blobHash.fromString('a'.repeat(7)),
|
||||
hash: blobHashFromString('a'.repeat(7)),
|
||||
byteLength: 7,
|
||||
stringLength: 7,
|
||||
},
|
||||
@@ -291,7 +291,7 @@ for (const scenario of scenarios) {
|
||||
{
|
||||
version: 21,
|
||||
blob: {
|
||||
hash: blobHash.fromString('a'.repeat(16)),
|
||||
hash: blobHashFromString('a'.repeat(16)),
|
||||
byteLength: 16,
|
||||
stringLength: 16,
|
||||
},
|
||||
|
||||
@@ -1,15 +0,0 @@
|
||||
'use strict'
|
||||
|
||||
const { expect } = require('chai')
|
||||
const storage = require('../../../../storage')
|
||||
const blobHash = storage.blobHash
|
||||
|
||||
describe('blobHash', function () {
|
||||
it('can hash non-ASCII strings', function () {
|
||||
// checked with git hash-object
|
||||
const testString = 'å\n'
|
||||
const testHash = 'aad321caf77ca6c5ab09e6c638c237705f93b001'
|
||||
|
||||
expect(blobHash.fromString(testString)).to.equal(testHash)
|
||||
})
|
||||
})
|
||||
@@ -1,4 +1,5 @@
|
||||
const path = require('node:path')
|
||||
const { File } = require('overleaf-editor-core')
|
||||
|
||||
exports.path = function (pathname) {
|
||||
return path.join(__dirname, '..', 'files', pathname)
|
||||
@@ -19,7 +20,7 @@ exports.NON_BMP_TXT_BYTE_LENGTH = 57
|
||||
exports.NULL_CHARACTERS_TXT_HASH = '4227ca4e8736af63036e7457e2db376ddf7e5795'
|
||||
exports.NULL_CHARACTERS_TXT_BYTE_LENGTH = 3
|
||||
|
||||
exports.EMPTY_FILE_HASH = 'e69de29bb2d1d6434b8b29ae775ad8c2e48c5391'
|
||||
exports.EMPTY_FILE_HASH = File.EMPTY_FILE_HASH
|
||||
exports.EMPTY_FILE_BYTE_LENGTH = 0
|
||||
|
||||
// git hashes of some short strings for testing
|
||||
|
||||
Reference in New Issue
Block a user