mirror of
https://github.com/yu-i-i/overleaf-cep.git
synced 2026-11-12 13:13:45 +01:00
[editor-core] extract logic for determining root doc from snapshot (#37043)
* [editor-core] hold the one threshold that decides which store owns the file tree HISTORY_FILE_TREE_STAGE is five literals across the services that consume this library -- web backend, web editor, real-time, third-party-datastore, github-sync -- linked by comments saying which other copies they are aligned with. What the docstring here says is what goes wrong: this is the single thing that decides which storage is authoritative, a project crosses it once with no way back, and a copy that drifts sends a whole service's writes to the store nothing reads any more. Silently, and for good. overleaf-editor-core is where it goes: every one of the five already depends on it, the editor included, and the library that defines what a file tree is can say where one lives. The consumers adopt it separately; this is the definition they adopt. historyIsSourceOfTruth goes with it, and refuses a stage that is neither absent nor a number rather than reading it as low -- web's copy coerced, which turned a corrupt stage into a permanent mongo write. Absent stays "below the gate": a project that never migrated has no stage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [editor-core] hold which doc a project opens on History records it as `main: true` on the file, and nothing derives it: every writer that creates a project has to record it, or the project compiles nothing until the user picks a file. So the rule for choosing one, and the operations that record it, go where a file tree is defined rather than with any one writer. The rule is the one web has always applied, over a snapshot instead of a mongo tree: of the docs that declare a document class, the one nearest the project root, alphabetically among equals, so that the same files always give the same answer. A project that already records a root doc keeps it -- that is a user's choice and not something to re-derive. Whether a doc could be the one is asked of content rather than of a store to read it from: a writer creating a project has each doc's content in its hands as it stores it, so reading them back would be a second read of every one. The choice is then made from the pathnames alone. The extension list joins the others in text_file_defaults for the reason they are there: a writer holding a shorter copy leaves a project with nothing to compile. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [editor-core] say why a store built for a consumer does not need getBlob ReadWriteBlobStore is BlobStore without the lookup by hash, and the type said so without saying why a consumer can do without it. getBlob is reached only for a file stored as a bare hash, and one of those does not leave history-v1: chunk_store's lazyLoadHistoryFiles attaches blob metadata to every snapshot and change it serves, so a file arrives already knowing its own lengths. That is what the contract rests on, and a store that reimplemented the classification rule to answer the lookup anyway would be a second opinion on whether a file is editable. The reasoning was what was missing, rather than the type. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [editor-core] say what makes a doc a root doc candidate The sentence naming the two halves of the check had lost the first one, and read "named as a root doc may be named". It is the extension that is tested: one of DEFAULT_ROOT_DOC_EXTENSIONS, and a document class declared in the content. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [editor-core] hold the doc flags a file's metadata can carry in one place `main` is not the only flag a doc carries -- `mainBibliography` marks the file references are added to -- and both live in the same metadata, which SetFileMetadataOperation replaces wholesale rather than merges. So every writer that sets one has to write the others out again, and every reader that asks whether a file is a doc has to know the whole list. root_doc was spelling all three of those for itself against `main` alone: it tested the key by hand, refused a file whose metadata held anything else, and cleared the metadata to `{}` when moving the root doc elsewhere. Against a file that is also the main bibliography, the first two refuse a legitimate root doc and the third silently drops the bibliography. The helpers answer it once: which keys are doc flags, whether metadata holds only those, and what metadata with one flag changed looks like. root_doc uses them throughout, and the schema takes both flags on one file. Co-Authored-By: Mathias Jakobsen <mathias.jakobsen@overleaf.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [editor-core] ask the snapshot which doc carries a flag Finding the file a doc flag is set on is the same search whichever flag is being asked about, and it belongs to the snapshot rather than to the caller: it walks the snapshot's own pathnames, and it is the snapshot that knows a file's metadata without its content being loaded. root_doc's mainPathname is that search for `main`, and now delegates. It gains the check it was missing: a file whose metadata holds anything besides doc flags came from outside the editor, so a stale `main` on it does not make it the root doc -- which is already what setMainPathnameOperations refuses to write. Co-Authored-By: Mathias Jakobsen <mathias.jakobsen@overleaf.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [editor-core] drop mainPathname for the snapshot method behind it It became a second name for `snapshot.getPathnameWithDocFlag('main')` and nothing in the library was left calling it but chooseRootDoc, which asks the snapshot directly now. Its two tests go with it: Snapshot's own cover the same two answers, and more of them. Callers outside the library ask the snapshot too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Mathias Jakobsen <mathias.jakobsen@overleaf.com> GitOrigin-RevId: b3f503aa3e94ea8b6524d83f910fe33dc381c0a5
This commit is contained in:
committed by
Copybot
co-authored by
Claude Opus 5
Mathias Jakobsen
parent
0ec99c7e3e
commit
e039ad26c5
@@ -28,6 +28,15 @@ const RestoreFileOrigin = require('./lib/origin/restore_file_origin')
|
|||||||
const Origin = require('./lib/origin')
|
const Origin = require('./lib/origin')
|
||||||
const { EDITOR_ORIGIN_KIND } = require('./lib/origin')
|
const { EDITOR_ORIGIN_KIND } = require('./lib/origin')
|
||||||
const OtClient = require('./lib/ot_client')
|
const OtClient = require('./lib/ot_client')
|
||||||
|
const {
|
||||||
|
HISTORY_FILE_TREE_STAGE,
|
||||||
|
historyIsSourceOfTruth,
|
||||||
|
} = require('./lib/ot_migration_stages')
|
||||||
|
const {
|
||||||
|
chooseRootDoc,
|
||||||
|
isRootDocCandidate,
|
||||||
|
setMainPathnameOperations,
|
||||||
|
} = require('./lib/root_doc')
|
||||||
const rebaseChanges = require('./lib/rebase')
|
const rebaseChanges = require('./lib/rebase')
|
||||||
const {
|
const {
|
||||||
editorChangeIdentity,
|
editorChangeIdentity,
|
||||||
@@ -38,6 +47,12 @@ const {
|
|||||||
const TextOperation = require('./lib/operation/text_operation')
|
const TextOperation = require('./lib/operation/text_operation')
|
||||||
const EditOperation = require('./lib/operation/edit_operation')
|
const EditOperation = require('./lib/operation/edit_operation')
|
||||||
const safePathname = require('./lib/safe_pathname')
|
const safePathname = require('./lib/safe_pathname')
|
||||||
|
const {
|
||||||
|
DOCUMENT_METADATA_KEYS,
|
||||||
|
isDocumentMetadata,
|
||||||
|
hasDocumentMetadataFlag,
|
||||||
|
withDocumentMetadataFlag,
|
||||||
|
} = require('./lib/file_metadata')
|
||||||
const Snapshot = require('./lib/snapshot')
|
const Snapshot = require('./lib/snapshot')
|
||||||
const {
|
const {
|
||||||
DEFAULT_TEXT_EXTENSIONS,
|
DEFAULT_TEXT_EXTENSIONS,
|
||||||
@@ -98,6 +113,11 @@ exports.RestoreFileOrigin = RestoreFileOrigin
|
|||||||
exports.Origin = Origin
|
exports.Origin = Origin
|
||||||
exports.EDITOR_ORIGIN_KIND = EDITOR_ORIGIN_KIND
|
exports.EDITOR_ORIGIN_KIND = EDITOR_ORIGIN_KIND
|
||||||
exports.OtClient = OtClient
|
exports.OtClient = OtClient
|
||||||
|
exports.HISTORY_FILE_TREE_STAGE = HISTORY_FILE_TREE_STAGE
|
||||||
|
exports.historyIsSourceOfTruth = historyIsSourceOfTruth
|
||||||
|
exports.chooseRootDoc = chooseRootDoc
|
||||||
|
exports.isRootDocCandidate = isRootDocCandidate
|
||||||
|
exports.setMainPathnameOperations = setMainPathnameOperations
|
||||||
exports.rebaseChanges = rebaseChanges
|
exports.rebaseChanges = rebaseChanges
|
||||||
exports.editorChangeIdentity = editorChangeIdentity
|
exports.editorChangeIdentity = editorChangeIdentity
|
||||||
exports.editorChangeIdentityOf = editorChangeIdentityOf
|
exports.editorChangeIdentityOf = editorChangeIdentityOf
|
||||||
@@ -106,6 +126,10 @@ exports.isChangeFrom = isChangeFrom
|
|||||||
exports.TextOperation = TextOperation
|
exports.TextOperation = TextOperation
|
||||||
exports.EditOperation = EditOperation
|
exports.EditOperation = EditOperation
|
||||||
exports.safePathname = safePathname
|
exports.safePathname = safePathname
|
||||||
|
exports.DOCUMENT_METADATA_KEYS = DOCUMENT_METADATA_KEYS
|
||||||
|
exports.isDocumentMetadata = isDocumentMetadata
|
||||||
|
exports.hasDocumentMetadataFlag = hasDocumentMetadataFlag
|
||||||
|
exports.withDocumentMetadataFlag = withDocumentMetadataFlag
|
||||||
exports.Snapshot = Snapshot
|
exports.Snapshot = Snapshot
|
||||||
exports.util = util
|
exports.util = util
|
||||||
exports.V2DocVersions = V2DocVersions
|
exports.V2DocVersions = V2DocVersions
|
||||||
|
|||||||
@@ -0,0 +1,74 @@
|
|||||||
|
// @ts-check
|
||||||
|
'use strict'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @import { FileMetadata } from './types'
|
||||||
|
*/
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The metadata keys a doc can carry: `main` marks the project's root doc,
|
||||||
|
* `mainBibliography` the bibliography references are added to. Everything else a
|
||||||
|
* file's metadata holds records where it came from.
|
||||||
|
*/
|
||||||
|
const DOCUMENT_METADATA_KEYS = /** @type {const} */ ([
|
||||||
|
'main',
|
||||||
|
'mainBibliography',
|
||||||
|
])
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Whether metadata holds nothing but the keys a doc can carry, which is what
|
||||||
|
* makes a file a doc rather than a file.
|
||||||
|
*
|
||||||
|
* @param {FileMetadata} [metadata]
|
||||||
|
* @returns {boolean}
|
||||||
|
*/
|
||||||
|
function isDocumentMetadata(metadata) {
|
||||||
|
return Object.keys(metadata ?? {}).every(key =>
|
||||||
|
/** @type {readonly string[]} */ (DOCUMENT_METADATA_KEYS).includes(key)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Whether metadata carries a document flag.
|
||||||
|
*
|
||||||
|
* @param {FileMetadata | undefined} metadata
|
||||||
|
* @param {typeof DOCUMENT_METADATA_KEYS[number]} key
|
||||||
|
* @returns {boolean}
|
||||||
|
*/
|
||||||
|
function hasDocumentMetadataFlag(metadata, key) {
|
||||||
|
return Boolean(/** @type {Record<string, unknown>} */ (metadata ?? {})[key])
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Metadata with one document flag set or cleared, keeping the others.
|
||||||
|
*
|
||||||
|
* Metadata is replaced wholesale by SetFileMetadataOperation, so changing one
|
||||||
|
* flag means writing out the rest of them too. A cleared flag is left out rather
|
||||||
|
* than set to false, so a file carrying none has empty metadata.
|
||||||
|
*
|
||||||
|
* @param {FileMetadata} metadata
|
||||||
|
* @param {typeof DOCUMENT_METADATA_KEYS[number]} key
|
||||||
|
* @param {boolean} value
|
||||||
|
* @returns {FileMetadata}
|
||||||
|
*/
|
||||||
|
function withDocumentMetadataFlag(metadata, key, value) {
|
||||||
|
/** @type {FileMetadata} */
|
||||||
|
const next = {}
|
||||||
|
for (const documentKey of DOCUMENT_METADATA_KEYS) {
|
||||||
|
if (
|
||||||
|
documentKey === key
|
||||||
|
? value
|
||||||
|
: hasDocumentMetadataFlag(metadata, documentKey)
|
||||||
|
) {
|
||||||
|
next[documentKey] = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return next
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = {
|
||||||
|
DOCUMENT_METADATA_KEYS,
|
||||||
|
isDocumentMetadata,
|
||||||
|
hasDocumentMetadataFlag,
|
||||||
|
withDocumentMetadataFlag,
|
||||||
|
}
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
// @ts-check
|
||||||
|
'use strict'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Which storage a project's file tree lives in.
|
||||||
|
*
|
||||||
|
* A project migrates onto history-as-source-of-truth once, and there is no way
|
||||||
|
* back: above the gate, web's mongo rootFolder/docs/fileRefs are no longer
|
||||||
|
* authoritative and history-v1 holds the tree. Exactly one thing decides which
|
||||||
|
* storage is authoritative, and this is it.
|
||||||
|
*
|
||||||
|
* It lives here because every service that has to make that decision -- web
|
||||||
|
* backend and editor, real-time, third-party-datastore, github-sync -- already
|
||||||
|
* depends on this library, and because a copy per service is five things that
|
||||||
|
* have to agree: one of them drifting misroutes a whole service's writes into
|
||||||
|
* the store nothing reads any more, silently and with no way back.
|
||||||
|
*
|
||||||
|
* Deliberately a leaf: it imports nothing. A test that mocks a sibling of a
|
||||||
|
* module holding this constant must not be able to leave it undefined, because
|
||||||
|
* `undefined >= 11` is false and a silently disabled gate reads as "below the
|
||||||
|
* gate" for every project. Reference the constant, never the number.
|
||||||
|
*/
|
||||||
|
|
||||||
|
/** The stage at or above which history holds the project's file tree. */
|
||||||
|
const HISTORY_FILE_TREE_STAGE = 11
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Whether history holds the file tree of a project at this migration stage.
|
||||||
|
*
|
||||||
|
* A project that has never migrated has no stage at all, which is below the
|
||||||
|
* gate. Anything else that is not a number is corrupt rather than low, and
|
||||||
|
* answering "below the gate" for it would route the project's writes into mongo
|
||||||
|
* for good, so it is refused instead.
|
||||||
|
*
|
||||||
|
* @param {unknown} otMigrationStage as stored on `overleaf.history`
|
||||||
|
* @return {boolean}
|
||||||
|
* @throws {TypeError} where the stage is neither absent nor a number
|
||||||
|
*/
|
||||||
|
function historyIsSourceOfTruth(otMigrationStage) {
|
||||||
|
if (otMigrationStage == null) return false
|
||||||
|
if (
|
||||||
|
typeof otMigrationStage !== 'number' ||
|
||||||
|
!Number.isFinite(otMigrationStage)
|
||||||
|
) {
|
||||||
|
throw new TypeError(
|
||||||
|
`otMigrationStage is not a number: ${JSON.stringify(otMigrationStage)}`
|
||||||
|
)
|
||||||
|
}
|
||||||
|
return otMigrationStage >= HISTORY_FILE_TREE_STAGE
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { HISTORY_FILE_TREE_STAGE, historyIsSourceOfTruth }
|
||||||
@@ -0,0 +1,157 @@
|
|||||||
|
// @ts-check
|
||||||
|
'use strict'
|
||||||
|
|
||||||
|
const OError = require('@overleaf/o-error')
|
||||||
|
const {
|
||||||
|
hasDocumentMetadataFlag,
|
||||||
|
isDocumentMetadata,
|
||||||
|
withDocumentMetadataFlag,
|
||||||
|
} = require('./file_metadata')
|
||||||
|
const Operation = require('./operation')
|
||||||
|
const { DEFAULT_ROOT_DOC_EXTENSIONS } = require('./text_file_defaults')
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @import Snapshot from './snapshot'
|
||||||
|
* @import { FileMetadata } from './types'
|
||||||
|
*/
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Which doc a project opens on, and how history records that it is the one.
|
||||||
|
*
|
||||||
|
* History holds it as `main: true` on the file's metadata, and nothing derives it:
|
||||||
|
* every writer that creates a project has to record it, or the project opens on no
|
||||||
|
* doc at all. So the rule for choosing one, and the operations that write it, live
|
||||||
|
* here rather than with any one of them.
|
||||||
|
*/
|
||||||
|
|
||||||
|
/** How much of a doc is read looking for its document class. */
|
||||||
|
const MAX_CONTENT_TO_SCAN = 30 * 1000
|
||||||
|
|
||||||
|
const ROOT_DOC_EXTENSION = new RegExp(
|
||||||
|
`\\.(${DEFAULT_ROOT_DOC_EXTENSIONS.join('|')})$`,
|
||||||
|
'i'
|
||||||
|
)
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Whether this doc could be the one a project compiles from: carrying an extension a
|
||||||
|
* root doc may have, and declaring a document class.
|
||||||
|
*
|
||||||
|
* Content rather than a store to read it from: a writer creating a project has every
|
||||||
|
* doc's content in its hands as it stores it, and reading them back to answer this
|
||||||
|
* would be a second read of each.
|
||||||
|
*
|
||||||
|
* Only the start of a doc is read, and each line only from its beginning: a data file
|
||||||
|
* is a doc as far as history is concerned, and one 500KB line of it scanned for a
|
||||||
|
* pattern that can only match at a line start is what used to lock up a CPU.
|
||||||
|
*
|
||||||
|
* @param {string} pathname
|
||||||
|
* @param {string} content
|
||||||
|
* @return {boolean}
|
||||||
|
*/
|
||||||
|
function isRootDocCandidate(pathname, content) {
|
||||||
|
if (!ROOT_DOC_EXTENSION.test(pathname)) return false
|
||||||
|
return content
|
||||||
|
.substring(0, MAX_CONTENT_TO_SCAN)
|
||||||
|
.split('\n')
|
||||||
|
.some(line => /^\s*\\documentclass/.test(line))
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @param {Snapshot} snapshot
|
||||||
|
* @param {string} pathname
|
||||||
|
* @return {FileMetadata}
|
||||||
|
*/
|
||||||
|
function metadataOf(snapshot, pathname) {
|
||||||
|
return snapshot.getFile(pathname)?.getMetadata() ?? {}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Make the file at `pathname` the project's root doc.
|
||||||
|
*
|
||||||
|
* The change unsets `main` wherever it is now and sets it on the chosen file. Both are
|
||||||
|
* needed: metadata is replaced wholesale rather than merged, and nothing stops two
|
||||||
|
* files claiming it.
|
||||||
|
*
|
||||||
|
* A file carrying metadata that is not a doc flag came from outside the editor, and
|
||||||
|
* cannot also be the root doc: history's metadata shapes do not combine, so the change
|
||||||
|
* would have to throw that away to fit the doc one. Refused here rather than written.
|
||||||
|
*
|
||||||
|
* @param {Snapshot} snapshot
|
||||||
|
* @param {string} pathname
|
||||||
|
* @return {Operation[]} empty where history already records this file
|
||||||
|
*/
|
||||||
|
function setMainPathnameOperations(snapshot, pathname) {
|
||||||
|
if (!isDocumentMetadata(metadataOf(snapshot, pathname))) {
|
||||||
|
throw new OError('only a doc can be the root doc', { pathname })
|
||||||
|
}
|
||||||
|
|
||||||
|
const operations = []
|
||||||
|
for (const candidate of snapshot.getFilePathnames().sort()) {
|
||||||
|
const metadata = metadataOf(snapshot, candidate)
|
||||||
|
const main = candidate === pathname
|
||||||
|
if (hasDocumentMetadataFlag(metadata, 'main') === main) continue
|
||||||
|
operations.push(
|
||||||
|
Operation.setFileMetadata(
|
||||||
|
candidate,
|
||||||
|
withDocumentMetadataFlag(metadata, 'main', main)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
return operations
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The doc a project should open on, and the operations that record it as such.
|
||||||
|
*
|
||||||
|
* Of the docs that could be it, the one nearest the project root, alphabetically among
|
||||||
|
* equals: a main file that only includes the others sits beside the project rather
|
||||||
|
* than under it, and the rest is so that the same files always give the same answer.
|
||||||
|
*
|
||||||
|
* A project whose root doc history already records keeps it -- that is a user's choice
|
||||||
|
* and not something to re-derive.
|
||||||
|
*
|
||||||
|
* @param {Snapshot} snapshot
|
||||||
|
* @param {Iterable<string>} candidates pathnames of the docs that could be it, as
|
||||||
|
* `isRootDocCandidate` decides
|
||||||
|
* @return {{pathname: string, operations: Operation[]} | null} null where there is no
|
||||||
|
* candidate, and where one is already recorded
|
||||||
|
*/
|
||||||
|
function chooseRootDoc(snapshot, candidates) {
|
||||||
|
if (snapshot.getPathnameWithDocFlag('main') !== undefined) return null
|
||||||
|
|
||||||
|
// Only what the snapshot holds: a doc the caller could not write is not one the
|
||||||
|
// project has, and recording it would leave `main` on a pathname with no file.
|
||||||
|
const pathname = [...candidates]
|
||||||
|
.filter(candidate => snapshot.getFile(candidate) != null)
|
||||||
|
.sort(byDepthThenName)[0]
|
||||||
|
if (pathname === undefined) return null
|
||||||
|
return { pathname, operations: setMainPathnameOperations(snapshot, pathname) }
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @param {string} a
|
||||||
|
* @param {string} b
|
||||||
|
* @return {number}
|
||||||
|
*/
|
||||||
|
function byDepthThenName(a, b) {
|
||||||
|
const depth = countSlashes(a) - countSlashes(b)
|
||||||
|
return depth !== 0 ? depth : a < b ? -1 : a > b ? 1 : 0
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @param {string} pathname
|
||||||
|
* @return {number}
|
||||||
|
*/
|
||||||
|
function countSlashes(pathname) {
|
||||||
|
let count = 0
|
||||||
|
for (const character of pathname) {
|
||||||
|
if (character === '/') count++
|
||||||
|
}
|
||||||
|
return count
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = {
|
||||||
|
chooseRootDoc,
|
||||||
|
isRootDocCandidate,
|
||||||
|
setMainPathnameOperations,
|
||||||
|
}
|
||||||
@@ -166,9 +166,10 @@ const rawFileMetadata = z.union([
|
|||||||
// linked-file v2
|
// linked-file v2
|
||||||
rawLinkedFileData,
|
rawLinkedFileData,
|
||||||
|
|
||||||
// main-file v1
|
// main-file v1, and the doc flags the editor sets
|
||||||
z.strictObject({
|
z.strictObject({
|
||||||
main: z.boolean(),
|
main: z.boolean().optional(),
|
||||||
|
mainBibliography: z.boolean().optional(),
|
||||||
importedAt: z.iso.datetime().optional(),
|
importedAt: z.iso.datetime().optional(),
|
||||||
}),
|
}),
|
||||||
// legacy v1 linked file
|
// legacy v1 linked file
|
||||||
|
|||||||
@@ -6,6 +6,12 @@ const OError = require('@overleaf/o-error')
|
|||||||
|
|
||||||
const FileMap = require('./file_map')
|
const FileMap = require('./file_map')
|
||||||
const V2DocVersions = require('./v2_doc_versions')
|
const V2DocVersions = require('./v2_doc_versions')
|
||||||
|
const {
|
||||||
|
hasDocumentMetadataFlag,
|
||||||
|
isDocumentMetadata,
|
||||||
|
} = require('./file_metadata')
|
||||||
|
|
||||||
|
/** @import { DOCUMENT_METADATA_KEYS } from "./file_metadata" */
|
||||||
|
|
||||||
const FILE_LOAD_CONCURRENCY = 50
|
const FILE_LOAD_CONCURRENCY = 50
|
||||||
|
|
||||||
@@ -156,6 +162,27 @@ class Snapshot {
|
|||||||
return this.fileMap.getFile(pathname)
|
return this.fileMap.getFile(pathname)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The pathname of the doc carrying a document flag (`main`, `mainBibliography`),
|
||||||
|
* if this snapshot records one.
|
||||||
|
*
|
||||||
|
* Sorted, so that a project which somehow has two answers gives the same one
|
||||||
|
* every time it is asked.
|
||||||
|
*
|
||||||
|
* @param {typeof DOCUMENT_METADATA_KEYS[number]} key
|
||||||
|
* @return {string | undefined}
|
||||||
|
*/
|
||||||
|
getPathnameWithDocFlag(key) {
|
||||||
|
return this.getFilePathnames()
|
||||||
|
.sort()
|
||||||
|
.find(pathname => {
|
||||||
|
const metadata = this.getFile(pathname)?.getMetadata()
|
||||||
|
return (
|
||||||
|
hasDocumentMetadataFlag(metadata, key) && isDocumentMetadata(metadata)
|
||||||
|
)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Add the given file to the snapshot.
|
* Add the given file to the snapshot.
|
||||||
* @see FileMap#addFile
|
* @see FileMap#addFile
|
||||||
|
|||||||
@@ -70,6 +70,18 @@ const DEFAULT_TEXT_EXTENSIONS = Object.freeze([
|
|||||||
'inc',
|
'inc',
|
||||||
])
|
])
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Extensions a doc may have to be the one a project compiles from.
|
||||||
|
*
|
||||||
|
* A subset of the text extensions above: every one of these is editable, and most of
|
||||||
|
* the editable ones cannot be a root doc. Shared for the same reason as the list it is
|
||||||
|
* drawn from -- the answer has to be the same wherever a project is created, and a
|
||||||
|
* writer holding a shorter copy leaves a project with nothing to compile.
|
||||||
|
*
|
||||||
|
* @type {ReadonlyArray<string>}
|
||||||
|
*/
|
||||||
|
const DEFAULT_ROOT_DOC_EXTENSIONS = Object.freeze(['tex', 'Rtex', 'ltx', 'Rnw'])
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Whole file names, lower case, editable whatever their extension says.
|
* Whole file names, lower case, editable whatever their extension says.
|
||||||
*
|
*
|
||||||
@@ -82,4 +94,8 @@ const DEFAULT_EDITABLE_FILENAMES = Object.freeze([
|
|||||||
'gnumakefile',
|
'gnumakefile',
|
||||||
])
|
])
|
||||||
|
|
||||||
module.exports = { DEFAULT_TEXT_EXTENSIONS, DEFAULT_EDITABLE_FILENAMES }
|
module.exports = {
|
||||||
|
DEFAULT_TEXT_EXTENSIONS,
|
||||||
|
DEFAULT_EDITABLE_FILENAMES,
|
||||||
|
DEFAULT_ROOT_DOC_EXTENSIONS,
|
||||||
|
}
|
||||||
|
|||||||
@@ -17,6 +17,14 @@ export type ReadonlyBlobStore = Pick<BlobStore, 'getString' | 'getObject'>
|
|||||||
/**
|
/**
|
||||||
* Everything but looking a blob up by its hash: reading content and storing
|
* Everything but looking a blob up by its hash: reading content and storing
|
||||||
* content, which is all it takes to build operations over content.
|
* content, which is all it takes to build operations over content.
|
||||||
|
*
|
||||||
|
* Looking a blob up is only ever needed for a file stored as a bare hash
|
||||||
|
* (HashFileData), and one of those does not reach a consumer outside history-v1:
|
||||||
|
* every snapshot and change it serves has blob metadata attached first
|
||||||
|
* (chunk_store's lazyLoadHistoryFiles), so a file arrives already knowing its
|
||||||
|
* own lengths. A store built for a consumer therefore does not need `getBlob`,
|
||||||
|
* and one that reimplements the classification rule to answer it would be a
|
||||||
|
* second opinion on whether a file is editable.
|
||||||
*/
|
*/
|
||||||
export type ReadWriteBlobStore = Omit<BlobStore, 'getBlob'>
|
export type ReadWriteBlobStore = Omit<BlobStore, 'getBlob'>
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,73 @@
|
|||||||
|
// @ts-check
|
||||||
|
'use strict'
|
||||||
|
|
||||||
|
const { expect } = require('chai')
|
||||||
|
|
||||||
|
const {
|
||||||
|
isDocumentMetadata,
|
||||||
|
withDocumentMetadataFlag,
|
||||||
|
} = require('../../lib/file_metadata')
|
||||||
|
|
||||||
|
describe('file metadata', function () {
|
||||||
|
describe('isDocumentMetadata', function () {
|
||||||
|
it('accepts no metadata at all', function () {
|
||||||
|
expect(isDocumentMetadata()).to.equal(true)
|
||||||
|
expect(isDocumentMetadata({})).to.equal(true)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('accepts the doc flags', function () {
|
||||||
|
expect(isDocumentMetadata({ main: true })).to.equal(true)
|
||||||
|
expect(isDocumentMetadata({ mainBibliography: true })).to.equal(true)
|
||||||
|
expect(
|
||||||
|
isDocumentMetadata({ main: true, mainBibliography: true })
|
||||||
|
).to.equal(true)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('rejects metadata recording where the file came from', function () {
|
||||||
|
expect(
|
||||||
|
isDocumentMetadata({ importedAt: '2026-01-02T00:00:00.000Z' })
|
||||||
|
).to.equal(false)
|
||||||
|
expect(isDocumentMetadata({ main: true, provider: 'zotero' })).to.equal(
|
||||||
|
false
|
||||||
|
)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('withDocumentMetadataFlag', function () {
|
||||||
|
it('sets a flag on metadata that has none', function () {
|
||||||
|
expect(withDocumentMetadataFlag({}, 'mainBibliography', true)).to.eql({
|
||||||
|
mainBibliography: true,
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
it('keeps the other flag when setting one', function () {
|
||||||
|
expect(
|
||||||
|
withDocumentMetadataFlag({ main: true }, 'mainBibliography', true)
|
||||||
|
).to.eql({ main: true, mainBibliography: true })
|
||||||
|
})
|
||||||
|
|
||||||
|
it('keeps the other flag when clearing one', function () {
|
||||||
|
expect(
|
||||||
|
withDocumentMetadataFlag(
|
||||||
|
{ main: true, mainBibliography: true },
|
||||||
|
'mainBibliography',
|
||||||
|
false
|
||||||
|
)
|
||||||
|
).to.eql({ main: true })
|
||||||
|
})
|
||||||
|
|
||||||
|
it('leaves a cleared flag out rather than setting it false', function () {
|
||||||
|
expect(withDocumentMetadataFlag({ main: true }, 'main', false)).to.eql({})
|
||||||
|
})
|
||||||
|
|
||||||
|
it('drops metadata that is not a doc flag', function () {
|
||||||
|
expect(
|
||||||
|
withDocumentMetadataFlag(
|
||||||
|
{ main: true, importedAt: '2026-01-02T00:00:00.000Z' },
|
||||||
|
'mainBibliography',
|
||||||
|
true
|
||||||
|
)
|
||||||
|
).to.eql({ main: true, mainBibliography: true })
|
||||||
|
})
|
||||||
|
})
|
||||||
|
})
|
||||||
@@ -0,0 +1,192 @@
|
|||||||
|
// @ts-check
|
||||||
|
'use strict'
|
||||||
|
|
||||||
|
const { expect } = require('chai')
|
||||||
|
|
||||||
|
const File = require('../../lib/file')
|
||||||
|
const Snapshot = require('../../lib/snapshot')
|
||||||
|
const {
|
||||||
|
chooseRootDoc,
|
||||||
|
isRootDocCandidate,
|
||||||
|
setMainPathnameOperations,
|
||||||
|
} = require('../../lib/root_doc')
|
||||||
|
|
||||||
|
const WITH_CLASS = '\\documentclass{article}\n\\begin{document}\nhi\n'
|
||||||
|
const WITHOUT_CLASS = '\\section{One}\nsome text\n'
|
||||||
|
|
||||||
|
describe('root_doc', function () {
|
||||||
|
/**
|
||||||
|
* @param {Record<string, string>} docs
|
||||||
|
* @return {Snapshot}
|
||||||
|
*/
|
||||||
|
function snapshotOf(docs) {
|
||||||
|
const snapshot = new Snapshot()
|
||||||
|
for (const [pathname, content] of Object.entries(docs)) {
|
||||||
|
snapshot.addFile(pathname, File.fromString(content))
|
||||||
|
}
|
||||||
|
return snapshot
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('isRootDocCandidate', function () {
|
||||||
|
it('takes a doc that declares a document class', function () {
|
||||||
|
expect(isRootDocCandidate('paper.tex', WITH_CLASS)).to.be.true
|
||||||
|
expect(isRootDocCandidate('paper.Rtex', WITH_CLASS)).to.be.true
|
||||||
|
})
|
||||||
|
|
||||||
|
it('leaves a doc that declares none', function () {
|
||||||
|
expect(isRootDocCandidate('chapter.tex', WITHOUT_CLASS)).to.be.false
|
||||||
|
})
|
||||||
|
|
||||||
|
it('leaves a name a root doc cannot have', function () {
|
||||||
|
// The content says document class, but the compiler is never given the file.
|
||||||
|
expect(isRootDocCandidate('README.md', WITH_CLASS)).to.be.false
|
||||||
|
})
|
||||||
|
|
||||||
|
it('reads a declaration from the start of a line, indented or not', function () {
|
||||||
|
expect(isRootDocCandidate('paper.tex', ' \\documentclass{article}\n')).to
|
||||||
|
.be.true
|
||||||
|
// Not a declaration this project has: commented out, or the argument of
|
||||||
|
// something else. Web's rule, and what keeps the scan to a line's beginning.
|
||||||
|
expect(isRootDocCandidate('paper.tex', '% \\documentclass{article}\n')).to
|
||||||
|
.be.false
|
||||||
|
expect(
|
||||||
|
isRootDocCandidate('paper.tex', '\\input{\\documentclass{article}}\n')
|
||||||
|
).to.be.false
|
||||||
|
})
|
||||||
|
|
||||||
|
it('reads only the start of a doc', function () {
|
||||||
|
const buried = 'x\n'.repeat(20_000) + WITH_CLASS
|
||||||
|
expect(isRootDocCandidate('paper.tex', buried)).to.be.false
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('chooseRootDoc', function () {
|
||||||
|
it('records the only doc that could be it', function () {
|
||||||
|
const snapshot = snapshotOf({
|
||||||
|
'notes.tex': WITHOUT_CLASS,
|
||||||
|
'paper.tex': WITH_CLASS,
|
||||||
|
})
|
||||||
|
|
||||||
|
const chosen = chooseRootDoc(snapshot, ['paper.tex'])
|
||||||
|
|
||||||
|
expect(chosen?.pathname).to.equal('paper.tex')
|
||||||
|
expect(
|
||||||
|
chosen?.operations.map(operation => operation.toRaw())
|
||||||
|
).to.deep.equal([{ pathname: 'paper.tex', metadata: { main: true } }])
|
||||||
|
})
|
||||||
|
|
||||||
|
it('prefers a doc beside the project to one below it', function () {
|
||||||
|
// A main file that only includes the others sits at the top.
|
||||||
|
const snapshot = snapshotOf({
|
||||||
|
'chapters/one.tex': WITH_CLASS,
|
||||||
|
'main.tex': WITH_CLASS,
|
||||||
|
})
|
||||||
|
|
||||||
|
const chosen = chooseRootDoc(snapshot, ['chapters/one.tex', 'main.tex'])
|
||||||
|
|
||||||
|
expect(chosen?.pathname).to.equal('main.tex')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('answers the same way whichever order it is given them in', function () {
|
||||||
|
const snapshot = snapshotOf({
|
||||||
|
'a/deep.tex': WITH_CLASS,
|
||||||
|
'b/deep.tex': WITH_CLASS,
|
||||||
|
})
|
||||||
|
|
||||||
|
expect(
|
||||||
|
chooseRootDoc(snapshot, ['a/deep.tex', 'b/deep.tex'])?.pathname
|
||||||
|
).to.equal(
|
||||||
|
chooseRootDoc(snapshot, ['b/deep.tex', 'a/deep.tex'])?.pathname
|
||||||
|
)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('leaves a root doc history already records alone', function () {
|
||||||
|
// A user's choice, which nothing here is in a position to improve on.
|
||||||
|
const snapshot = snapshotOf({
|
||||||
|
'chosen.tex': WITHOUT_CLASS,
|
||||||
|
'paper.tex': WITH_CLASS,
|
||||||
|
})
|
||||||
|
snapshot.getFile('chosen.tex')?.setMetadata({ main: true })
|
||||||
|
|
||||||
|
expect(chooseRootDoc(snapshot, ['paper.tex'])).to.be.null
|
||||||
|
})
|
||||||
|
|
||||||
|
it('answers for a project with nothing to compile', function () {
|
||||||
|
const snapshot = snapshotOf({ 'notes.tex': WITHOUT_CLASS })
|
||||||
|
|
||||||
|
expect(chooseRootDoc(snapshot, [])).to.be.null
|
||||||
|
})
|
||||||
|
|
||||||
|
it('leaves out a doc the project does not have', function () {
|
||||||
|
// A doc whose write was skipped: recording it would leave the mark on a
|
||||||
|
// pathname with no file behind it.
|
||||||
|
const snapshot = snapshotOf({ 'main.tex': WITH_CLASS })
|
||||||
|
|
||||||
|
const chosen = chooseRootDoc(snapshot, ['dropped.tex', 'main.tex'])
|
||||||
|
|
||||||
|
expect(chosen?.pathname).to.equal('main.tex')
|
||||||
|
expect(chooseRootDoc(snapshot, ['dropped.tex'])).to.be.null
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('setMainPathnameOperations', function () {
|
||||||
|
it('moves the record off the file that held it', function () {
|
||||||
|
const snapshot = snapshotOf({
|
||||||
|
'old.tex': WITH_CLASS,
|
||||||
|
'new.tex': WITH_CLASS,
|
||||||
|
})
|
||||||
|
snapshot.getFile('old.tex')?.setMetadata({ main: true })
|
||||||
|
|
||||||
|
const operations = setMainPathnameOperations(snapshot, 'new.tex')
|
||||||
|
|
||||||
|
expect(operations.map(operation => operation.toRaw())).to.deep.equal([
|
||||||
|
{ pathname: 'new.tex', metadata: { main: true } },
|
||||||
|
{ pathname: 'old.tex', metadata: {} },
|
||||||
|
])
|
||||||
|
})
|
||||||
|
|
||||||
|
it('has nothing to do for the file that already holds it', function () {
|
||||||
|
const snapshot = snapshotOf({ 'main.tex': WITH_CLASS })
|
||||||
|
snapshot.getFile('main.tex')?.setMetadata({ main: true })
|
||||||
|
|
||||||
|
expect(setMainPathnameOperations(snapshot, 'main.tex')).to.deep.equal([])
|
||||||
|
})
|
||||||
|
|
||||||
|
it('keeps the other doc flags of both files', function () {
|
||||||
|
// Metadata is replaced rather than merged, so a bibliography a file is also
|
||||||
|
// the main one of has to be written out again with the change.
|
||||||
|
const snapshot = snapshotOf({
|
||||||
|
'old.tex': WITH_CLASS,
|
||||||
|
'new.tex': WITH_CLASS,
|
||||||
|
})
|
||||||
|
snapshot
|
||||||
|
.getFile('old.tex')
|
||||||
|
?.setMetadata({ main: true, mainBibliography: true })
|
||||||
|
snapshot.getFile('new.tex')?.setMetadata({ mainBibliography: true })
|
||||||
|
|
||||||
|
const operations = setMainPathnameOperations(snapshot, 'new.tex')
|
||||||
|
|
||||||
|
expect(operations.map(operation => operation.toRaw())).to.deep.equal([
|
||||||
|
{
|
||||||
|
pathname: 'new.tex',
|
||||||
|
metadata: { main: true, mainBibliography: true },
|
||||||
|
},
|
||||||
|
{ pathname: 'old.tex', metadata: { mainBibliography: true } },
|
||||||
|
])
|
||||||
|
})
|
||||||
|
|
||||||
|
it('refuses a file carrying other metadata', function () {
|
||||||
|
// Metadata shapes do not combine, so recording this would throw the rest away.
|
||||||
|
const snapshot = snapshotOf({ 'linked.tex': WITH_CLASS })
|
||||||
|
snapshot.getFile('linked.tex')?.setMetadata({
|
||||||
|
provider: 'project_file',
|
||||||
|
source_entity_path: '/main.tex',
|
||||||
|
source_project_id: 'p',
|
||||||
|
})
|
||||||
|
|
||||||
|
expect(() => setMainPathnameOperations(snapshot, 'linked.tex')).to.throw(
|
||||||
|
'only a doc can be the root doc'
|
||||||
|
)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
})
|
||||||
@@ -285,6 +285,28 @@ describe('schemas', function () {
|
|||||||
expect(result.success).to.equal(false)
|
expect(result.success).to.equal(false)
|
||||||
})
|
})
|
||||||
|
|
||||||
|
it('accepts a main bibliography flag', function () {
|
||||||
|
const result = rawFileMetadata.safeParse({
|
||||||
|
mainBibliography: true,
|
||||||
|
})
|
||||||
|
expect(result.success).to.equal(true)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('accepts both doc flags on one file', function () {
|
||||||
|
const result = rawFileMetadata.safeParse({
|
||||||
|
main: true,
|
||||||
|
mainBibliography: true,
|
||||||
|
})
|
||||||
|
expect(result.success).to.equal(true)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('rejects a doc flag that is not a boolean', function () {
|
||||||
|
const result = rawFileMetadata.safeParse({
|
||||||
|
mainBibliography: 'yes',
|
||||||
|
})
|
||||||
|
expect(result.success).to.equal(false)
|
||||||
|
})
|
||||||
|
|
||||||
it('accepts an empty object', function () {
|
it('accepts an empty object', function () {
|
||||||
const result = rawFileMetadata.safeParse({})
|
const result = rawFileMetadata.safeParse({})
|
||||||
expect(result.success).to.equal(true)
|
expect(result.success).to.equal(true)
|
||||||
|
|||||||
@@ -55,6 +55,60 @@ describe('Snapshot', function () {
|
|||||||
})
|
})
|
||||||
})
|
})
|
||||||
|
|
||||||
|
describe('getPathnameWithDocFlag', function () {
|
||||||
|
it('finds the doc carrying the flag', function () {
|
||||||
|
const snapshot = new Snapshot()
|
||||||
|
snapshot.addFile('a.tex', File.fromString('', { main: true }))
|
||||||
|
expect(snapshot.getPathnameWithDocFlag('main')).to.equal('a.tex')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('answers nothing when no file carries the flag', function () {
|
||||||
|
const snapshot = new Snapshot()
|
||||||
|
snapshot.addFile('a.tex', File.fromString(''))
|
||||||
|
expect(snapshot.getPathnameWithDocFlag('main')).to.be.undefined
|
||||||
|
})
|
||||||
|
|
||||||
|
it('is not confused by a different flag', function () {
|
||||||
|
const snapshot = new Snapshot()
|
||||||
|
snapshot.addFile('a.tex', File.fromString('', { mainBibliography: true }))
|
||||||
|
expect(snapshot.getPathnameWithDocFlag('main')).to.be.undefined
|
||||||
|
})
|
||||||
|
|
||||||
|
it('picks the flagged file out of several', function () {
|
||||||
|
const snapshot = new Snapshot()
|
||||||
|
snapshot.addFile('a.tex', File.fromString(''))
|
||||||
|
snapshot.addFile('b.tex', File.fromString('', { main: true }))
|
||||||
|
snapshot.addFile('c.tex', File.fromString(''))
|
||||||
|
expect(snapshot.getPathnameWithDocFlag('main')).to.equal('b.tex')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('ignores the flag on a file that is not a doc', function () {
|
||||||
|
// A file with anything else in its metadata came from outside the editor, so
|
||||||
|
// a stale flag there does not count -- even though its content is not loaded
|
||||||
|
// yet, and so is not known to be editable either.
|
||||||
|
const snapshot = new Snapshot()
|
||||||
|
snapshot.addFile(
|
||||||
|
'a.bin',
|
||||||
|
File.fromHash(File.EMPTY_FILE_HASH, undefined, {
|
||||||
|
main: true,
|
||||||
|
importedAt: '2026-01-01T00:00:00.000Z',
|
||||||
|
})
|
||||||
|
)
|
||||||
|
expect(snapshot.getPathnameWithDocFlag('main')).to.be.undefined
|
||||||
|
})
|
||||||
|
|
||||||
|
it('finds the flag on a doc whose content is not loaded yet', function () {
|
||||||
|
// As a file straight off a freshly fetched chunk is: known only by its hash,
|
||||||
|
// with document-only metadata already saying it is a doc.
|
||||||
|
const snapshot = new Snapshot()
|
||||||
|
snapshot.addFile(
|
||||||
|
'a.tex',
|
||||||
|
File.fromHash('a'.repeat(40), undefined, { main: true })
|
||||||
|
)
|
||||||
|
expect(snapshot.getPathnameWithDocFlag('main')).to.equal('a.tex')
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
describe('applyAll', function () {
|
describe('applyAll', function () {
|
||||||
let snapshot
|
let snapshot
|
||||||
let change
|
let change
|
||||||
|
|||||||
Reference in New Issue
Block a user