Merge pull request #30798 from overleaf/ar-reapply-request-overrides

[monorepo] reapply request overrides

GitOrigin-RevId: 55d19cc011435a00ffc1e5c63f523473cd0fc2dc

README.md

@@ -6,58 +6,52 @@
 <h4 align="center">An open-source online real-time collaborative LaTeX editor.</h4>
 
 <p align="center">
-  <a href="https://github.com/yu-i-i/overleaf-cep/wiki">Wiki</a> •
+  <a href="https://github.com/overleaf/overleaf/wiki">Wiki</a> •
   <a href="https://www.overleaf.com/for/enterprises">Server Pro</a> •
+  <a href="#contributing">Contributing</a> •
+  <a href="https://mailchi.mp/overleaf.com/community-edition-and-server-pro">Mailing List</a> •
   <a href="#authors">Authors</a> •
   <a href="#license">License</a>
 </p>
 
-<img src="doc/screenshot.png" alt="A screenshot of a project being edited in Overleaf Extended Community Edition">
+<img src="doc/screenshot.png" alt="A screenshot of a project being edited in Overleaf Community Edition">
 <p align="center">
-  Figure 1: A screenshot of a project being edited in Overleaf Extended Community Edition.
+  Figure 1: A screenshot of a project being edited in Overleaf Community Edition.
 </p>
 
 ## Community Edition
 
-[Overleaf](https://www.overleaf.com) is an open-source online real-time collaborative LaTeX editor. Overleaf runs a hosted version at [www.overleaf.com](https://www.overleaf.com), but you can also run your own local version, and contribute to the development of Overleaf.
-
-## Extended Community Edition
-
-The present "extended" version of Overleaf CE includes:
-
-- Template Gallery
-- Sandboxed Compiles with TeX Live image selection
-- LDAP authentication
-- SAML authentication
-- OpenID Connect authentication
-- Real-time track changes and comments
-- Autocomplete of reference keys
-- Symbol Palette
-- "From External URL" feature
+[Overleaf](https://www.overleaf.com) is an open-source online real-time collaborative LaTeX editor. We run a hosted version at [www.overleaf.com](https://www.overleaf.com), but you can also run your own local version, and contribute to the development of Overleaf.
 
 > [!CAUTION]
-> Overleaf Community Edition is intended for use in environments where **all** users are trusted. Community Edition is **not** appropriate for scenarios where isolation of users is required due to Sandbox Compiles not being available. When not using Sandboxed Compiles, users have full read and write access to the `sharelatex` container resources (filesystem, network, environment variables) when running LaTeX compiles. 
-Therefore, in any environment where not all users can be fully trusted, it is strongly recommended to enable the Sandboxed Compiles feature available in the Extended Community Edition.
+> Overleaf Community Edition is intended for use in environments where **all** users are trusted. Community Edition is **not** appropriate for scenarios where isolation of users is required due to Sandbox Compiles not being available. When not using Sandboxed Compiles, users have full read and write access to the `sharelatex` container resources (filesystem, network, environment variables) when running LaTeX compiles.
 
-For more information on Sandbox Compiles check out Overleaf [documentation](https://docs.overleaf.com/on-premises/configuration/overleaf-toolkit/server-pro-only-configuration/sandboxed-compiles).
+For more information on Sandbox Compiles check out our [documentation](https://docs.overleaf.com/on-premises/configuration/overleaf-toolkit/server-pro-only-configuration/sandboxed-compiles).
 
 ## Enterprise
 
-If you want help installing and maintaining Overleaf in your lab or workplace, Overleaf offers an officially supported version called [Overleaf Server Pro](https://www.overleaf.com/for/enterprises).
+If you want help installing and maintaining Overleaf in your lab or workplace, we offer an officially supported version called [Overleaf Server Pro](https://www.overleaf.com/for/enterprises). It also includes more features for security (SSO with LDAP or SAML), administration and collaboration (e.g. tracked changes). [Find out more!](https://www.overleaf.com/for/enterprises)
+
+## Keeping up to date
+
+Sign up to the [mailing list](https://mailchi.mp/overleaf.com/community-edition-and-server-pro) to get updates on Overleaf releases and development.
 
 ## Installation
 
-Detailed installation instructions can be found in the [Overleaf Toolkit](https://github.com/overleaf/toolkit/).
-Configuration details and release history for the Extended Community Edition can be found on the [Extended CE Wiki Page](https://github.com/yu-i-i/overleaf-cep/wiki).
+We have detailed installation instructions in the [Overleaf Toolkit](https://github.com/overleaf/toolkit/).
+
+## Upgrading
+
+If you are upgrading from a previous version of Overleaf, please see the [Release Notes section on the Wiki](https://github.com/overleaf/overleaf/wiki#release-notes) for all of the versions between your current version and the version you are upgrading to.
 
 ## Overleaf Docker Image
 
 This repo contains two dockerfiles, [`Dockerfile-base`](server-ce/Dockerfile-base), which builds the
-`sharelatex/sharelatex-base:ext-ce` image, and [`Dockerfile`](server-ce/Dockerfile) which builds the
-`sharelatex/sharelatex:ext-ce` image.
+`sharelatex/sharelatex-base` image, and [`Dockerfile`](server-ce/Dockerfile) which builds the
+`sharelatex/sharelatex` (or "community") image.
 
 The Base image generally contains the basic dependencies like `wget`, plus `texlive`.
-This is split out because it's a pretty heavy set of
+We split this out because it's a pretty heavy set of
 dependencies, and it's nice to not have to rebuild all of that every time.
 
 The `sharelatex/sharelatex` image extends the base image and adds the actual Overleaf code
@@ -65,16 +59,20 @@ and services.
 
 Use `make build-base` and `make build-community` from `server-ce/` to build these images.
 
-The [Phusion base-image](https://github.com/phusion/baseimage-docker)
-(which is extended by the `base` image) provides a VM-like container
+We use the [Phusion base-image](https://github.com/phusion/baseimage-docker)
+(which is extended by our `base` image) to provide us with a VM-like container
 in which to run the Overleaf services. Baseimage uses the `runit` service
-manager to manage services, and init scripts from the `server-ce/runit`
-folder are added.
+manager to manage services, and we add our init-scripts from the `server-ce/runit`
+folder.
+
+
+## Contributing
+
+Please see the [CONTRIBUTING](CONTRIBUTING.md) file for information on contributing to the development of Overleaf.
 
 ## Authors
 
-[The Overleaf Team](https://www.overleaf.com/about)  
-[yu-i-i](https://github.com/yu-i-i/overleaf-cep) — Extensions for CE unless otherwise noted
+[The Overleaf Team](https://www.overleaf.com/about)
 
 ## License
 

develop/README.md

@@ -77,10 +77,8 @@ each service:
 | `filestore`        | 9235 |
 | `notifications`    | 9236 |
 | `real-time`        | 9237 |
-| `references`       | 9238 |
 | `history-v1`       | 9239 |
 | `project-history`  | 9240 |
-| `linked-url-proxy` | 9241 |
 
 To attach to a service using Chrome's _remote debugging_, go to
 <chrome://inspect/> and make sure _Discover network targets_ is checked. Next

develop/dev.env

@@ -7,7 +7,6 @@ FILESTORE_HOST=filestore
 GRACEFUL_SHUTDOWN_DELAY_SECONDS=0
 HISTORY_V1_HOST=history-v1
 HISTORY_REDIS_HOST=redis
-LINKED_URL_PROXY_HOST=linked-url-proxy
 LISTEN_ADDRESS=0.0.0.0
 MONGO_HOST=mongo
 MONGO_URL=mongodb://mongo/sharelatex?directConnection=true
@@ -17,7 +16,6 @@ QUEUES_REDIS_HOST=redis
 DSMP_REDIS_HOST=redis
 REALTIME_HOST=real-time
 REDIS_HOST=redis
-REFERENCES_HOST=references
 SESSION_SECRET=foo
 V1_HISTORY_HOST=history-v1
 WEBPACK_HOST=webpack

develop/docker-compose.dev.yml

@@ -79,17 +79,6 @@ services:
       - ../services/history-v1/knexfile.js:/overleaf/services/history-v1/knexfile.js
       - ../services/history-v1/migrations:/overleaf/services/history-v1/migrations
 
-  linked-url-proxy:
-    command: ["node", "--watch", "app.mjs"]
-    environment:
-      - NODE_OPTIONS=--inspect=0.0.0.0:9229
-    ports:
-      - "127.0.0.1:9241:9229"
-    volumes:
-      - ../services/linked-url-proxy/app:/overleaf/services/linked-url-proxy/app
-      - ../services/linked-url-proxy/config:/overleaf/services/linked-url-proxy/config
-      - ../services/linked-url-proxy/app.js:/overleaf/services/linked-url-proxy/app.js
-
   notifications:
     command: ["node", "--watch", "app.ts"]
     environment:
@@ -123,17 +112,6 @@ services:
       - ../services/real-time/app.js:/overleaf/services/real-time/app.js
       - ../services/real-time/config:/overleaf/services/real-time/config
 
-  references:
-    command: ["node", "--watch", "app.js"]
-    environment:
-      - NODE_OPTIONS=--inspect=0.0.0.0:9229
-    ports:
-      - "127.0.0.1:9238:9229"
-    volumes:
-      - ../services/references/app:/overleaf/services/references/app
-      - ../services/references/config:/overleaf/services/references/config
-      - ../services/references/app.js:/overleaf/services/references/app.js
-
   web:
     command: ["node", "--watch", "app.mjs", "--watch-locales"]
     environment:

develop/docker-compose.yml

@@ -86,13 +86,6 @@ services:
     volumes:
         - history-v1-buckets:/buckets
 
-  linked-url-proxy:
-    build:
-      context: ..
-      dockerfile: services/linked-url-proxy/Dockerfile
-    env_file:
-      - dev.env
-
   mongo:
     image: mongo:6.0
     command: --replSet overleaf
@@ -129,7 +122,7 @@ services:
       dockerfile: services/real-time/Dockerfile
     env_file:
       - dev.env
- 
+
   redis:
     image: redis:5
     ports:
@@ -137,13 +130,6 @@ services:
     volumes:
       - redis-data:/data
 
-  references:
-    build:
-      context: ..
-      dockerfile: services/references/Dockerfile
-    env_file:
-      - dev.env
-
   web:
     build:
       context: ..
@@ -171,11 +157,9 @@ services:
       - document-updater
       - filestore
       - history-v1
-      - linked-url-proxy
       - notifications
       - project-history
       - real-time
-      - references
 
   webpack:
     build:

libraries/access-token-encryptor/.jenkinsIncludeFile

@@ -1,7 +0,0 @@
-# Autogenerated by build scripts. Do not edit.
-.eslint*
-.prettier*
-libraries/access-token-encryptor/**
-package-lock.json
-package.json
-patches/**

libraries/access-token-encryptor/Jenkinsfile

@@ -1,82 +0,0 @@
-// Autogenerated by build scripts. Do not edit.
-
-pipeline {
-  agent {
-    node {
-      label 'jenkins-agent-web'
-      customWorkspace '/workspace'
-    }
-  }
-  options {
-    timestamps()
-    timeout(time: 15, unit: 'MINUTES')
-  }
-  environment {
-    BRANCH_NAME = "${env.CHANGE_BRANCH ? env.CHANGE_BRANCH : env.BRANCH_NAME}"
-    JENKINS_BUILD_NUMBER = "${BUILD_NUMBER}"
-    BUILD_NUMBER = "${SHORT_SHA}_${BUILD_NUMBER}"
-    COMMIT_SHA = "${GIT_COMMIT}"
-    SHORT_SHA = "${GIT_COMMIT.take(7)}"
-  }
-  stages {
-    stage('Stage 1') {
-      parallel {
-        stage('Install monorepo') {
-          steps {
-            retry(count: 3) {
-              sh 'make monorepo_setup'
-            }
-          }
-        }
-        stage('Create reports folder') {
-          steps {
-            sh 'mkdir libraries/access-token-encryptor/reports'
-          }
-        }
-      }
-    }
-    stage('Stage 2') {
-      parallel {
-        stage('Lint') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/access-token-encryptor monorepo npm run lint -- --format json --output-file reports/eslint.json'
-          }
-          post {
-            always {
-              sh """
-                sed -i 's_"filePath":"/overleaf_"filePath":"/workspace_g' libraries/access-token-encryptor/reports/eslint.json
-              """
-              recordIssues checksAnnotationScope: 'ALL', enabledForFailure: true, failOnError: true, id: 'access-token-encryptor-eslint', name: 'access-token-encryptor eslint', qualityGates: [[integerThreshold: 1, threshold: 1.0, type: 'TOTAL']], sourceCodeRetention: 'LAST_BUILD', tools: [esLint(pattern: 'libraries/access-token-encryptor/reports/eslint.json')]
-            }
-          }
-        }
-        stage('Format') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/access-token-encryptor monorepo npm run format'
-          }
-        }
-        stage('Typecheck') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/access-token-encryptor monorepo npm run types:check'
-          }
-        }
-        stage('Test') {
-          steps {
-            retry(count: 3) {
-              sh 'bin/run -w /overleaf/libraries/access-token-encryptor monorepo npm run test:ci'
-            }
-          }
-        }
-      }
-    }
-  }
-  post {
-    always {
-      junit checksName: 'access-token-encryptor test results', testResults: 'libraries/access-token-encryptor/reports/junit-*.xml'
-    }
-    cleanup {
-      sh 'rm -rf libraries/access-token-encryptor/reports'
-      sh 'make clean_jenkins -j10'
-    }
-  }
-}

libraries/access-token-encryptor/buildscript.txt

@@ -5,4 +5,5 @@ access-token-encryptor
 --esmock-loader=False
 --is-library=True
 --node-version=22.18.0
+--pipeline-owner=🚉 Platform
 --public-repo=False

libraries/access-token-encryptor/test/mocha-multi-reporters.cjs

@@ -4,5 +4,6 @@ module.exports = {
     mochaFile: `reports/junit-mocha-${process.env.MOCHA_GREP}.xml`,
     includePending: true,
     jenkinsMode: true,
+    output: true,
   },
 }

libraries/eslint-plugin/index.js

@@ -0,0 +1,11 @@
+module.exports = {
+  rules: {
+    'no-unnecessary-trans': require('./no-unnecessary-trans'),
+    'prefer-kebab-url': require('./prefer-kebab-url'),
+    'should-unescape-trans': require('./should-unescape-trans'),
+    'no-generated-editor-themes': require('./no-generated-editor-themes'),
+    'require-script-runner': require('./require-script-runner'),
+    'require-vi-doMock-valid-path': require('./require-vi-doMock-valid-path'),
+    'require-loading-label': require('./require-loading-label'),
+  },
+}

libraries/eslint-plugin/no-generated-editor-themes.js

@@ -0,0 +1,21 @@
+module.exports = {
+  meta: {
+    type: 'error',
+    docs: {
+      description:
+        'Prohibit CodeMirror themes that are generated in a function',
+    },
+  },
+  create(context) {
+    return {
+      ':matches(ArrowFunctionExpression, FunctionDeclaration, FunctionExpression) CallExpression > MemberExpression[object.name="EditorView"]:matches([property.name="theme"],[property.name="baseTheme"])'(
+        node
+      ) {
+        context.report({
+          node,
+          message: `EditorView.theme and EditorView.baseTheme each add CSS to the page for every instance of the theme. Store the theme in a variable and reuse it instead.`,
+        })
+      },
+    }
+  },
+}

libraries/eslint-plugin/no-unnecessary-trans.js

@@ -0,0 +1,43 @@
+module.exports = {
+  meta: {
+    type: 'problem',
+    fixable: 'code',
+    docs: {
+      description: 'Prohibit Trans with no components or values',
+    },
+  },
+  create(context) {
+    return {
+      'JSXOpeningElement[name.name="Trans"]'(node) {
+        const attributes = new Map(
+          node.attributes.map(attr => [attr.name.name, attr])
+        )
+
+        if (!attributes.has('components')) {
+          if (node.parent.children.length > 0) {
+            context.report({
+              node,
+              message: `Trans components must not have child elements`,
+            })
+          } else if (attributes.has('values')) {
+            context.report({
+              node,
+              message: `Use t('…') when there are no components`,
+            })
+          } else {
+            context.report({
+              node,
+              message: `Use t('…') when there are no components`,
+              fix(fixer) {
+                const i18nKey = attributes.get('i18nKey').value.value
+
+                // Note: Prettier can fix indentation
+                return fixer.replaceText(node.parent, `{t('${i18nKey}')}`)
+              },
+            })
+          }
+        }
+      },
+    }
+  },
+}

libraries/eslint-plugin/package.json

@@ -0,0 +1,17 @@
+{
+  "name": "@overleaf/eslint-plugin",
+  "version": "0.1.0",
+  "author": "Overleaf (https://www.overleaf.com)",
+  "license": "AGPL-3.0-only",
+  "main": "index.js",
+  "dependencies": {
+    "eslint": "^8.51.0",
+    "lodash": "^4.17.21"
+  },
+  "devDependencies": {
+    "@typescript-eslint/parser": "^8.50.0"
+  },
+  "scripts": {
+    "test": "node rules.test.js"
+  }
+}

libraries/eslint-plugin/prefer-kebab-url-ignore.js

@@ -0,0 +1,83 @@
+// URL parts should be kebab-case, but we didn't have this rule in the past.
+// The ESLint rule `prefer-kebab-url` will ignore these "legacy" URL parts.
+
+const ignoreWords = {
+  snake: new Set([
+    'clear_saml_data',
+    'confirm_link',
+    'confirm_university_domain',
+    'create_recurly_account',
+    'current_history_content',
+    'current_user',
+    'default_email',
+    'disable_managed_users',
+    'doc_snapshot',
+    'enable_history_ranges_support',
+    'features_override',
+    'generate_password_reset_url',
+    'get_assignment',
+    'get_clone',
+    'health_check',
+    'institutional_emails',
+    'latest_template',
+    'link_after_saml_response',
+    'linked_file',
+    'metrics_segmentation',
+    'new_users',
+    'no_autostart_post_gateway',
+    'personal_info',
+    'planned_maintenance',
+    'refresh_features',
+    'register_admin',
+    'register_ldap_admin',
+    'register_saml_admin',
+    'restore_file',
+    'revert_file',
+    'saved_vers',
+    'send_test_email',
+    'session_maintenance',
+    'set_in_session',
+    'sign_in_to_link',
+    'split_test',
+    'sso_configuration_test',
+    'sso_email',
+    'sso_enrollment',
+    'track_changes',
+    'update_admin',
+    'user_details',
+  ]),
+  camel: new Set([
+    'addWorkflowScope',
+    'aiErrorAssistant',
+    'beginAuth',
+    'brandVariationId',
+    'closeEditor',
+    'completeRegistration',
+    'deactivateOldProjects',
+    'deletedSubscription',
+    'disconnectAllUsers',
+    'editingSession',
+    'emailSubscription',
+    'enableManagedUsers',
+    'externalCollaboration',
+    'flushProjectToTpds',
+    'indexAll',
+    'offboardManagedUser',
+    'openEditor',
+    'perfTest',
+    'pollDropboxForUser',
+    'resendInvite',
+    'resendManagedUserInvite',
+    'salesContactForm',
+    'showSupport',
+  ]),
+  other: new Set([
+    'Project',
+    'disableSSO',
+    'enableSSO',
+    'resendSSOLinkInvite',
+    'usersCSV',
+  ]),
+}
+
+module.exports = { ignoreWords }

libraries/eslint-plugin/prefer-kebab-url.js

@@ -0,0 +1,91 @@
+const _ = require('lodash')
+const { ignoreWords } = require('./prefer-kebab-url-ignore')
+
+const removeTextBetweenBrackets = text => {
+  while (text.includes('[') || text.includes('(')) {
+    text = text.replaceAll(/\[[^[\]]*]/g, '')
+    text = text.replaceAll(/\([^()]*\)/g, '')
+  }
+  return text
+}
+
+const shouldIgnoreWord = str =>
+  str.includes(':') ||
+  str.includes('(') ||
+  str === '*' ||
+  str.match(/^[a-z0-9.]+$/) ||
+  ignoreWords.snake.has(str) ||
+  ignoreWords.camel.has(str) ||
+  ignoreWords.other.has(str)
+
+const getSuggestion = routePath => {
+  if (typeof routePath === 'string') {
+    const kebabed = routePath
+      .split('/')
+      .map(word => (shouldIgnoreWord(word) ? word : _.kebabCase(word)))
+      .join('/')
+    return kebabed === routePath ? null : `'${kebabed}'`
+  }
+
+  if (routePath instanceof RegExp) {
+    const words = removeTextBetweenBrackets(routePath.source).match(/[\w-]+/g)
+    if (!words) return routePath
+
+    let newSource = routePath.source
+    for (const word of words) {
+      if (!shouldIgnoreWord(word)) {
+        newSource = newSource.replaceAll(
+          new RegExp(`\\b${word}\\b`, 'g'),
+          _.kebabCase(word)
+        )
+      }
+    }
+
+    const kebabed = new RegExp(newSource, routePath.flags)
+    return kebabed.source.toString() === routePath.source.toString()
+      ? null
+      : kebabed
+  }
+}
+
+module.exports = {
+  meta: {
+    type: 'problem',
+    fixable: 'code',
+    hasSuggestions: true,
+    docs: {
+      description: 'Enforce using kebab-case for URL paths',
+    },
+  },
+  create: context => ({
+    CallExpression(node) {
+      if (
+        node.callee.type === 'MemberExpression' &&
+        node.arguments[0]?.type === 'Literal' &&
+        [/app/i, /router/i].some(callee =>
+          typeof callee === 'string'
+            ? node.callee.object.name === callee
+            : callee.test(node.callee.object.name)
+        ) &&
+        ['get', 'post', 'put', 'delete'].includes(node.callee.property.name)
+      ) {
+        const routePath = node.arguments[0].value
+
+        const suggestion = getSuggestion(routePath)
+
+        if (suggestion) {
+          context.report({
+            node: node.arguments[0],
+            message: 'Route path should be in kebab-case.',
+            suggest: [
+              {
+                desc: `Change to kebab-case: ${suggestion}`,
+                fix: fixer => fixer.replaceText(node.arguments[0], suggestion),
+              },
+            ],
+          })
+        }
+      }
+    },
+  }),
+}

libraries/eslint-plugin/require-loading-label.js

@@ -0,0 +1,49 @@
+module.exports = {
+  meta: {
+    type: 'problem',
+    fixable: null,
+    docs: {
+      description:
+        'Require loadingLabel prop when isLoading is specified on OLButton',
+    },
+    schema: [],
+  },
+  create(context) {
+    return {
+      'JSXOpeningElement[name.name="OLButton"]'(node) {
+        const attributes = new Map(
+          node.attributes.map(attr => [attr.name?.name, attr])
+        )
+
+        const isLoadingAttr = attributes.get('isLoading')
+        const loadingLabelAttr = attributes.get('loadingLabel')
+
+        if (isLoadingAttr && !loadingLabelAttr) {
+          const isLoadingValue = isLoadingAttr.value
+
+          if (
+            !isLoadingValue ||
+            (isLoadingValue.type === 'JSXExpressionContainer' &&
+              isLoadingValue.expression.type === 'Literal' &&
+              isLoadingValue.expression.value === true)
+          ) {
+            context.report({
+              node: isLoadingAttr,
+              message:
+                'Button with isLoading prop must also specify loadingLabel',
+            })
+          } else if (
+            isLoadingValue.type === 'JSXExpressionContainer' &&
+            isLoadingValue.expression.type !== 'Literal'
+          ) {
+            context.report({
+              node: isLoadingAttr,
+              message:
+                'Button with isLoading prop must also specify loadingLabel',
+            })
+          }
+        }
+      },
+    }
+  },
+}

libraries/eslint-plugin/require-script-runner.js

@@ -0,0 +1,28 @@
+module.exports = {
+  meta: {
+    type: 'suggestion',
+    docs: {
+      description: 'Require Script Runner for scripts',
+    },
+  },
+  create(context) {
+    let hasImport = false
+
+    return {
+      ImportDeclaration(node) {
+        if (node.source.value.endsWith('lib/ScriptRunner.mjs')) {
+          hasImport = true
+        }
+      },
+      'Program:exit'() {
+        if (!hasImport) {
+          context.report({
+            loc: { line: 1, column: 0 },
+            message:
+              'Please use Script Runner for scripts. Refer to the developer manual (https://manual.dev-overleaf.com/development/code/web_scripts/#monitor-script-execution-and-usage-with-script-runner) for more information.',
+          })
+        }
+      },
+    }
+  },
+}

libraries/eslint-plugin/require-vi-doMock-valid-path.js

@@ -0,0 +1,138 @@
+const path = require('node:path')
+const fs = require('node:fs')
+
+module.exports = {
+  meta: {
+    type: 'problem',
+    docs: {
+      description: 'Ensure vi.doMock first argument is a resolvable path.',
+      category: 'Best Practices',
+      recommended: false,
+      url: '',
+    },
+    fixable: 'code',
+    hasSuggestions: true,
+    schema: [],
+    messages: {
+      unresolvablePath:
+        'The path "{{pathValue}}" in vi.doMock() cannot be resolved relative to the current file.',
+      notAStringLiteral:
+        'The first argument of vi.doMock() must be (or resolve to) a string literal representing a path.',
+      noArguments: 'vi.doMock() called with no arguments.',
+    },
+  },
+  create(context) {
+    const currentFilePath = context.getFilename()
+    // ESLint can sometimes pass <text> or <input> for snippets not in a file
+    if (currentFilePath === '<text>' || currentFilePath === '<input>') {
+      return {}
+    }
+    const currentDirectory = path.dirname(currentFilePath)
+
+    function canResolve(modulePath) {
+      try {
+        require.resolve(path.resolve(currentDirectory, modulePath))
+        return true
+      } catch (e) {
+        const absolutePath = path.resolve(currentDirectory, modulePath)
+        const extensions = [
+          '',
+          '.js',
+          '.mjs',
+          '.ts',
+          '.jsx',
+          '.tsx',
+          '.json',
+          '.node',
+          '/index.js',
+          '/index.ts',
+        ] // Add common extensions
+        for (const ext of extensions) {
+          if (fs.existsSync(absolutePath + ext)) {
+            return true
+          }
+        }
+        return false
+      }
+    }
+
+    return {
+      CallExpression(node) {
+        if (
+          node.callee.type === 'MemberExpression' &&
+          node.callee.object.type === 'Identifier' &&
+          node.callee.object.name === 'vi' &&
+          node.callee.property.type === 'Identifier' &&
+          node.callee.property.name === 'doMock'
+        ) {
+          if (node.arguments.length === 0) {
+            context.report({
+              node,
+              messageId: 'noArguments',
+            })
+            return
+          }
+
+          const firstArg = node.arguments[0]
+          let pathValue = firstArg.value
+
+          if (
+            firstArg.type !== 'Literal' ||
+            typeof firstArg.value !== 'string'
+          ) {
+            if (firstArg.type === 'Identifier') {
+              const variable = context
+                .getScope()
+                .variables.find(v => v.name === firstArg.name)
+              if (
+                variable &&
+                variable.defs.length > 0 &&
+                variable.defs[0].node.init &&
+                variable.defs[0].node.init.type === 'Literal' &&
+                typeof variable.defs[0].node.init.value === 'string'
+              ) {
+                pathValue = variable.defs[0].node.init.value
+                if (canResolve(pathValue)) {
+                  return
+                }
+                // If the first argument was a variable that didn't resolve then we can't auto-fix it
+              }
+            }
+            context.report({
+              node: firstArg,
+              messageId: 'notAStringLiteral',
+            })
+            return
+          }
+
+          if (!pathValue.startsWith('.')) {
+            return
+          }
+
+          if (!canResolve(pathValue)) {
+            const mjsPath = pathValue.replace('.js', '.mjs')
+            const additionalReportOptions = {}
+            if (canResolve(mjsPath)) {
+              additionalReportOptions.fix = fixer =>
+                fixer.replaceText(firstArg, `'${mjsPath}'`)
+              additionalReportOptions.suggest = [
+                {
+                  desc: `Replace with "${pathValue.replace('.js', '.mjs')}"`,
+                  fix: fixer => fixer.replaceText(firstArg, `'${mjsPath}'`),
+                },
+              ]
+            }
+            context.report({
+              node: firstArg,
+              messageId: 'unresolvablePath',
+              data: {
+                pathValue,
+              },
+              ...additionalReportOptions,
+            })
+          }
+        }
+      },
+    }
+  },
+}

libraries/eslint-plugin/rules.test.js

@@ -0,0 +1,168 @@
+const { RuleTester } = require('eslint')
+const preferKebabUrl = require('./prefer-kebab-url')
+const noUnnecessaryTrans = require('./no-unnecessary-trans')
+const shouldUnescapeTrans = require('./should-unescape-trans')
+const noGeneratedEditorThemes = require('./no-generated-editor-themes')
+const viDoMockValidPath = require('./require-vi-doMock-valid-path')
+
+const ruleTester = new RuleTester({
+  parser: require.resolve('@typescript-eslint/parser'),
+  parserOptions: {
+    ecmaVersion: 'latest',
+    ecmaFeatures: { jsx: true },
+  },
+})
+
+ruleTester.run('prefer-kebab-url', preferKebabUrl, {
+  valid: [
+    { code: `app.get('/foo-bar')` },
+    { code: `app.get('/foo-bar/:id')` },
+    { code: `router.post('/foo-bar')` },
+    { code: `router.get('/foo-bar/:id/:name/:age')` },
+    { code: `webRouter.get('/foo-bar/:user_id/(ProjectName)/get-info')` },
+    { code: `webApp.post('/foo-bar/:user_id/(ProjectName)/get-info')` },
+    {
+      code: `router.get(/^\\/download\\/project\\/([^/]*)\\/output\\/output\\.pdf$/)`,
+    },
+    {
+      code: `webRouter.get(/^\\/project\\/([^/]*)\\/user\\/([0-9a-f]+)\\/build\\/([0-9a-f-]+)\\/output\\/(.*)$/)`,
+    },
+  ],
+  invalid: [
+    {
+      code: `app.get('/fooBar')`,
+      errors: [{ message: 'Route path should be in kebab-case.' }],
+    },
+    {
+      code: `app.get('/fooBar/:id')`,
+      errors: [{ message: 'Route path should be in kebab-case.' }],
+    },
+    {
+      code: `webRouter.get('/foo_bar/:id/FooBar/:name/fooBar')`,
+      errors: [{ message: 'Route path should be in kebab-case.' }],
+    },
+    {
+      code: `router.get(/^\\/downLoad\\/pro-ject\\/([^/]*)\\/OutPut\\/out-put\\.pdf$/)`,
+      errors: [{ message: 'Route path should be in kebab-case.' }],
+    },
+  ],
+})
+
+ruleTester.run('no-unnecessary-trans', noUnnecessaryTrans, {
+  valid: [
+    { code: `<Trans i18nKey="test" components={{ strong: <strong/> }}/>` },
+  ],
+  invalid: [
+    {
+      code: `<Trans i18nKey="test" values={{ test: 'foo '}}/>`,
+      errors: [{ message: `Use t('…') when there are no components` }],
+    },
+    {
+      code: `<Trans i18nKey="test" />`,
+      errors: [{ message: `Use t('…') when there are no components` }],
+      output: `{t('test')}`,
+    },
+  ],
+})
+
+ruleTester.run('should-unescape-trans', shouldUnescapeTrans, {
+  valid: [
+    {
+      code: `<Trans i18nKey="test" components={{ strong: <strong/> }}/>`,
+    },
+    {
+      code: `<Trans i18nKey="test" values={{ foo: 'bar' }} components={{ strong: <strong/> }} shouldUnescape tOptions={{ interpolation: { escapeValue: true } }}/>`,
+    },
+  ],
+  invalid: [
+    {
+      code: `<Trans i18nKey="test" values={{ foo: 'bar' }} components={{ strong: <strong/> }} />`,
+      errors: [{ message: 'Trans with values must have shouldUnescape' }],
+      output: `<Trans i18nKey="test" values={{ foo: 'bar' }}\nshouldUnescape components={{ strong: <strong/> }} />`,
+    },
+    {
+      code: `<Trans i18nKey="test" values={{ foo: 'bar' }} components={{ strong: <strong/> }} shouldUnescape />`,
+      errors: [
+        {
+          message:
+            'Trans with shouldUnescape must have tOptions.interpolation.escapeValue',
+        },
+      ],
+      output: `<Trans i18nKey="test" values={{ foo: 'bar' }} components={{ strong: <strong/> }} shouldUnescape\ntOptions={{ interpolation: { escapeValue: true } }} />`,
+    },
+  ],
+})
+
+const noGeneratedEditorThemesError =
+  'EditorView.theme and EditorView.baseTheme each add CSS to the page for every instance of the theme. Store the theme in a variable and reuse it instead.'
+ruleTester.run('no-generated-editor-themes', noGeneratedEditorThemes, {
+  valid: [
+    {
+      code: `EditorView.theme({ '.cm-editor': { color: 'black' } })`,
+    },
+    {
+      code: `const theme = EditorView.theme({ '.cm-editor': { color: 'black' } })`,
+    },
+  ],
+  invalid: [
+    {
+      code: `function createTheme() { return EditorView.theme({ '.cm-editor': { color: 'black' } }) }`,
+      errors: [
+        {
+          message: noGeneratedEditorThemesError,
+        },
+      ],
+    },
+    {
+      code: `() => EditorView.theme({ '.cm-editor': { color: 'black' } })`,
+      errors: [
+        {
+          message: noGeneratedEditorThemesError,
+        },
+      ],
+    },
+    {
+      code: `class Foo { createTheme() { return EditorView.theme({ '.cm-editor': { color: 'black' } }) } }`,
+      errors: [
+        {
+          message: noGeneratedEditorThemesError,
+        },
+      ],
+    },
+  ],
+})
+
+ruleTester.run('domock-require-valid-path', viDoMockValidPath, {
+  valid: [
+    {
+      code: 'vi.doMock("./require-vi-doMock-valid-path.js")',
+      filename: __filename,
+    },
+    {
+      code: 'const filename = "./require-vi-doMock-valid-path.js"; vi.doMock(filename);',
+      filename: __filename,
+    },
+  ],
+  invalid: [
+    {
+      code: "vi.doMock('./require-vi-doMock-valid-path2')",
+      filename: __filename,
+      errors: [
+        {
+          message:
+            'The path "./require-vi-doMock-valid-path2" in vi.doMock() cannot be resolved relative to the current file.',
+        },
+      ],
+    },
+    {
+      code: 'const filename = "./require-vi-doMock-valid-path2.js"; vi.doMock(filename);',
+      filename: __filename,
+      errors: [
+        {
+          message:
+            'The first argument of vi.doMock() must be (or resolve to) a string literal representing a path.',
+        },
+      ],
+    },
+  ],
+})

libraries/eslint-plugin/should-unescape-trans.js

@@ -0,0 +1,60 @@
+module.exports = {
+  meta: {
+    type: 'problem',
+    fixable: 'code',
+    docs: {
+      description: 'Ensure that Trans with values has shouldUnescape',
+    },
+  },
+  create(context) {
+    return {
+      'JSXOpeningElement[name.name="Trans"]'(node) {
+        const attributes = new Map(
+          node.attributes.map(attr => [attr.name.name, attr])
+        )
+
+        if (attributes.has('values') && !attributes.has('shouldUnescape')) {
+          context.report({
+            node,
+            message: 'Trans with values must have shouldUnescape',
+            fix(fixer) {
+              return fixer.insertTextAfter(
+                attributes.get('values'),
+                '\nshouldUnescape' // Note: Prettier can fix indentation
+              )
+            },
+          })
+        }
+
+        if (attributes.has('values') && attributes.has('shouldUnescape')) {
+          const tOptions = attributes.get('tOptions')
+          if (!tOptions) {
+            context.report({
+              node,
+              message:
+                'Trans with shouldUnescape must have tOptions.interpolation.escapeValue',
+              fix(fixer) {
+                return fixer.insertTextAfter(
+                  attributes.get('shouldUnescape'),
+                  '\ntOptions={{ interpolation: { escapeValue: true } }}' // Note: Prettier can fix indentation
+                )
+              },
+            })
+          } else {
+            const property = tOptions.value.expression.properties
+              .find(p => p.key.name === 'interpolation')
+              ?.value.properties.find(p => p.key.name === 'escapeValue')
+
+            if (property?.value.value !== true) {
+              context.report({
+                node,
+                message:
+                  'Trans with shouldUnescape must have tOptions.interpolation.escapeValue set to true',
+              })
+            }
+          }
+        }
+      },
+    }
+  },
+}

libraries/eslint-plugin/tsconfig.json

@@ -0,0 +1,6 @@
+{
+  "extends": "../../tsconfig.backend.json",
+  "include": [
+    "**/*.js",
+  ]
+}

libraries/fetch-utils/.jenkinsIncludeFile

@@ -1,8 +0,0 @@
-# Autogenerated by build scripts. Do not edit.
-.eslint*
-.prettier*
-libraries/fetch-utils/**
-libraries/o-error/**
-package-lock.json
-package.json
-patches/**

libraries/fetch-utils/Jenkinsfile

@@ -1,82 +0,0 @@
-// Autogenerated by build scripts. Do not edit.
-
-pipeline {
-  agent {
-    node {
-      label 'jenkins-agent-web'
-      customWorkspace '/workspace'
-    }
-  }
-  options {
-    timestamps()
-    timeout(time: 15, unit: 'MINUTES')
-  }
-  environment {
-    BRANCH_NAME = "${env.CHANGE_BRANCH ? env.CHANGE_BRANCH : env.BRANCH_NAME}"
-    JENKINS_BUILD_NUMBER = "${BUILD_NUMBER}"
-    BUILD_NUMBER = "${SHORT_SHA}_${BUILD_NUMBER}"
-    COMMIT_SHA = "${GIT_COMMIT}"
-    SHORT_SHA = "${GIT_COMMIT.take(7)}"
-  }
-  stages {
-    stage('Stage 1') {
-      parallel {
-        stage('Install monorepo') {
-          steps {
-            retry(count: 3) {
-              sh 'make monorepo_setup'
-            }
-          }
-        }
-        stage('Create reports folder') {
-          steps {
-            sh 'mkdir libraries/fetch-utils/reports'
-          }
-        }
-      }
-    }
-    stage('Stage 2') {
-      parallel {
-        stage('Lint') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/fetch-utils monorepo npm run lint -- --format json --output-file reports/eslint.json'
-          }
-          post {
-            always {
-              sh """
-                sed -i 's_"filePath":"/overleaf_"filePath":"/workspace_g' libraries/fetch-utils/reports/eslint.json
-              """
-              recordIssues checksAnnotationScope: 'ALL', enabledForFailure: true, failOnError: true, id: 'fetch-utils-eslint', name: 'fetch-utils eslint', qualityGates: [[integerThreshold: 1, threshold: 1.0, type: 'TOTAL']], sourceCodeRetention: 'LAST_BUILD', tools: [esLint(pattern: 'libraries/fetch-utils/reports/eslint.json')]
-            }
-          }
-        }
-        stage('Format') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/fetch-utils monorepo npm run format'
-          }
-        }
-        stage('Typecheck') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/fetch-utils monorepo npm run types:check'
-          }
-        }
-        stage('Test') {
-          steps {
-            retry(count: 3) {
-              sh 'bin/run -w /overleaf/libraries/fetch-utils monorepo npm run test:ci'
-            }
-          }
-        }
-      }
-    }
-  }
-  post {
-    always {
-      junit checksName: 'fetch-utils test results', testResults: 'libraries/fetch-utils/reports/junit-*.xml'
-    }
-    cleanup {
-      sh 'rm -rf libraries/fetch-utils/reports'
-      sh 'make clean_jenkins -j10'
-    }
-  }
-}

libraries/fetch-utils/buildscript.txt

@@ -5,4 +5,5 @@ fetch-utils
 --esmock-loader=False
 --is-library=True
 --node-version=22.18.0
+--pipeline-owner=🚉 Platform
 --public-repo=False

libraries/fetch-utils/package.json

@@ -17,10 +17,10 @@
   "license": "AGPL-3.0-only",
   "devDependencies": {
     "@types/node-fetch": "^2.6.13",
-    "body-parser": "^1.20.3",
+    "body-parser": "1.20.4",
     "chai": "^4.3.6",
     "chai-as-promised": "^7.1.1",
-    "express": "^4.21.2",
+    "express": "4.22.1",
     "mocha": "^11.1.0",
     "mocha-junit-reporter": "^2.2.1",
     "mocha-multi-reporters": "^1.5.1",

libraries/fetch-utils/test/mocha-multi-reporters.cjs

@@ -4,5 +4,6 @@ module.exports = {
     mochaFile: `reports/junit-mocha-${process.env.MOCHA_GREP}.xml`,
     includePending: true,
     jenkinsMode: true,
+    output: true,
   },
 }

libraries/logger/.jenkinsIncludeFile

@@ -1,9 +0,0 @@
-# Autogenerated by build scripts. Do not edit.
-.eslint*
-.prettier*
-libraries/fetch-utils/**
-libraries/logger/**
-libraries/o-error/**
-package-lock.json
-package.json
-patches/**

libraries/logger/Jenkinsfile

@@ -1,82 +0,0 @@
-// Autogenerated by build scripts. Do not edit.
-
-pipeline {
-  agent {
-    node {
-      label 'jenkins-agent-web'
-      customWorkspace '/workspace'
-    }
-  }
-  options {
-    timestamps()
-    timeout(time: 15, unit: 'MINUTES')
-  }
-  environment {
-    BRANCH_NAME = "${env.CHANGE_BRANCH ? env.CHANGE_BRANCH : env.BRANCH_NAME}"
-    JENKINS_BUILD_NUMBER = "${BUILD_NUMBER}"
-    BUILD_NUMBER = "${SHORT_SHA}_${BUILD_NUMBER}"
-    COMMIT_SHA = "${GIT_COMMIT}"
-    SHORT_SHA = "${GIT_COMMIT.take(7)}"
-  }
-  stages {
-    stage('Stage 1') {
-      parallel {
-        stage('Install monorepo') {
-          steps {
-            retry(count: 3) {
-              sh 'make monorepo_setup'
-            }
-          }
-        }
-        stage('Create reports folder') {
-          steps {
-            sh 'mkdir libraries/logger/reports'
-          }
-        }
-      }
-    }
-    stage('Stage 2') {
-      parallel {
-        stage('Lint') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/logger monorepo npm run lint -- --format json --output-file reports/eslint.json'
-          }
-          post {
-            always {
-              sh """
-                sed -i 's_"filePath":"/overleaf_"filePath":"/workspace_g' libraries/logger/reports/eslint.json
-              """
-              recordIssues checksAnnotationScope: 'ALL', enabledForFailure: true, failOnError: true, id: 'logger-eslint', name: 'logger eslint', qualityGates: [[integerThreshold: 1, threshold: 1.0, type: 'TOTAL']], sourceCodeRetention: 'LAST_BUILD', tools: [esLint(pattern: 'libraries/logger/reports/eslint.json')]
-            }
-          }
-        }
-        stage('Format') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/logger monorepo npm run format'
-          }
-        }
-        stage('Typecheck') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/logger monorepo npm run types:check'
-          }
-        }
-        stage('Test') {
-          steps {
-            retry(count: 3) {
-              sh 'bin/run -w /overleaf/libraries/logger monorepo npm run test:ci'
-            }
-          }
-        }
-      }
-    }
-  }
-  post {
-    always {
-      junit checksName: 'logger test results', testResults: 'libraries/logger/reports/junit-*.xml'
-    }
-    cleanup {
-      sh 'rm -rf libraries/logger/reports'
-      sh 'make clean_jenkins -j10'
-    }
-  }
-}

libraries/logger/buildscript.txt

@@ -5,4 +5,5 @@ logger
 --esmock-loader=False
 --is-library=True
 --node-version=22.18.0
+--pipeline-owner=🚉 Platform
 --public-repo=False

libraries/logger/test-log-recorder.js

@@ -0,0 +1,20 @@
+const logger = require('./')
+const bunyan = require('bunyan')
+const serializers = require('./serializers')
+
+function testLogRecorder() {
+  const currentTest = this.currentTest
+  for (const level of ['error', 'fatal']) {
+    logger[level] = (info, msg) => {
+      const entry = { level, ...info, msg }
+      for (const [name, fn] of Object.entries(serializers)) {
+        if (name in entry) entry[name] = fn(entry[name])
+      }
+      currentTest.consoleErrors = (currentTest.consoleErrors || []).concat(
+        JSON.stringify(info, bunyan.safeCycles())
+      )
+    }
+  }
+}
+
+module.exports = testLogRecorder

libraries/logger/test/mocha-multi-reporters.cjs

@@ -4,5 +4,6 @@ module.exports = {
     mochaFile: `reports/junit-mocha-${process.env.MOCHA_GREP}.xml`,
     includePending: true,
     jenkinsMode: true,
+    output: true,
   },
 }

libraries/metrics/.jenkinsIncludeFile

@@ -1,7 +0,0 @@
-# Autogenerated by build scripts. Do not edit.
-.eslint*
-.prettier*
-libraries/metrics/**
-package-lock.json
-package.json
-patches/**

libraries/metrics/Jenkinsfile

@@ -1,82 +0,0 @@
-// Autogenerated by build scripts. Do not edit.
-
-pipeline {
-  agent {
-    node {
-      label 'jenkins-agent-web'
-      customWorkspace '/workspace'
-    }
-  }
-  options {
-    timestamps()
-    timeout(time: 15, unit: 'MINUTES')
-  }
-  environment {
-    BRANCH_NAME = "${env.CHANGE_BRANCH ? env.CHANGE_BRANCH : env.BRANCH_NAME}"
-    JENKINS_BUILD_NUMBER = "${BUILD_NUMBER}"
-    BUILD_NUMBER = "${SHORT_SHA}_${BUILD_NUMBER}"
-    COMMIT_SHA = "${GIT_COMMIT}"
-    SHORT_SHA = "${GIT_COMMIT.take(7)}"
-  }
-  stages {
-    stage('Stage 1') {
-      parallel {
-        stage('Install monorepo') {
-          steps {
-            retry(count: 3) {
-              sh 'make monorepo_setup'
-            }
-          }
-        }
-        stage('Create reports folder') {
-          steps {
-            sh 'mkdir libraries/metrics/reports'
-          }
-        }
-      }
-    }
-    stage('Stage 2') {
-      parallel {
-        stage('Lint') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/metrics monorepo npm run lint -- --format json --output-file reports/eslint.json'
-          }
-          post {
-            always {
-              sh """
-                sed -i 's_"filePath":"/overleaf_"filePath":"/workspace_g' libraries/metrics/reports/eslint.json
-              """
-              recordIssues checksAnnotationScope: 'ALL', enabledForFailure: true, failOnError: true, id: 'metrics-eslint', name: 'metrics eslint', qualityGates: [[integerThreshold: 1, threshold: 1.0, type: 'TOTAL']], sourceCodeRetention: 'LAST_BUILD', tools: [esLint(pattern: 'libraries/metrics/reports/eslint.json')]
-            }
-          }
-        }
-        stage('Format') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/metrics monorepo npm run format'
-          }
-        }
-        stage('Typecheck') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/metrics monorepo npm run types:check'
-          }
-        }
-        stage('Test') {
-          steps {
-            retry(count: 3) {
-              sh 'bin/run -w /overleaf/libraries/metrics monorepo npm run test:ci'
-            }
-          }
-        }
-      }
-    }
-  }
-  post {
-    always {
-      junit checksName: 'metrics test results', testResults: 'libraries/metrics/reports/junit-*.xml'
-    }
-    cleanup {
-      sh 'rm -rf libraries/metrics/reports'
-      sh 'make clean_jenkins -j10'
-    }
-  }
-}

libraries/metrics/buildscript.txt

@@ -5,4 +5,5 @@ metrics
 --esmock-loader=False
 --is-library=True
 --node-version=22.18.0
+--pipeline-owner=🚉 Platform
 --public-repo=False

libraries/metrics/index.js

@@ -97,6 +97,12 @@ function histogram(key, value, buckets, labels = {}) {
 }
 
 class Timer {
+  /**
+   * @param {string} key
+   * @param {number} sampleRate
+   * @param {Record<string, any>} labels
+   * @param {Array<number>} buckets
+   */
   constructor(key, sampleRate = 1, labels = {}, buckets = undefined) {
     if (typeof sampleRate === 'object') {
       // called with (key, labels, buckets)

libraries/metrics/test/mocha-multi-reporters.cjs

@@ -4,5 +4,6 @@ module.exports = {
     mochaFile: `reports/junit-mocha-${process.env.MOCHA_GREP}.xml`,
     includePending: true,
     jenkinsMode: true,
+    output: true,
   },
 }

libraries/mongo-utils/.jenkinsIncludeFile

@@ -1,7 +0,0 @@
-# Autogenerated by build scripts. Do not edit.
-.eslint*
-.prettier*
-libraries/mongo-utils/**
-package-lock.json
-package.json
-patches/**

libraries/mongo-utils/Jenkinsfile

@@ -1,72 +0,0 @@
-// Autogenerated by build scripts. Do not edit.
-
-pipeline {
-  agent {
-    node {
-      label 'jenkins-agent-web'
-      customWorkspace '/workspace'
-    }
-  }
-  options {
-    timestamps()
-    timeout(time: 15, unit: 'MINUTES')
-  }
-  environment {
-    BRANCH_NAME = "${env.CHANGE_BRANCH ? env.CHANGE_BRANCH : env.BRANCH_NAME}"
-    JENKINS_BUILD_NUMBER = "${BUILD_NUMBER}"
-    BUILD_NUMBER = "${SHORT_SHA}_${BUILD_NUMBER}"
-    COMMIT_SHA = "${GIT_COMMIT}"
-    SHORT_SHA = "${GIT_COMMIT.take(7)}"
-  }
-  stages {
-    stage('Stage 1') {
-      parallel {
-        stage('Install monorepo') {
-          steps {
-            retry(count: 3) {
-              sh 'make monorepo_setup'
-            }
-          }
-        }
-        stage('Create reports folder') {
-          steps {
-            sh 'mkdir libraries/mongo-utils/reports'
-          }
-        }
-      }
-    }
-    stage('Stage 2') {
-      parallel {
-        stage('Lint') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/mongo-utils monorepo npm run lint -- --format json --output-file reports/eslint.json'
-          }
-          post {
-            always {
-              sh """
-                sed -i 's_"filePath":"/overleaf_"filePath":"/workspace_g' libraries/mongo-utils/reports/eslint.json
-              """
-              recordIssues checksAnnotationScope: 'ALL', enabledForFailure: true, failOnError: true, id: 'mongo-utils-eslint', name: 'mongo-utils eslint', qualityGates: [[integerThreshold: 1, threshold: 1.0, type: 'TOTAL']], sourceCodeRetention: 'LAST_BUILD', tools: [esLint(pattern: 'libraries/mongo-utils/reports/eslint.json')]
-            }
-          }
-        }
-        stage('Format') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/mongo-utils monorepo npm run format'
-          }
-        }
-        stage('Typecheck') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/mongo-utils monorepo npm run types:check'
-          }
-        }
-      }
-    }
-  }
-  post {
-    cleanup {
-      sh 'rm -rf libraries/mongo-utils/reports'
-      sh 'make clean_jenkins -j10'
-    }
-  }
-}

libraries/mongo-utils/buildscript.txt

@@ -5,4 +5,5 @@ mongo-utils
 --esmock-loader=False
 --is-library=True
 --node-version=22.18.0
+--pipeline-owner=🚉 Platform
 --public-repo=False

libraries/o-error/.jenkinsIncludeFile

@@ -1,7 +0,0 @@
-# Autogenerated by build scripts. Do not edit.
-.eslint*
-.prettier*
-libraries/o-error/**
-package-lock.json
-package.json
-patches/**

libraries/o-error/Jenkinsfile

@@ -1,82 +0,0 @@
-// Autogenerated by build scripts. Do not edit.
-
-pipeline {
-  agent {
-    node {
-      label 'jenkins-agent-web'
-      customWorkspace '/workspace'
-    }
-  }
-  options {
-    timestamps()
-    timeout(time: 15, unit: 'MINUTES')
-  }
-  environment {
-    BRANCH_NAME = "${env.CHANGE_BRANCH ? env.CHANGE_BRANCH : env.BRANCH_NAME}"
-    JENKINS_BUILD_NUMBER = "${BUILD_NUMBER}"
-    BUILD_NUMBER = "${SHORT_SHA}_${BUILD_NUMBER}"
-    COMMIT_SHA = "${GIT_COMMIT}"
-    SHORT_SHA = "${GIT_COMMIT.take(7)}"
-  }
-  stages {
-    stage('Stage 1') {
-      parallel {
-        stage('Install monorepo') {
-          steps {
-            retry(count: 3) {
-              sh 'make monorepo_setup'
-            }
-          }
-        }
-        stage('Create reports folder') {
-          steps {
-            sh 'mkdir libraries/o-error/reports'
-          }
-        }
-      }
-    }
-    stage('Stage 2') {
-      parallel {
-        stage('Lint') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/o-error monorepo npm run lint -- --format json --output-file reports/eslint.json'
-          }
-          post {
-            always {
-              sh """
-                sed -i 's_"filePath":"/overleaf_"filePath":"/workspace_g' libraries/o-error/reports/eslint.json
-              """
-              recordIssues checksAnnotationScope: 'ALL', enabledForFailure: true, failOnError: true, id: 'o-error-eslint', name: 'o-error eslint', qualityGates: [[integerThreshold: 1, threshold: 1.0, type: 'TOTAL']], sourceCodeRetention: 'LAST_BUILD', tools: [esLint(pattern: 'libraries/o-error/reports/eslint.json')]
-            }
-          }
-        }
-        stage('Format') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/o-error monorepo npm run format'
-          }
-        }
-        stage('Typecheck') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/o-error monorepo npm run types:check'
-          }
-        }
-        stage('Test') {
-          steps {
-            retry(count: 3) {
-              sh 'bin/run -w /overleaf/libraries/o-error monorepo npm run test:ci'
-            }
-          }
-        }
-      }
-    }
-  }
-  post {
-    always {
-      junit checksName: 'o-error test results', testResults: 'libraries/o-error/reports/junit-*.xml'
-    }
-    cleanup {
-      sh 'rm -rf libraries/o-error/reports'
-      sh 'make clean_jenkins -j10'
-    }
-  }
-}

libraries/o-error/buildscript.txt

@@ -5,4 +5,5 @@ o-error
 --esmock-loader=False
 --is-library=True
 --node-version=22.18.0
+--pipeline-owner=🚉 Platform
 --public-repo=False

libraries/o-error/test/mocha-multi-reporters.cjs

@@ -4,5 +4,6 @@ module.exports = {
     mochaFile: `reports/junit-mocha-${process.env.MOCHA_GREP}.xml`,
     includePending: true,
     jenkinsMode: true,
+    output: true,
   },
 }

libraries/object-persistor/.jenkinsIncludeFile

@@ -1,12 +0,0 @@
-# Autogenerated by build scripts. Do not edit.
-.eslint*
-.prettier*
-libraries/fetch-utils/**
-libraries/logger/**
-libraries/metrics/**
-libraries/o-error/**
-libraries/object-persistor/**
-libraries/stream-utils/**
-package-lock.json
-package.json
-patches/**

libraries/object-persistor/Jenkinsfile

@@ -1,82 +0,0 @@
-// Autogenerated by build scripts. Do not edit.
-
-pipeline {
-  agent {
-    node {
-      label 'jenkins-agent-web'
-      customWorkspace '/workspace'
-    }
-  }
-  options {
-    timestamps()
-    timeout(time: 15, unit: 'MINUTES')
-  }
-  environment {
-    BRANCH_NAME = "${env.CHANGE_BRANCH ? env.CHANGE_BRANCH : env.BRANCH_NAME}"
-    JENKINS_BUILD_NUMBER = "${BUILD_NUMBER}"
-    BUILD_NUMBER = "${SHORT_SHA}_${BUILD_NUMBER}"
-    COMMIT_SHA = "${GIT_COMMIT}"
-    SHORT_SHA = "${GIT_COMMIT.take(7)}"
-  }
-  stages {
-    stage('Stage 1') {
-      parallel {
-        stage('Install monorepo') {
-          steps {
-            retry(count: 3) {
-              sh 'make monorepo_setup'
-            }
-          }
-        }
-        stage('Create reports folder') {
-          steps {
-            sh 'mkdir libraries/object-persistor/reports'
-          }
-        }
-      }
-    }
-    stage('Stage 2') {
-      parallel {
-        stage('Lint') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/object-persistor monorepo npm run lint -- --format json --output-file reports/eslint.json'
-          }
-          post {
-            always {
-              sh """
-                sed -i 's_"filePath":"/overleaf_"filePath":"/workspace_g' libraries/object-persistor/reports/eslint.json
-              """
-              recordIssues checksAnnotationScope: 'ALL', enabledForFailure: true, failOnError: true, id: 'object-persistor-eslint', name: 'object-persistor eslint', qualityGates: [[integerThreshold: 1, threshold: 1.0, type: 'TOTAL']], sourceCodeRetention: 'LAST_BUILD', tools: [esLint(pattern: 'libraries/object-persistor/reports/eslint.json')]
-            }
-          }
-        }
-        stage('Format') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/object-persistor monorepo npm run format'
-          }
-        }
-        stage('Typecheck') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/object-persistor monorepo npm run types:check'
-          }
-        }
-        stage('Test') {
-          steps {
-            retry(count: 3) {
-              sh 'bin/run -w /overleaf/libraries/object-persistor monorepo npm run test:ci'
-            }
-          }
-        }
-      }
-    }
-  }
-  post {
-    always {
-      junit checksName: 'object-persistor test results', testResults: 'libraries/object-persistor/reports/junit-*.xml'
-    }
-    cleanup {
-      sh 'rm -rf libraries/object-persistor/reports'
-      sh 'make clean_jenkins -j10'
-    }
-  }
-}

libraries/object-persistor/buildscript.txt

@@ -5,4 +5,5 @@ object-persistor
 --esmock-loader=False
 --is-library=True
 --node-version=22.18.0
+--pipeline-owner=🚉 Platform
 --public-repo=False

libraries/object-persistor/package.json

@@ -30,7 +30,7 @@
     "@overleaf/o-error": "*",
     "@overleaf/stream-utils": "*",
     "fast-crc32c": "overleaf/node-fast-crc32c#aae6b2a4c7a7a159395df9cc6c38dfde702d6f51",
-    "glob": "^7.1.6",
+    "glob": "^12.0.0",
     "range-parser": "^1.2.1",
     "tiny-async-pool": "^1.1.0"
   },

libraries/object-persistor/src/AbstractPersistor.js

@@ -160,4 +160,38 @@ module.exports = class AbstractPersistor {
       name,
     })
   }
+
+  /**
+   * List objects in a directory, returning the full keys.
+   *
+   * Suitable only for directories where the number of keys is known to be small.
+   *
+   * @param {string} location
+   * @param {string} prefix
+   * @returns {Promise<Array<string>>}
+   */
+  async listDirectoryKeys(location, prefix) {
+    throw new NotImplementedError('method not implemented in persistor', {
+      method: 'listDirectoryKeys',
+      location,
+      prefix,
+    })
+  }
+
+  /**
+   * List objects in a directory, returning key and size information.
+   *
+   * Suitable only for directories where the number of keys is known to be small.
+   *
+   * @param {string} location
+   * @param {string} prefix
+   * @returns {Promise<Array<{key: string, size: number}>>}
+   */
+  async listDirectoryStats(location, prefix) {
+    throw new NotImplementedError('method not implemented in persistor', {
+      method: 'listDirectoryStats',
+      location,
+      prefix,
+    })
+  }
 }

libraries/object-persistor/src/FSPersistor.js

@@ -1,18 +1,15 @@
 const crypto = require('node:crypto')
 const fs = require('node:fs')
 const fsPromises = require('node:fs/promises')
-const globCallbacks = require('glob')
+const { glob } = require('glob')
 const Path = require('node:path')
 const { PassThrough } = require('node:stream')
 const { pipeline } = require('node:stream/promises')
-const { promisify } = require('node:util')
 
 const AbstractPersistor = require('./AbstractPersistor')
 const { ReadError, WriteError, NotImplementedError } = require('./Errors')
 const PersistorHelper = require('./PersistorHelper')
 
-const glob = promisify(globCallbacks)
-
 module.exports = class FSPersistor extends AbstractPersistor {
   constructor(settings = {}) {
     if (settings.storageClass) {
@@ -197,6 +194,50 @@ module.exports = class FSPersistor extends AbstractPersistor {
     }
   }
 
+  async listDirectoryKeys(location, name) {
+    const fsPath = this._getFsPath(location, name)
+    const paths = await this._listDirectory(fsPath)
+
+    // Filter to only return files, not directories
+    const files = []
+    for (const path of paths) {
+      try {
+        const stat = await fsPromises.stat(path)
+        if (stat.isFile()) {
+          files.push(path)
+        }
+      } catch (err) {
+        // ignore files that may have just been deleted
+        if (err.code !== 'ENOENT') {
+          throw err
+        }
+      }
+    }
+    return files
+  }
+
+  async listDirectoryStats(location, name) {
+    const fsPath = this._getFsPath(location, name)
+    const paths = await this._listDirectory(fsPath)
+
+    // Filter to only return files, not directories, with their sizes
+    const stats = []
+    for (const path of paths) {
+      try {
+        const stat = await fsPromises.stat(path)
+        if (stat.isFile()) {
+          stats.push({ key: path, size: stat.size })
+        }
+      } catch (err) {
+        // ignore files that may have just been deleted
+        if (err.code !== 'ENOENT') {
+          throw err
+        }
+      }
+    }
+    return stats
+  }
+
   async checkIfObjectExists(location, name) {
     const fsPath = this._getFsPath(location, name)
     try {
@@ -305,10 +346,8 @@ module.exports = class FSPersistor extends AbstractPersistor {
 
   async _listDirectory(path) {
     if (this.useSubdirectories) {
-      // eslint-disable-next-line @typescript-eslint/return-await
       return await glob(Path.join(path, '**'))
     } else {
-      // eslint-disable-next-line @typescript-eslint/return-await
       return await glob(`${path}_*`)
     }
   }

libraries/object-persistor/src/GcsPersistor.js

@@ -288,23 +288,25 @@ module.exports = class GcsPersistor extends AbstractPersistor {
     } while (query)
   }
 
-  async directorySize(bucketName, key) {
-    let files
-    const prefix = ensurePrefixIsDirectory(key)
-
+  async #listDirectory(bucketName, prefix) {
     try {
       const [response] = await this.storage
         .bucket(bucketName)
         .getFiles({ prefix })
-      files = response
+      return response
     } catch (err) {
       throw PersistorHelper.wrapError(
         err,
         'failed to list objects in GCS',
-        { bucketName, key },
+        { bucketName, prefix },
         ReadError
       )
     }
+  }
+
+  async directorySize(bucketName, key) {
+    const prefix = ensurePrefixIsDirectory(key)
+    const files = await this.#listDirectory(bucketName, prefix)
 
     return files.reduce(
       (acc, file) => parseInt(file.metadata.size, 10) + acc,
@@ -312,6 +314,25 @@ module.exports = class GcsPersistor extends AbstractPersistor {
     )
   }
 
+  async listDirectoryKeys(bucketName, prefix) {
+    const files = await this.#listDirectory(
+      bucketName,
+      ensurePrefixIsDirectory(prefix)
+    )
+    return files.map(file => file.name)
+  }
+
+  async listDirectoryStats(bucketName, prefix) {
+    const files = await this.#listDirectory(
+      bucketName,
+      ensurePrefixIsDirectory(prefix)
+    )
+    return files.map(file => ({
+      key: file.name,
+      size: parseInt(file.metadata.size, 10),
+    }))
+  }
+
   async checkIfObjectExists(bucketName, key) {
     try {
       const [response] = await this.storage

libraries/object-persistor/src/PerProjectEncryptedS3Persistor.js

@@ -417,10 +417,20 @@ class CachedPerProjectEncryptedS3Persistor {
    *
    * @param {string} bucketName
    * @param {string} path
-   * @return {Promise<ListDirectoryResult>}
+   * @return {Promise<string[]>}
    */
-  async listDirectory(bucketName, path) {
-    return await this.#parent.listDirectory(bucketName, path)
+  async listDirectoryKeys(bucketName, path) {
+    return await this.#parent.listDirectoryKeys(bucketName, path)
+  }
+
+  /**
+   *
+   * @param {string} bucketName
+   * @param {string} path
+   * @return {Promise<Array<{key: string, size: number}>>}
+   */
+  async listDirectoryStats(bucketName, path) {
+    return await this.#parent.listDirectoryStats(bucketName, path)
   }
 
   /**

libraries/object-persistor/src/ProjectKey.js

@@ -1,4 +1,3 @@
-// Keep in sync with services/filestore/app/js/project_key.js
 const path = require('node:path')
 
 //
@@ -21,3 +20,4 @@ function naiveReverse(string) {
 
 exports.format = format
 exports.pad = pad
+exports.naiveReverse = naiveReverse

libraries/object-persistor/src/S3Persistor.js

@@ -266,7 +266,7 @@ class S3Persistor extends AbstractPersistor {
    * @return {Promise<void>}
    */
   async deleteDirectory(bucketName, key, continuationToken) {
-    const { contents, response } = await this.listDirectory(
+    const { contents, response } = await this.#listDirectory(
       bucketName,
       key,
       continuationToken
@@ -309,7 +309,7 @@ class S3Persistor extends AbstractPersistor {
    * @param {string} [continuationToken]
    * @return {Promise<ListDirectoryResult>}
    */
-  async listDirectory(bucketName, key, continuationToken) {
+  async #listDirectory(bucketName, key, continuationToken) {
     let response
     const options = { Bucket: bucketName, Prefix: key }
     if (continuationToken) {
@@ -535,6 +535,59 @@ class S3Persistor extends AbstractPersistor {
     }
   }
 
+  /**
+   * @param {string} bucketName
+   * @param {string} prefix
+   * @return {Promise<Array<string>>}
+   */
+  async listDirectoryKeys(bucketName, prefix) {
+    const keys = []
+    let continuationToken
+
+    do {
+      const { contents, response } = await this.#listDirectory(
+        bucketName,
+        prefix,
+        continuationToken
+      )
+      keys.push(...contents.map(item => item.Key || ''))
+      continuationToken = response.IsTruncated
+        ? response.NextContinuationToken
+        : undefined
+    } while (continuationToken)
+
+    return keys
+  }
+
+  /**
+   * @param {string} bucketName
+   * @param {string} prefix
+   * @return {Promise<Array<{key: string, size: number}>>}
+   */
+  async listDirectoryStats(bucketName, prefix) {
+    const stats = []
+    let continuationToken
+
+    do {
+      const { contents, response } = await this.#listDirectory(
+        bucketName,
+        prefix,
+        continuationToken
+      )
+      stats.push(
+        ...contents.map(item => ({
+          key: item.Key || '',
+          size: item.Size ?? -1,
+        }))
+      )
+      continuationToken = response.IsTruncated
+        ? response.NextContinuationToken
+        : undefined
+    } while (continuationToken)
+
+    return stats
+  }
+
   /**
    * @param {string} bucket
    * @return {S3Client}

libraries/object-persistor/test/mocha-multi-reporters.cjs

@@ -4,5 +4,6 @@ module.exports = {
     mochaFile: `reports/junit-mocha-${process.env.MOCHA_GREP}.xml`,
     includePending: true,
     jenkinsMode: true,
+    output: true,
   },
 }

libraries/object-persistor/test/setup.js

@@ -27,7 +27,7 @@ SandboxedModule.configure({
   globals: { Buffer, Math, console, process, URL },
   sourceTransformers: {
     removeNodePrefix: function (source) {
-      return source.replace(/require\(['"]node:/g, "require('")
+      return source.replace(/require\((['"])node:/g, 'require($1')
     },
   },
 })

libraries/object-persistor/test/unit/FSPersistorTests.js

@@ -400,6 +400,56 @@ describe('FSPersistorTests', function () {
           ).to.equal(0)
         })
       })
+
+      describe('listDirectoryKeys', function () {
+        beforeEach(async function () {
+          for (const file of Object.values(files)) {
+            await persistor.sendFile(location, file, '/uploads/info.txt')
+          }
+        })
+
+        it('should list directory keys', async function () {
+          const keys = await persistor.listDirectoryKeys(location, 'animals')
+          expect(keys).to.have.lengthOf(2)
+          expect(keys).to.include(scenario.fsPath(files.wombat))
+          expect(keys).to.include(scenario.fsPath(files.giraffe))
+        })
+
+        it('should return empty array for non-existing directories', async function () {
+          const keys = await persistor.listDirectoryKeys(
+            location,
+            'does-not-exist'
+          )
+          expect(keys).to.deep.equal([])
+        })
+      })
+
+      describe('listDirectoryStats', function () {
+        beforeEach(async function () {
+          for (const file of Object.values(files)) {
+            await persistor.sendFile(location, file, '/uploads/info.txt')
+          }
+        })
+
+        it('should list directory stats', async function () {
+          const stats = await persistor.listDirectoryStats(location, 'animals')
+          expect(stats).to.have.lengthOf(2)
+          const keys = stats.map(s => s.key)
+          expect(keys).to.include(scenario.fsPath(files.wombat))
+          expect(keys).to.include(scenario.fsPath(files.giraffe))
+          for (const stat of stats) {
+            expect(stat.size).to.equal(localFiles['/uploads/info.txt'].length)
+          }
+        })
+
+        it('should return empty array for non-existing directories', async function () {
+          const stats = await persistor.listDirectoryStats(
+            location,
+            'does-not-exist'
+          )
+          expect(stats).to.deep.equal([])
+        })
+      })
     })
   }
 })

libraries/object-persistor/test/unit/GcsPersistorTests.js

@@ -723,4 +723,127 @@ describe('GcsPersistorTests', function () {
       })
     })
   })
+
+  describe('listDirectoryKeys', function () {
+    describe('with valid parameters', function () {
+      let keys
+
+      beforeEach(async function () {
+        const filesWithNames = files.map((file, i) => ({
+          ...file,
+          name: i === 0 ? 'llama' : 'hippo',
+        }))
+        GcsBucket.getFiles.resolves([filesWithNames])
+        keys = await GcsPersistor.listDirectoryKeys(bucket, key)
+      })
+
+      it('should list the objects in the directory', function () {
+        expect(Storage.prototype.bucket).to.have.been.calledWith(bucket)
+        expect(GcsBucket.getFiles).to.have.been.calledWith({
+          prefix: `${key}/`,
+        })
+      })
+
+      it('should return the keys', function () {
+        expect(keys).to.deep.equal(['llama', 'hippo'])
+      })
+    })
+
+    describe('when there are no files', function () {
+      let keys
+
+      beforeEach(async function () {
+        GcsBucket.getFiles.resolves([[]])
+        keys = await GcsPersistor.listDirectoryKeys(bucket, key)
+      })
+
+      it('should return an empty array', function () {
+        expect(keys).to.deep.equal([])
+      })
+    })
+
+    describe('when there is an error listing the objects', function () {
+      let error
+
+      beforeEach(async function () {
+        GcsBucket.getFiles.rejects(genericError)
+        try {
+          await GcsPersistor.listDirectoryKeys(bucket, key)
+        } catch (err) {
+          error = err
+        }
+      })
+
+      it('should generate a ReadError', function () {
+        expect(error).to.be.an.instanceOf(Errors.ReadError)
+      })
+
+      it('should wrap the error', function () {
+        expect(error.cause).to.equal(genericError)
+      })
+    })
+  })
+
+  describe('listDirectoryStats', function () {
+    describe('with valid parameters', function () {
+      let stats
+
+      beforeEach(async function () {
+        const filesWithNames = files.map((file, i) => ({
+          ...file,
+          name: i === 0 ? 'llama' : 'hippo',
+        }))
+        GcsBucket.getFiles.resolves([filesWithNames])
+        stats = await GcsPersistor.listDirectoryStats(bucket, key)
+      })
+
+      it('should list the objects in the directory', function () {
+        expect(Storage.prototype.bucket).to.have.been.calledWith(bucket)
+        expect(GcsBucket.getFiles).to.have.been.calledWith({
+          prefix: `${key}/`,
+        })
+      })
+
+      it('should return the stats', function () {
+        expect(stats).to.deep.equal([
+          { key: 'llama', size: 11 },
+          { key: 'hippo', size: 22 },
+        ])
+      })
+    })
+
+    describe('when there are no files', function () {
+      let stats
+
+      beforeEach(async function () {
+        GcsBucket.getFiles.resolves([[]])
+        stats = await GcsPersistor.listDirectoryStats(bucket, key)
+      })
+
+      it('should return an empty array', function () {
+        expect(stats).to.deep.equal([])
+      })
+    })
+
+    describe('when there is an error listing the objects', function () {
+      let error
+
+      beforeEach(async function () {
+        GcsBucket.getFiles.rejects(genericError)
+        try {
+          await GcsPersistor.listDirectoryStats(bucket, key)
+        } catch (err) {
+          error = err
+        }
+      })
+
+      it('should generate a ReadError', function () {
+        expect(error).to.be.an.instanceOf(Errors.ReadError)
+      })
+
+      it('should wrap the error', function () {
+        expect(error.cause).to.equal(genericError)
+      })
+    })
+  })
 })

libraries/object-persistor/test/unit/ProjectKeyTests.js

@@ -1,8 +1,6 @@
-'use strict'
-
 const { expect } = require('chai')
 
-const { format, pad } = require('../../../../storage/lib/project_key')
+const { format, pad } = require('../../src/ProjectKey.js')
 
 describe('projectKey', function () {
   it('reverses padded keys', function () {

libraries/object-persistor/test/unit/S3PersistorTests.js

@@ -1046,4 +1046,215 @@ describe('S3PersistorTests', function () {
       })
     })
   })
+
+  describe('listDirectoryKeys', function () {
+    describe('with valid parameters', function () {
+      let keys
+
+      beforeEach(async function () {
+        S3.mockSend(S3.ListObjectsV2Command, {
+          Contents: files.map(file => ({ Key: file.Key })),
+          IsTruncated: false,
+        })
+        keys = await S3Persistor.listDirectoryKeys(bucket, key)
+      })
+
+      it('should list the objects in the directory', function () {
+        S3.assertSendCalledWith(S3.ListObjectsV2Command, {
+          Bucket: bucket,
+          Prefix: key,
+        })
+      })
+
+      it('should return the keys', function () {
+        expect(keys).to.deep.equal(['llama', 'hippo'])
+      })
+    })
+
+    describe('when there are no files', function () {
+      let keys
+
+      beforeEach(async function () {
+        S3.mockSend(S3.ListObjectsV2Command, {
+          Contents: [],
+          IsTruncated: false,
+        })
+        keys = await S3Persistor.listDirectoryKeys(bucket, key)
+      })
+
+      it('should return an empty array', function () {
+        expect(keys).to.deep.equal([])
+      })
+    })
+
+    describe('when there are more files available', function () {
+      const continuationToken = 'wombat'
+      let keys
+
+      beforeEach(async function () {
+        S3.mockSend(
+          S3.ListObjectsV2Command,
+          {
+            Contents: [files[0]].map(file => ({ Key: file.Key })),
+            IsTruncated: true,
+            NextContinuationToken: continuationToken,
+          },
+          {
+            nextResponses: [
+              {
+                Contents: [files[1]].map(file => ({ Key: file.Key })),
+                IsTruncated: false,
+              },
+            ],
+          }
+        )
+        keys = await S3Persistor.listDirectoryKeys(bucket, key)
+      })
+
+      it('should list the objects in multiple calls', function () {
+        S3.assertSendCallCount(S3.ListObjectsV2Command, 2)
+        expect(S3.s3ClientStub.send.firstCall.args[0].payload).to.deep.equal({
+          Bucket: bucket,
+          Prefix: key,
+        })
+        expect(S3.s3ClientStub.send.secondCall.args[0].payload).to.deep.equal({
+          Bucket: bucket,
+          Prefix: key,
+          ContinuationToken: continuationToken,
+        })
+      })
+
+      it('should return all keys', function () {
+        expect(keys).to.deep.equal(['llama', 'hippo'])
+      })
+    })
+
+    describe('when there is an error listing the objects', function () {
+      let error
+
+      beforeEach(async function () {
+        S3.mockSend(S3.ListObjectsV2Command, genericError, { rejects: true })
+        try {
+          await S3Persistor.listDirectoryKeys(bucket, key)
+        } catch (err) {
+          error = err
+        }
+      })
+
+      it('should generate a ReadError', function () {
+        expect(error).to.be.an.instanceOf(Errors.ReadError)
+      })
+
+      it('should wrap the error', function () {
+        expect(error.cause).to.exist
+      })
+    })
+  })
+
+  describe('listDirectoryStats', function () {
+    describe('with valid parameters', function () {
+      let stats
+
+      beforeEach(async function () {
+        S3.mockSend(S3.ListObjectsV2Command, {
+          Contents: files.map(file => ({ Key: file.Key, Size: file.Size })),
+          IsTruncated: false,
+        })
+        stats = await S3Persistor.listDirectoryStats(bucket, key)
+      })
+
+      it('should list the objects in the directory', function () {
+        S3.assertSendCalledWith(S3.ListObjectsV2Command, {
+          Bucket: bucket,
+          Prefix: key,
+        })
+      })
+
+      it('should return the stats', function () {
+        expect(stats).to.deep.equal([
+          { key: 'llama', size: 11 },
+          { key: 'hippo', size: 22 },
+        ])
+      })
+    })
+
+    describe('when there are no files', function () {
+      let stats
+
+      beforeEach(async function () {
+        S3.mockSend(S3.ListObjectsV2Command, {
+          Contents: [],
+          IsTruncated: false,
+        })
+        stats = await S3Persistor.listDirectoryStats(bucket, key)
+      })
+
+      it('should return an empty array', function () {
+        expect(stats).to.deep.equal([])
+      })
+    })
+
+    describe('when there are more files available', function () {
+      const continuationToken = 'wombat'
+      let stats
+
+      beforeEach(async function () {
+        S3.mockSend(
+          S3.ListObjectsV2Command,
+          {
+            Contents: [files[0]].map(file => ({
+              Key: file.Key,
+              Size: file.Size,
+            })),
+            IsTruncated: true,
+            NextContinuationToken: continuationToken,
+          },
+          {
+            nextResponses: [
+              {
+                Contents: [files[1]].map(file => ({
+                  Key: file.Key,
+                  Size: file.Size,
+                })),
+                IsTruncated: false,
+              },
+            ],
+          }
+        )
+        stats = await S3Persistor.listDirectoryStats(bucket, key)
+      })
+
+      it('should list the objects in multiple calls', function () {
+        S3.assertSendCallCount(S3.ListObjectsV2Command, 2)
+      })
+
+      it('should return all stats', function () {
+        expect(stats).to.deep.equal([
+          { key: 'llama', size: 11 },
+          { key: 'hippo', size: 22 },
+        ])
+      })
+    })
+
+    describe('when there is an error listing the objects', function () {
+      let error
+
+      beforeEach(async function () {
+        S3.mockSend(S3.ListObjectsV2Command, genericError, { rejects: true })
+        try {
+          await S3Persistor.listDirectoryStats(bucket, key)
+        } catch (err) {
+          error = err
+        }
+      })
+
+      it('should generate a ReadError', function () {
+        expect(error).to.be.an.instanceOf(Errors.ReadError)
+      })
+
+      it('should wrap the error', function () {
+        expect(error.cause).to.exist
+      })
+    })
+  })
 })

libraries/overleaf-editor-core/.jenkinsIncludeFile

@@ -1,8 +0,0 @@
-# Autogenerated by build scripts. Do not edit.
-.eslint*
-.prettier*
-libraries/o-error/**
-libraries/overleaf-editor-core/**
-package-lock.json
-package.json
-patches/**

libraries/overleaf-editor-core/Jenkinsfile

@@ -1,82 +0,0 @@
-// Autogenerated by build scripts. Do not edit.
-
-pipeline {
-  agent {
-    node {
-      label 'jenkins-agent-web'
-      customWorkspace '/workspace'
-    }
-  }
-  options {
-    timestamps()
-    timeout(time: 15, unit: 'MINUTES')
-  }
-  environment {
-    BRANCH_NAME = "${env.CHANGE_BRANCH ? env.CHANGE_BRANCH : env.BRANCH_NAME}"
-    JENKINS_BUILD_NUMBER = "${BUILD_NUMBER}"
-    BUILD_NUMBER = "${SHORT_SHA}_${BUILD_NUMBER}"
-    COMMIT_SHA = "${GIT_COMMIT}"
-    SHORT_SHA = "${GIT_COMMIT.take(7)}"
-  }
-  stages {
-    stage('Stage 1') {
-      parallel {
-        stage('Install monorepo') {
-          steps {
-            retry(count: 3) {
-              sh 'make monorepo_setup'
-            }
-          }
-        }
-        stage('Create reports folder') {
-          steps {
-            sh 'mkdir libraries/overleaf-editor-core/reports'
-          }
-        }
-      }
-    }
-    stage('Stage 2') {
-      parallel {
-        stage('Lint') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/overleaf-editor-core monorepo npm run lint -- --format json --output-file reports/eslint.json'
-          }
-          post {
-            always {
-              sh """
-                sed -i 's_"filePath":"/overleaf_"filePath":"/workspace_g' libraries/overleaf-editor-core/reports/eslint.json
-              """
-              recordIssues checksAnnotationScope: 'ALL', enabledForFailure: true, failOnError: true, id: 'overleaf-editor-core-eslint', name: 'overleaf-editor-core eslint', qualityGates: [[integerThreshold: 1, threshold: 1.0, type: 'TOTAL']], sourceCodeRetention: 'LAST_BUILD', tools: [esLint(pattern: 'libraries/overleaf-editor-core/reports/eslint.json')]
-            }
-          }
-        }
-        stage('Format') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/overleaf-editor-core monorepo npm run format'
-          }
-        }
-        stage('Typecheck') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/overleaf-editor-core monorepo npm run types:check'
-          }
-        }
-        stage('Test') {
-          steps {
-            retry(count: 3) {
-              sh 'bin/run -w /overleaf/libraries/overleaf-editor-core monorepo npm run test:ci'
-            }
-          }
-        }
-      }
-    }
-  }
-  post {
-    always {
-      junit checksName: 'overleaf-editor-core test results', testResults: 'libraries/overleaf-editor-core/reports/junit-*.xml'
-    }
-    cleanup {
-      sh 'rm -rf libraries/overleaf-editor-core/reports'
-      sh 'make clean_jenkins -j10'
-    }
-  }
-}

libraries/overleaf-editor-core/buildscript.txt

@@ -5,4 +5,5 @@ overleaf-editor-core
 --esmock-loader=False
 --is-library=True
 --node-version=22.18.0
+--pipeline-owner=🥑 Core
 --public-repo=False

libraries/overleaf-editor-core/test/mocha-multi-reporters.cjs

@@ -4,5 +4,6 @@ module.exports = {
     mochaFile: `reports/junit-mocha-${process.env.MOCHA_GREP}.xml`,
     includePending: true,
     jenkinsMode: true,
+    output: true,
   },
 }

libraries/promise-utils/.jenkinsIncludeFile

@@ -1,7 +0,0 @@
-# Autogenerated by build scripts. Do not edit.
-.eslint*
-.prettier*
-libraries/promise-utils/**
-package-lock.json
-package.json
-patches/**

libraries/promise-utils/Jenkinsfile

@@ -1,82 +0,0 @@
-// Autogenerated by build scripts. Do not edit.
-
-pipeline {
-  agent {
-    node {
-      label 'jenkins-agent-web'
-      customWorkspace '/workspace'
-    }
-  }
-  options {
-    timestamps()
-    timeout(time: 15, unit: 'MINUTES')
-  }
-  environment {
-    BRANCH_NAME = "${env.CHANGE_BRANCH ? env.CHANGE_BRANCH : env.BRANCH_NAME}"
-    JENKINS_BUILD_NUMBER = "${BUILD_NUMBER}"
-    BUILD_NUMBER = "${SHORT_SHA}_${BUILD_NUMBER}"
-    COMMIT_SHA = "${GIT_COMMIT}"
-    SHORT_SHA = "${GIT_COMMIT.take(7)}"
-  }
-  stages {
-    stage('Stage 1') {
-      parallel {
-        stage('Install monorepo') {
-          steps {
-            retry(count: 3) {
-              sh 'make monorepo_setup'
-            }
-          }
-        }
-        stage('Create reports folder') {
-          steps {
-            sh 'mkdir libraries/promise-utils/reports'
-          }
-        }
-      }
-    }
-    stage('Stage 2') {
-      parallel {
-        stage('Lint') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/promise-utils monorepo npm run lint -- --format json --output-file reports/eslint.json'
-          }
-          post {
-            always {
-              sh """
-                sed -i 's_"filePath":"/overleaf_"filePath":"/workspace_g' libraries/promise-utils/reports/eslint.json
-              """
-              recordIssues checksAnnotationScope: 'ALL', enabledForFailure: true, failOnError: true, id: 'promise-utils-eslint', name: 'promise-utils eslint', qualityGates: [[integerThreshold: 1, threshold: 1.0, type: 'TOTAL']], sourceCodeRetention: 'LAST_BUILD', tools: [esLint(pattern: 'libraries/promise-utils/reports/eslint.json')]
-            }
-          }
-        }
-        stage('Format') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/promise-utils monorepo npm run format'
-          }
-        }
-        stage('Typecheck') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/promise-utils monorepo npm run types:check'
-          }
-        }
-        stage('Test') {
-          steps {
-            retry(count: 3) {
-              sh 'bin/run -w /overleaf/libraries/promise-utils monorepo npm run test:ci'
-            }
-          }
-        }
-      }
-    }
-  }
-  post {
-    always {
-      junit checksName: 'promise-utils test results', testResults: 'libraries/promise-utils/reports/junit-*.xml'
-    }
-    cleanup {
-      sh 'rm -rf libraries/promise-utils/reports'
-      sh 'make clean_jenkins -j10'
-    }
-  }
-}

libraries/promise-utils/buildscript.txt

@@ -5,4 +5,5 @@ promise-utils
 --esmock-loader=False
 --is-library=True
 --node-version=22.18.0
+--pipeline-owner=🚉 Platform
 --public-repo=False

libraries/promise-utils/test/mocha-multi-reporters.cjs

@@ -4,5 +4,6 @@ module.exports = {
     mochaFile: `reports/junit-mocha-${process.env.MOCHA_GREP}.xml`,
     includePending: true,
     jenkinsMode: true,
+    output: true,
   },
 }

libraries/ranges-tracker/.jenkinsIncludeFile

@@ -1,7 +0,0 @@
-# Autogenerated by build scripts. Do not edit.
-.eslint*
-.prettier*
-libraries/ranges-tracker/**
-package-lock.json
-package.json
-patches/**

libraries/ranges-tracker/Jenkinsfile

@@ -1,82 +0,0 @@
-// Autogenerated by build scripts. Do not edit.
-
-pipeline {
-  agent {
-    node {
-      label 'jenkins-agent-web'
-      customWorkspace '/workspace'
-    }
-  }
-  options {
-    timestamps()
-    timeout(time: 15, unit: 'MINUTES')
-  }
-  environment {
-    BRANCH_NAME = "${env.CHANGE_BRANCH ? env.CHANGE_BRANCH : env.BRANCH_NAME}"
-    JENKINS_BUILD_NUMBER = "${BUILD_NUMBER}"
-    BUILD_NUMBER = "${SHORT_SHA}_${BUILD_NUMBER}"
-    COMMIT_SHA = "${GIT_COMMIT}"
-    SHORT_SHA = "${GIT_COMMIT.take(7)}"
-  }
-  stages {
-    stage('Stage 1') {
-      parallel {
-        stage('Install monorepo') {
-          steps {
-            retry(count: 3) {
-              sh 'make monorepo_setup'
-            }
-          }
-        }
-        stage('Create reports folder') {
-          steps {
-            sh 'mkdir libraries/ranges-tracker/reports'
-          }
-        }
-      }
-    }
-    stage('Stage 2') {
-      parallel {
-        stage('Lint') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/ranges-tracker monorepo npm run lint -- --format json --output-file reports/eslint.json'
-          }
-          post {
-            always {
-              sh """
-                sed -i 's_"filePath":"/overleaf_"filePath":"/workspace_g' libraries/ranges-tracker/reports/eslint.json
-              """
-              recordIssues checksAnnotationScope: 'ALL', enabledForFailure: true, failOnError: true, id: 'ranges-tracker-eslint', name: 'ranges-tracker eslint', qualityGates: [[integerThreshold: 1, threshold: 1.0, type: 'TOTAL']], sourceCodeRetention: 'LAST_BUILD', tools: [esLint(pattern: 'libraries/ranges-tracker/reports/eslint.json')]
-            }
-          }
-        }
-        stage('Format') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/ranges-tracker monorepo npm run format'
-          }
-        }
-        stage('Typecheck') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/ranges-tracker monorepo npm run types:check'
-          }
-        }
-        stage('Test') {
-          steps {
-            retry(count: 3) {
-              sh 'bin/run -w /overleaf/libraries/ranges-tracker monorepo npm run test:ci'
-            }
-          }
-        }
-      }
-    }
-  }
-  post {
-    always {
-      junit checksName: 'ranges-tracker test results', testResults: 'libraries/ranges-tracker/reports/junit-*.xml'
-    }
-    cleanup {
-      sh 'rm -rf libraries/ranges-tracker/reports'
-      sh 'make clean_jenkins -j10'
-    }
-  }
-}

libraries/ranges-tracker/buildscript.txt

@@ -5,4 +5,5 @@ ranges-tracker
 --esmock-loader=False
 --is-library=True
 --node-version=22.18.0
+--pipeline-owner=🥑 Core
 --public-repo=False

libraries/ranges-tracker/test/mocha-multi-reporters.cjs

@@ -4,5 +4,6 @@ module.exports = {
     mochaFile: `reports/junit-mocha-${process.env.MOCHA_GREP}.xml`,
     includePending: true,
     jenkinsMode: true,
+    output: true,
   },
 }

libraries/redis-wrapper/.jenkinsIncludeFile

@@ -1,10 +0,0 @@
-# Autogenerated by build scripts. Do not edit.
-.eslint*
-.prettier*
-libraries/fetch-utils/**
-libraries/logger/**
-libraries/o-error/**
-libraries/redis-wrapper/**
-package-lock.json
-package.json
-patches/**

libraries/redis-wrapper/Jenkinsfile

@@ -1,82 +0,0 @@
-// Autogenerated by build scripts. Do not edit.
-
-pipeline {
-  agent {
-    node {
-      label 'jenkins-agent-web'
-      customWorkspace '/workspace'
-    }
-  }
-  options {
-    timestamps()
-    timeout(time: 15, unit: 'MINUTES')
-  }
-  environment {
-    BRANCH_NAME = "${env.CHANGE_BRANCH ? env.CHANGE_BRANCH : env.BRANCH_NAME}"
-    JENKINS_BUILD_NUMBER = "${BUILD_NUMBER}"
-    BUILD_NUMBER = "${SHORT_SHA}_${BUILD_NUMBER}"
-    COMMIT_SHA = "${GIT_COMMIT}"
-    SHORT_SHA = "${GIT_COMMIT.take(7)}"
-  }
-  stages {
-    stage('Stage 1') {
-      parallel {
-        stage('Install monorepo') {
-          steps {
-            retry(count: 3) {
-              sh 'make monorepo_setup'
-            }
-          }
-        }
-        stage('Create reports folder') {
-          steps {
-            sh 'mkdir libraries/redis-wrapper/reports'
-          }
-        }
-      }
-    }
-    stage('Stage 2') {
-      parallel {
-        stage('Lint') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/redis-wrapper monorepo npm run lint -- --format json --output-file reports/eslint.json'
-          }
-          post {
-            always {
-              sh """
-                sed -i 's_"filePath":"/overleaf_"filePath":"/workspace_g' libraries/redis-wrapper/reports/eslint.json
-              """
-              recordIssues checksAnnotationScope: 'ALL', enabledForFailure: true, failOnError: true, id: 'redis-wrapper-eslint', name: 'redis-wrapper eslint', qualityGates: [[integerThreshold: 1, threshold: 1.0, type: 'TOTAL']], sourceCodeRetention: 'LAST_BUILD', tools: [esLint(pattern: 'libraries/redis-wrapper/reports/eslint.json')]
-            }
-          }
-        }
-        stage('Format') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/redis-wrapper monorepo npm run format'
-          }
-        }
-        stage('Typecheck') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/redis-wrapper monorepo npm run types:check'
-          }
-        }
-        stage('Test') {
-          steps {
-            retry(count: 3) {
-              sh 'bin/run -w /overleaf/libraries/redis-wrapper monorepo npm run test:ci'
-            }
-          }
-        }
-      }
-    }
-  }
-  post {
-    always {
-      junit checksName: 'redis-wrapper test results', testResults: 'libraries/redis-wrapper/reports/junit-*.xml'
-    }
-    cleanup {
-      sh 'rm -rf libraries/redis-wrapper/reports'
-      sh 'make clean_jenkins -j10'
-    }
-  }
-}

libraries/redis-wrapper/buildscript.txt

@@ -5,4 +5,5 @@ redis-wrapper
 --esmock-loader=False
 --is-library=True
 --node-version=22.18.0
+--pipeline-owner=🚉 Platform
 --public-repo=False

libraries/redis-wrapper/test/mocha-multi-reporters.cjs

@@ -4,5 +4,6 @@ module.exports = {
     mochaFile: `reports/junit-mocha-${process.env.MOCHA_GREP}.xml`,
     includePending: true,
     jenkinsMode: true,
+    output: true,
   },
 }

libraries/settings/.jenkinsIncludeFile

@@ -1,7 +0,0 @@
-# Autogenerated by build scripts. Do not edit.
-.eslint*
-.prettier*
-libraries/settings/**
-package-lock.json
-package.json
-patches/**

libraries/settings/Jenkinsfile

@@ -1,72 +0,0 @@
-// Autogenerated by build scripts. Do not edit.
-
-pipeline {
-  agent {
-    node {
-      label 'jenkins-agent-web'
-      customWorkspace '/workspace'
-    }
-  }
-  options {
-    timestamps()
-    timeout(time: 15, unit: 'MINUTES')
-  }
-  environment {
-    BRANCH_NAME = "${env.CHANGE_BRANCH ? env.CHANGE_BRANCH : env.BRANCH_NAME}"
-    JENKINS_BUILD_NUMBER = "${BUILD_NUMBER}"
-    BUILD_NUMBER = "${SHORT_SHA}_${BUILD_NUMBER}"
-    COMMIT_SHA = "${GIT_COMMIT}"
-    SHORT_SHA = "${GIT_COMMIT.take(7)}"
-  }
-  stages {
-    stage('Stage 1') {
-      parallel {
-        stage('Install monorepo') {
-          steps {
-            retry(count: 3) {
-              sh 'make monorepo_setup'
-            }
-          }
-        }
-        stage('Create reports folder') {
-          steps {
-            sh 'mkdir libraries/settings/reports'
-          }
-        }
-      }
-    }
-    stage('Stage 2') {
-      parallel {
-        stage('Lint') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/settings monorepo npm run lint -- --format json --output-file reports/eslint.json'
-          }
-          post {
-            always {
-              sh """
-                sed -i 's_"filePath":"/overleaf_"filePath":"/workspace_g' libraries/settings/reports/eslint.json
-              """
-              recordIssues checksAnnotationScope: 'ALL', enabledForFailure: true, failOnError: true, id: 'settings-eslint', name: 'settings eslint', qualityGates: [[integerThreshold: 1, threshold: 1.0, type: 'TOTAL']], sourceCodeRetention: 'LAST_BUILD', tools: [esLint(pattern: 'libraries/settings/reports/eslint.json')]
-            }
-          }
-        }
-        stage('Format') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/settings monorepo npm run format'
-          }
-        }
-        stage('Typecheck') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/settings monorepo npm run types:check'
-          }
-        }
-      }
-    }
-  }
-  post {
-    cleanup {
-      sh 'rm -rf libraries/settings/reports'
-      sh 'make clean_jenkins -j10'
-    }
-  }
-}

libraries/settings/buildscript.txt

@@ -5,4 +5,5 @@ settings
 --esmock-loader=False
 --is-library=True
 --node-version=22.18.0
+--pipeline-owner=🚉 Platform
 --public-repo=False

libraries/stream-utils/.jenkinsIncludeFile

@@ -1,7 +0,0 @@
-# Autogenerated by build scripts. Do not edit.
-.eslint*
-.prettier*
-libraries/stream-utils/**
-package-lock.json
-package.json
-patches/**

libraries/stream-utils/Jenkinsfile

@@ -1,82 +0,0 @@
-// Autogenerated by build scripts. Do not edit.
-
-pipeline {
-  agent {
-    node {
-      label 'jenkins-agent-web'
-      customWorkspace '/workspace'
-    }
-  }
-  options {
-    timestamps()
-    timeout(time: 15, unit: 'MINUTES')
-  }
-  environment {
-    BRANCH_NAME = "${env.CHANGE_BRANCH ? env.CHANGE_BRANCH : env.BRANCH_NAME}"
-    JENKINS_BUILD_NUMBER = "${BUILD_NUMBER}"
-    BUILD_NUMBER = "${SHORT_SHA}_${BUILD_NUMBER}"
-    COMMIT_SHA = "${GIT_COMMIT}"
-    SHORT_SHA = "${GIT_COMMIT.take(7)}"
-  }
-  stages {
-    stage('Stage 1') {
-      parallel {
-        stage('Install monorepo') {
-          steps {
-            retry(count: 3) {
-              sh 'make monorepo_setup'
-            }
-          }
-        }
-        stage('Create reports folder') {
-          steps {
-            sh 'mkdir libraries/stream-utils/reports'
-          }
-        }
-      }
-    }
-    stage('Stage 2') {
-      parallel {
-        stage('Lint') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/stream-utils monorepo npm run lint -- --format json --output-file reports/eslint.json'
-          }
-          post {
-            always {
-              sh """
-                sed -i 's_"filePath":"/overleaf_"filePath":"/workspace_g' libraries/stream-utils/reports/eslint.json
-              """
-              recordIssues checksAnnotationScope: 'ALL', enabledForFailure: true, failOnError: true, id: 'stream-utils-eslint', name: 'stream-utils eslint', qualityGates: [[integerThreshold: 1, threshold: 1.0, type: 'TOTAL']], sourceCodeRetention: 'LAST_BUILD', tools: [esLint(pattern: 'libraries/stream-utils/reports/eslint.json')]
-            }
-          }
-        }
-        stage('Format') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/stream-utils monorepo npm run format'
-          }
-        }
-        stage('Typecheck') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/stream-utils monorepo npm run types:check'
-          }
-        }
-        stage('Test') {
-          steps {
-            retry(count: 3) {
-              sh 'bin/run -w /overleaf/libraries/stream-utils monorepo npm run test:ci'
-            }
-          }
-        }
-      }
-    }
-  }
-  post {
-    always {
-      junit checksName: 'stream-utils test results', testResults: 'libraries/stream-utils/reports/junit-*.xml'
-    }
-    cleanup {
-      sh 'rm -rf libraries/stream-utils/reports'
-      sh 'make clean_jenkins -j10'
-    }
-  }
-}

libraries/stream-utils/buildscript.txt

@@ -5,4 +5,5 @@ stream-utils
 --esmock-loader=False
 --is-library=True
 --node-version=22.18.0
+--pipeline-owner=🚉 Platform
 --public-repo=False

libraries/stream-utils/test/mocha-multi-reporters.cjs

@@ -4,5 +4,6 @@ module.exports = {
     mochaFile: `reports/junit-mocha-${process.env.MOCHA_GREP}.xml`,
     includePending: true,
     jenkinsMode: true,
+    output: true,
   },
 }

libraries/validation-tools/.jenkinsIncludeFile

@@ -1,8 +0,0 @@
-# Autogenerated by build scripts. Do not edit.
-.eslint*
-.prettier*
-libraries/o-error/**
-libraries/validation-tools/**
-package-lock.json
-package.json
-patches/**

libraries/validation-tools/Jenkinsfile

@@ -1,82 +0,0 @@
-// Autogenerated by build scripts. Do not edit.
-
-pipeline {
-  agent {
-    node {
-      label 'jenkins-agent-web'
-      customWorkspace '/workspace'
-    }
-  }
-  options {
-    timestamps()
-    timeout(time: 15, unit: 'MINUTES')
-  }
-  environment {
-    BRANCH_NAME = "${env.CHANGE_BRANCH ? env.CHANGE_BRANCH : env.BRANCH_NAME}"
-    JENKINS_BUILD_NUMBER = "${BUILD_NUMBER}"
-    BUILD_NUMBER = "${SHORT_SHA}_${BUILD_NUMBER}"
-    COMMIT_SHA = "${GIT_COMMIT}"
-    SHORT_SHA = "${GIT_COMMIT.take(7)}"
-  }
-  stages {
-    stage('Stage 1') {
-      parallel {
-        stage('Install monorepo') {
-          steps {
-            retry(count: 3) {
-              sh 'make monorepo_setup'
-            }
-          }
-        }
-        stage('Create reports folder') {
-          steps {
-            sh 'mkdir libraries/validation-tools/reports'
-          }
-        }
-      }
-    }
-    stage('Stage 2') {
-      parallel {
-        stage('Lint') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/validation-tools monorepo npm run lint -- --format json --output-file reports/eslint.json'
-          }
-          post {
-            always {
-              sh """
-                sed -i 's_"filePath":"/overleaf_"filePath":"/workspace_g' libraries/validation-tools/reports/eslint.json
-              """
-              recordIssues checksAnnotationScope: 'ALL', enabledForFailure: true, failOnError: true, id: 'validation-tools-eslint', name: 'validation-tools eslint', qualityGates: [[integerThreshold: 1, threshold: 1.0, type: 'TOTAL']], sourceCodeRetention: 'LAST_BUILD', tools: [esLint(pattern: 'libraries/validation-tools/reports/eslint.json')]
-            }
-          }
-        }
-        stage('Format') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/validation-tools monorepo npm run format'
-          }
-        }
-        stage('Typecheck') {
-          steps {
-            sh 'bin/run -w /overleaf/libraries/validation-tools monorepo npm run types:check'
-          }
-        }
-        stage('Test') {
-          steps {
-            retry(count: 3) {
-              sh 'bin/run -w /overleaf/libraries/validation-tools monorepo npm run test:ci'
-            }
-          }
-        }
-      }
-    }
-  }
-  post {
-    always {
-      junit checksName: 'validation-tools test results', testResults: 'libraries/validation-tools/reports/junit-*.xml'
-    }
-    cleanup {
-      sh 'rm -rf libraries/validation-tools/reports'
-      sh 'make clean_jenkins -j10'
-    }
-  }
-}

libraries/validation-tools/handleValidationError.js

@@ -0,0 +1,17 @@
+const { isZodErrorLike, fromError } = require('zod-validation-error')
+
+function createHandleValidationError(statusCode = 400) {
+  return [
+    (err, req, res, next) => {
+      if (!isZodErrorLike(err)) {
+        return next(err)
+      }
+
+      res.status(statusCode).json({ ...fromError(err), statusCode })
+    },
+  ]
+}
+
+const handleValidationError = createHandleValidationError(400)
+
+module.exports = { handleValidationError, createHandleValidationError }

libraries/validation-tools/index.js

@@ -1,11 +1,19 @@
 const { ParamsError } = require('./Errors')
 const { z } = require('zod')
 const { zz } = require('./zodHelpers')
-const { validateReq } = require('./validateReq')
+const { parseReq } = require('./parseReq')
+const { validateSchema } = require('./validateSchema')
+const {
+  handleValidationError,
+  createHandleValidationError,
+} = require('./handleValidationError')
 
 module.exports = {
   z,
   zz,
-  validateReq,
+  validateSchema,
+  parseReq,
+  handleValidationError,
+  createHandleValidationError,
   ParamsError,
 }

libraries/validation-tools/package.json

@@ -22,10 +22,11 @@
   "dependencies": {
     "@overleaf/o-error": "*",
     "mongodb": "^6.12.0",
-    "zod": "^4.1.8"
+    "zod": "^4.1.8",
+    "zod-validation-error": "^4.0.1"
   },
   "devDependencies": {
     "typescript": "^5.0.4",
-    "vitest": "^3.2.4"
+    "vitest": "^4.0.0"
   }
 }

libraries/validation-tools/parseReq.js

@@ -12,14 +12,14 @@ const { ParamsError } = require('./Errors')
  */
 
 /**
- * Validate a request against a zod schema
+ * Parse and validate a request against a Zod schema
  *
  * @template {ZodType} T
  * @param {Request} req - The Express request object
  * @param {T} schema - The Zod schema to validate against
  * @returns {output<T>} The validated request object
  */
-function validateReq(req, schema) {
+function parseReq(req, schema) {
   const parsed = schema.safeParse(req)
   if (parsed.success) {
     return parsed.data
@@ -31,4 +31,4 @@ function validateReq(req, schema) {
   }
 }
 
-module.exports = { validateReq }
+module.exports = { parseReq }

libraries/validation-tools/test/unit/src/validateReq.test.ts

@@ -1,10 +1,10 @@
-import { validateReq } from '../../../validateReq'
+import { parseReq } from '../../../parseReq'
 import { describe, expect, it } from 'vitest'
 import { z } from 'zod'
 import type { Request } from 'express'
 import { zz } from '../../../zodHelpers'
 
-describe('validateReq', () => {
+describe('parseReq', () => {
   describe('with a request that is valid for the schema', () => {
     it('should return the parsed request', () => {
       const req = {
@@ -25,7 +25,7 @@ describe('validateReq', () => {
         }),
       })
 
-      const result = validateReq(req, schema)
+      const result = parseReq(req, schema)
 
       expect(result).toEqual({
         params: {
@@ -46,7 +46,7 @@ describe('validateReq', () => {
       } as Request<{ id: string }>
 
       expect(() =>
-        validateReq(
+        parseReq(
           req,
           z.object({
             params: z.object({
@@ -65,7 +65,7 @@ describe('validateReq', () => {
       } as Request
 
       expect(() =>
-        validateReq(
+        parseReq(
           req,
           z.object({
             body: z.object({

libraries/validation-tools/test/unit/src/zodHelpers.test.ts

@@ -41,4 +41,180 @@ describe('zodHelpers', () => {
       expect(parsed.data?.toString()).toBe('507f1f77bcf86cd799439011')
     })
   })
+  describe('datetime', () => {
+    it('parses valid ISO 8601 datetime strings', () => {
+      const parsed = zz.datetime().safeParse('2024-01-01T12:00:00Z')
+      expect(parsed.success).toBe(true)
+      expect(parsed.data).toEqual(new Date('2024-01-01T12:00:00Z'))
+    })
+
+    it('parses a valid ISO 8601 datetime with offset', () => {
+      const parsed = zz
+        .datetime({ offset: true })
+        .safeParse('2024-01-01T12:00:00+00:00')
+      expect(parsed.success).toBe(true)
+      expect(parsed.data).toEqual(new Date('2024-01-01T12:00:00+00:00'))
+    })
+
+    it('parses a valid Date object', () => {
+      const date = new Date('2024-01-01T12:00:00Z')
+      const parsed = zz.datetime().safeParse(date)
+      expect(parsed.success).toBe(true)
+      expect(parsed.data).toEqual(date)
+    })
+
+    it('fails to parse datetime with offset when offset option is false', () => {
+      const parsed = zz
+        .datetime({ offset: false })
+        .safeParse('2024-01-01T12:00:00+00:00')
+      expect(parsed.success).toBe(false)
+      expect(parsed.error?.issues).toHaveLength(1)
+      expect(parsed.error?.issues).toMatchObject([
+        expect.objectContaining({
+          code: 'invalid_format',
+          format: 'datetime',
+          message: 'Invalid ISO datetime',
+        }),
+      ])
+    })
+
+    it('fails to parse null when schema is not nullable', () => {
+      const parsed = zz.datetime().safeParse(null)
+      expect(parsed.success).toBe(false)
+      expect(parsed.error?.message).toContain(
+        'Invalid input: expected date, received null'
+      )
+    })
+
+    it('fails to parse invalid datetime strings', () => {
+      const parsed = zz.datetime().safeParse('invalid-datetime')
+      expect(parsed.success).toBe(false)
+      expect(parsed.error?.issues).toHaveLength(1)
+      expect(parsed.error?.issues).toMatchObject([
+        expect.objectContaining({
+          code: 'invalid_format',
+          format: 'datetime',
+          message: 'Invalid ISO datetime',
+        }),
+      ])
+    })
+  })
+  describe('datetimeNullable', () => {
+    it('parses valid ISO 8601 datetime strings', () => {
+      const parsed = zz.datetimeNullable().safeParse('2024-01-01T12:00:00Z')
+      expect(parsed.success).toBe(true)
+      expect(parsed.data).toEqual(new Date('2024-01-01T12:00:00Z'))
+    })
+
+    it('parses a valid ISO 8601 datetime with offset', () => {
+      const parsed = zz
+        .datetimeNullable({ offset: true })
+        .safeParse('2024-01-01T12:00:00+00:00')
+      expect(parsed.success).toBe(true)
+      expect(parsed.data).toEqual(new Date('2024-01-01T12:00:00+00:00'))
+    })
+
+    it('parses a valid Date object', () => {
+      const date = new Date('2024-01-01T12:00:00Z')
+      const parsed = zz.datetimeNullable().safeParse(date)
+      expect(parsed.success).toBe(true)
+      expect(parsed.data).toEqual(date)
+    })
+
+    it('fails to parse datetime with offset when offset option is false', () => {
+      const parsed = zz
+        .datetimeNullable({ offset: false })
+        .safeParse('2024-01-01T12:00:00+00:00')
+      expect(parsed.success).toBe(false)
+      expect(parsed.error?.issues).toHaveLength(1)
+      expect(parsed.error?.issues).toMatchObject([
+        expect.objectContaining({
+          code: 'invalid_format',
+          format: 'datetime',
+          message: 'Invalid ISO datetime',
+        }),
+      ])
+    })
+
+    it('parses null when schema is nullable and input is null', () => {
+      const parsed = zz.datetimeNullable().safeParse(null)
+      expect(parsed.success).toBe(true)
+      expect(parsed.data).toBeNull()
+    })
+
+    it('fails to parse invalid datetime strings', () => {
+      const parsed = zz.datetimeNullable().safeParse('invalid-datetime')
+      expect(parsed.success).toBe(false)
+      expect(parsed.error?.issues).toHaveLength(1)
+      expect(parsed.error?.issues).toMatchObject([
+        expect.objectContaining({
+          code: 'invalid_format',
+          format: 'datetime',
+          message: 'Invalid ISO datetime',
+        }),
+      ])
+    })
+  })
+  describe('datetimeNullish', () => {
+    it('parses valid ISO 8601 datetime strings', () => {
+      const parsed = zz.datetimeNullish().safeParse('2024-01-01T12:00:00Z')
+      expect(parsed.success).toBe(true)
+      expect(parsed.data).toEqual(new Date('2024-01-01T12:00:00Z'))
+    })
+
+    it('parses a valid ISO 8601 datetime with offset', () => {
+      const parsed = zz
+        .datetimeNullish({ offset: true })
+        .safeParse('2024-01-01T12:00:00+00:00')
+      expect(parsed.success).toBe(true)
+      expect(parsed.data).toEqual(new Date('2024-01-01T12:00:00+00:00'))
+    })
+
+    it('parses a valid Date object', () => {
+      const date = new Date('2024-01-01T12:00:00Z')
+      const parsed = zz.datetimeNullish().safeParse(date)
+      expect(parsed.success).toBe(true)
+      expect(parsed.data).toEqual(date)
+    })
+
+    it('parses null when schema is nullable and input is null', () => {
+      const parsed = zz.datetimeNullish().safeParse(null)
+      expect(parsed.success).toBe(true)
+      expect(parsed.data).toBeNull()
+    })
+
+    it('parses undefined when schema is nullish and input is undefined', () => {
+      const parsed = zz.datetimeNullish().safeParse(undefined)
+      expect(parsed.success).toBe(true)
+      expect(parsed.data).toBeUndefined()
+    })
+
+    it('fails to parse datetime with offset when offset option is false', () => {
+      const parsed = zz
+        .datetimeNullish({ offset: false })
+        .safeParse('2024-01-01T12:00:00+00:00')
+      expect(parsed.success).toBe(false)
+      expect(parsed.error?.issues).toHaveLength(1)
+      expect(parsed.error?.issues).toMatchObject([
+        expect.objectContaining({
+          code: 'invalid_format',
+          format: 'datetime',
+          message: 'Invalid ISO datetime',
+        }),
+      ])
+    })
+
+    it('fails to parse invalid datetime strings', () => {
+      const parsed = zz.datetimeNullish().safeParse('invalid-datetime')
+      expect(parsed.success).toBe(false)
+      expect(parsed.error?.issues).toHaveLength(1)
+      expect(parsed.error?.issues).toMatchObject([
+        expect.objectContaining({
+          code: 'invalid_format',
+          format: 'datetime',
+          message: 'Invalid ISO datetime',
+        }),
+      ])
+    })
+  })
 })

libraries/validation-tools/types.d.ts

@@ -0,0 +1,6 @@
+import z from 'zod'
+
+export interface DatetimeSchemaOptions extends z.core.$ZodISODateTimeParams {
+  allowNull?: boolean
+  allowUndefined?: boolean
+}

libraries/validation-tools/validateSchema.js

@@ -0,0 +1,66 @@
+// @ts-check
+const { isZodErrorLike } = require('zod-validation-error')
+
+/**
+ * @typedef {import('zod').ZodType} ZodType
+ */
+/**
+ * @template T
+ * @typedef {import('zod').output<T>} output<T>
+ */
+
+/**
+ * A helper function to safely get a nested value from an object
+ * using a path array (e.g., ["query", "resource_type"])
+ * @param {any} data
+ * @param {Array<PropertyKey>} path
+ */
+function getPathValue(data, path) {
+  let current = data
+  for (const key of path) {
+    if (current === null || typeof current !== 'object') {
+      return undefined
+    }
+    current = current[key]
+  }
+  return current
+}
+
+/**
+ * @param {any} issue
+ * @param {any} value
+ */
+const isRequiredError = (issue, value) =>
+  value === undefined &&
+  (issue.code === 'invalid_type' || issue.code === 'invalid_union')
+
+/**
+ * Validates data against a Zod schema and throws a user-friendly error.
+ *
+ * @template {ZodType} T
+ * @param {T} schema - The Zod schema
+ * @param {unknown} data - The data to validate
+ * @returns {output<T>} The validated (and transformed) data
+ */
+function validateSchema(schema, data) {
+  try {
+    return schema.parse(data)
+  } catch (err) {
+    if (isZodErrorLike(err)) {
+      const errorMessages = err.issues.map(issue => {
+        const value = getPathValue(data, issue.path)
+        const fieldName = String(issue.path[issue.path.length - 1])
+        if (isRequiredError(issue, value)) {
+          return `"${fieldName}" is required`
+        }
+        return `"${fieldName}" - ` + issue.message
+      })
+
+      throw new Error(errorMessages.join('; '))
+    }
+
+    throw err
+  }
+}
+
+module.exports = { validateSchema }

libraries/validation-tools/zodHelpers.js

@@ -3,6 +3,23 @@ const mongodb = require('mongodb')
 
 const { ObjectId } = mongodb
 
+/**
+ * @import { DatetimeSchemaOptions } from './types'
+ */
+
+/**
+ * @param {DatetimeSchemaOptions} options
+ */
+const datetimeSchema = ({ allowNull, allowUndefined, ...zodOptions } = {}) => {
+  const union = [z.date(), z.iso.datetime(zodOptions)]
+  if (allowNull) union.push(z.null())
+  if (allowUndefined) union.push(z.undefined())
+  return z.union(union).transform(dt => {
+    if (allowNull && !dt) return dt === null ? null : undefined
+    return dt instanceof Date ? dt : new Date(dt)
+  })
+}
+
 const zz = {
   objectId: () =>
     z.string().refine(ObjectId.isValid, { message: 'invalid Mongo ObjectId' }),
@@ -12,6 +29,10 @@ const zz = {
       .refine(ObjectId.isValid, { message: 'invalid Mongo ObjectId' })
       .transform(val => new ObjectId(val)),
   hex: () => z.string().regex(/^[0-9a-f]*$/),
+  datetime: options => datetimeSchema(options),
+  datetimeNullable: options => datetimeSchema({ ...options, allowNull: true }),
+  datetimeNullish: options =>
+    datetimeSchema({ ...options, allowNull: true, allowUndefined: true }),
 }
 
 module.exports = { zz }

package.json

@@ -8,8 +8,9 @@
     "@types/chai": "^4.3.0",
     "@types/chai-as-promised": "^7.1.8",
     "@types/mocha": "^10.0.6",
-    "@typescript-eslint/eslint-plugin": "^8.30.1",
-    "@typescript-eslint/parser": "^8.30.1",
+    "@typescript-eslint/eslint-plugin": "8.50.0",
+    "@typescript-eslint/parser": "^8.50.0",
+    "@vitest/eslint-plugin": "^1.5.0",
     "eslint": "^8.15.0",
     "eslint-config-prettier": "^8.5.0",
     "eslint-config-standard": "^17.0.0",
@@ -22,22 +23,21 @@
     "eslint-plugin-prettier": "^4.0.0",
     "eslint-plugin-promise": "^6.0.0",
     "eslint-plugin-unicorn": "^56.0.0",
-    "prettier": "3.6.2",
+    "prettier": "3.7.4",
     "prettier-plugin-groovy": "0.2.1",
-    "typescript": "^5.8.3"
+    "typescript": "^5.9.3"
   },
   "engines": {
     "npm": "11.4.2"
   },
   "overrides": {
-    "swagger-tools@0.10.4": {
-      "path-to-regexp": "3.3.0",
-      "body-parser": "1.20.3",
-      "multer": "2.0.2"
-    },
     "request@2.88.2": {
       "tough-cookie": "5.1.2",
-      "form-data": "2.5.5"
+      "form-data": "2.5.5",
+      "qs": "6.14.1"
+    },
+    "streamdown": {
+      "mermaid": "file:./services/web/stubs/mermaid-stub"
     }
   },
   "scripts": {
@@ -76,6 +76,7 @@
     "services/third-party-references",
     "services/tpdsworker",
     "services/web",
+    "tools/dependency-management",
     "tools/saas-e2e",
     "tools/migrations"
   ]

patches/@google-cloud+storage++teeny-request+8.0.3.patch

@@ -1,37 +1,35 @@
 diff --git a/node_modules/@google-cloud/storage/node_modules/teeny-request/build/src/index.js b/node_modules/@google-cloud/storage/node_modules/teeny-request/build/src/index.js
-index a2251ca..e29e796 100644
+index af5d15e..8a9bf77 100644
 --- a/node_modules/@google-cloud/storage/node_modules/teeny-request/build/src/index.js
 +++ b/node_modules/@google-cloud/storage/node_modules/teeny-request/build/src/index.js
-@@ -166,27 +166,27 @@ function teenyRequest(reqOpts, callback) {
-     }
-     if (callback === undefined) {
+@@ -168,31 +168,34 @@ function teenyRequest(reqOpts, callback) {
          // Stream mode
--        const requestStream = streamEvents(new stream_1.PassThrough());
--        // eslint-disable-next-line @typescript-eslint/no-explicit-any
+         const requestStream = streamEvents(new stream_1.PassThrough());
+         // eslint-disable-next-line @typescript-eslint/no-explicit-any
 -        let responseStream;
 -        requestStream.once('reading', () => {
 -            if (responseStream) {
--                responseStream.pipe(requestStream);
+-                (0, stream_1.pipeline)(responseStream, requestStream, () => { });
 -            }
 -            else {
 -                requestStream.once('response', () => {
--                    responseStream.pipe(requestStream);
+-                    (0, stream_1.pipeline)(responseStream, requestStream, () => { });
 -                });
 -            }
 -        });
-+        const requestStream = new stream_1.PassThrough();
-+        // // eslint-disable-next-line @typescript-eslint/no-explicit-any
 +        // let responseStream;
 +        // requestStream.once('reading', () => {
-+        //     if (responseStream) {
-+        //         responseStream.pipe(requestStream);
-+        //     }
-+        //     else {
-+        //         requestStream.once('response', () => {
-+        //             responseStream.pipe(requestStream);
-+        //         });
-+        //     }
++        //    if (responseStream) {
++        //        (0, stream_1.pipeline)(responseStream, requestStream, () => { });
++        //    }
++        //    else {
++        //        requestStream.once('response', () => {
++        //            (0, stream_1.pipeline)(responseStream, requestStream, () => { });
++        //        });
++        //    }
 +        // });
++
++
          options.compress = false;
          teenyRequest.stats.requestStarting();
          (0, node_fetch_1.default)(uri, options).then(res => {
@@ -41,10 +39,17 @@ index a2251ca..e29e796 100644
 -                requestStream.emit('error', err);
 -            });
 +            teenyRequest.stats.requestFinished(); stream_1.pipeline(res.body, requestStream, () => {});
-+            // responseStream = res.body;
-+            // responseStream.on('error', (err) => {
-+            //     requestStream.emit('error', err);
-+            // });
++        //    responseStream = res.body;
++        //    responseStream.on('error', (err) => {
++        //        requestStream.emit('error', err);
++        //    });
              const response = fetchToRequestResponse(options, res);
              requestStream.emit('response', response);
          }, err => {
+             teenyRequest.stats.requestFinished();
+             requestStream.emit('error', err);
+         });
++
+         // fetch doesn't supply the raw HTTP stream, instead it
+         // returns a PassThrough piped from the HTTP response
+         // stream.

patches/@node-saml+node-saml+5.1.0.patch

@@ -1,23 +0,0 @@
-diff --git a/node_modules/@node-saml/node-saml/lib/saml.js b/node_modules/@node-saml/node-saml/lib/saml.js
-index fba15b9..a5778cb 100644
---- a/node_modules/@node-saml/node-saml/lib/saml.js
-+++ b/node_modules/@node-saml/node-saml/lib/saml.js
-@@ -336,7 +336,8 @@ class SAML {
-         const requestOrResponse = request || response;
-         (0, utility_1.assertRequired)(requestOrResponse, "either request or response is required");
-         let buffer;
--        if (this.options.skipRequestCompression) {
-+       // logout requestOrResponse must be compressed anyway
-+        if (this.options.skipRequestCompression && operation !== "logout") {
-             buffer = Buffer.from(requestOrResponse, "utf8");
-         }
-         else {
-@@ -495,7 +496,7 @@ class SAML {
-         try {
-             xml = Buffer.from(container.SAMLResponse, "base64").toString("utf8");
-             doc = await (0, xml_1.parseDomFromString)(xml);
--            const inResponseToNodes = xml_1.xpath.selectAttributes(doc, "/*[local-name()='Response']/@InResponseTo");
-+            const inResponseToNodes = xml_1.xpath.selectAttributes(doc, "/*[local-name()='Response' or local-name()='LogoutResponse']/@InResponseTo");
-             if (inResponseToNodes) {
-                 inResponseTo = inResponseToNodes.length ? inResponseToNodes[0].nodeValue : null;
-                 await this.validateInResponseTo(inResponseTo);

patches/ldapauth-fork+4.3.3.patch

@@ -1,64 +0,0 @@
-diff --git a/node_modules/ldapauth-fork/lib/ldapauth.js b/node_modules/ldapauth-fork/lib/ldapauth.js
-index 85ecf36a8b..a7d07e0f78 100644
---- a/node_modules/ldapauth-fork/lib/ldapauth.js
-+++ b/node_modules/ldapauth-fork/lib/ldapauth.js
-@@ -69,6 +69,7 @@ function LdapAuth(opts) {
-   this.opts.bindProperty || (this.opts.bindProperty = 'dn');
-   this.opts.groupSearchScope || (this.opts.groupSearchScope = 'sub');
-   this.opts.groupDnProperty || (this.opts.groupDnProperty = 'dn');
-+  this.opts.tlsStarted = false;
-
-   EventEmitter.call(this);
-
-@@ -108,21 +109,7 @@ function LdapAuth(opts) {
-   this._userClient.on('error', this._handleError.bind(this));
-
-   var self = this;
--  if (this.opts.starttls) {
--    // When starttls is enabled, this callback supplants the 'connect' callback
--    this._adminClient.starttls(this.opts.tlsOptions, this._adminClient.controls, function(err) {
--      if (err) {
--        self._handleError(err);
--      } else {
--        self._onConnectAdmin();
--      }
--    });
--    this._userClient.starttls(this.opts.tlsOptions, this._userClient.controls, function(err) {
--      if (err) {
--        self._handleError(err);
--      }
--    });
--  } else if (opts.reconnect) {
-+  if (opts.reconnect && !this.opts.starttls) {
-     this.once('_installReconnectListener', function() {
-       self.log && self.log.trace('install reconnect listener');
-       self._adminClient.on('connect', function() {
-@@ -384,6 +371,28 @@ LdapAuth.prototype._findGroups = function(user, callback) {
-  */
- LdapAuth.prototype.authenticate = function(username, password, callback) {
-   var self = this;
-+  if (this.opts.starttls && !this.opts.tlsStarted) {
-+    // When starttls is enabled, this callback supplants the 'connect' callback
-+    this._adminClient.starttls(this.opts.tlsOptions, this._adminClient.controls, function (err) {
-+      if (err) {
-+        self._handleError(err);
-+      } else {
-+        self._onConnectAdmin(function(){self._handleAuthenticate(username, password, callback);});
-+      }
-+    });
-+    this._userClient.starttls(this.opts.tlsOptions, this._userClient.controls, function (err) {
-+      if (err) {
-+        self._handleError(err);
-+      }
-+    });
-+  } else {
-+    self._handleAuthenticate(username, password, callback);
-+  }
-+};
-+
-+LdapAuth.prototype._handleAuthenticate = function (username, password, callback) {
-+  this.opts.tlsStarted = true;
-+  var self = this;
-
-   if (typeof password === 'undefined' || password === null || password === '') {
-     return callback(new Error('no password given'));