[saas-e2e] reduce disk IO and webpack overhead (#36997)

* [monorepo] reduce writes from chrome in cypress

Chrome puts its shared memory in /dev/shm, and docker's default 64MB is far
too small for it - which is why Cypress passes --disable-dev-shm-usage,
spilling those segments to files on the container's writable layer instead.
Under one spec that is a few hundred MB of write-then-delete churn; under the
CI fan-out it was most of the disk bandwidth on the whole box.

So every service built from dockerfiles/cypress gets shm_size 1gb and every
cypress config drops the flag again: the SaaS e2e service, web's CI and local
component-test services plus its standalone cypress one, and server-ce's e2e
and native cypress services. Both e2e-open variants inherit the setting
through `extends`; the native one is a service of its own, and would be left
running Chrome against 64MB once the flag is gone.

* [monorepo] adopt cypress-pnp-reporter

Materializing the reporter packages copied 267 packages / 3779 files
(59MB) into /tmp on every container start, once per spec. Resolving them
through PnP inside Cypress's Electron process instead drops that to
zero: the entrypoint shell went from 55.6MB of write_bytes to 0.

* [web] serve built assets from a prepared buffer between builds

Serving a chunk cost ~2ms whatever its size, and an editor page pulls ~144
of them, so webpack-dev-server burned 2-3 cores through an e2e run purely
answering requests for output that had not changed since the last build.

It is not bytes: cutting the bundle 2.6x with a cheaper devtool moved CPU
by 8%, and an empty 304 cost nearly as much as the full body. It is the
per-request work - reading the asset out of the in-memory filesystem and
re-hashing the whole body for a strong ETag, on every request.

Three things a build does that this has to respect, each found by tracing
the hooks rather than by reading them:

- `done` fires twice per build, for the compilation and for the manifest,
  both with the same hash. A flag would have been cleared by the second
  while the first was still running, so the gate is a counter.
- An incremental `done` reports only the assets it rewrote - 16, not 630 -
  so asset names accumulate. Resetting them on `invalid` left the
  manifest's two behind and nothing was served from here again.
- webpack-dev-middleware holds requests during a rebuild. This sits in
  front of it, so it steps aside for that window and drops the cache when
  a rebuild starts. Whether a response can be kept is then decided as it
  ends rather than as it began: an epoch bumped on `invalid` names the
  build its bytes were read from, and only a response from the current
  build with no rebuild in flight is stored. A rebuild starting
  mid-response can no longer write the previous build's bytes in behind
  the cache being dropped, which eviction by asset name could not catch
  anyway - an incremental build does not name the assets it left alone.

Measured on the editor entrypoint, 16 concurrent clients:
0.26-0.31s -> 0.045-0.050s of webpack CPU per page load, and 0.0455s
immediately after a rebuild. Same chunk graph, byte-identical output.

GitOrigin-RevId: 08b4ab2d665414069bc01b4763dd64faa9b4e5cb

libraries/cypress-pnp-reporter/.mocharc.cjs

@@ -0,0 +1,13 @@
+let reporterOptions = {}
+if (process.env.CI) {
+  reporterOptions = {
+    reporter: require.resolve('mocha-multi-reporters'),
+    'reporter-options': ['configFile=./test/mocha-multi-reporters.cjs'],
+  }
+}
+const all = {
+  require: 'test/setup.js',
+  ...reporterOptions,
+}
+
+module.exports = all

libraries/cypress-pnp-reporter/index.cjs

@@ -0,0 +1,76 @@
+'use strict'
+
+/**
+ * cypress-multi-reporters, loadable from inside Cypress.
+ *
+ * Cypress builds mocha reporters in its Electron process. That process resolves
+ * modules through PackherdModuleLoader and does not inherit NODE_OPTIONS, so
+ * Yarn PnP is not installed there and `require('cypress-multi-reporters')`
+ * fails with "Cannot find module".
+ *
+ * Installing PnP for the rest of that process is not an option either: Cypress
+ * asks packherd for its own bundled modules under ids like
+ * `packages/server/lib/browsers/`, which PnP reads as paths inside this
+ * workspace and then rejects ("it isn't declared in your dependencies"),
+ * breaking the browser launch.
+ *
+ * So install PnP, pull the reporters in through it, and hand resolution back to
+ * Cypress. Requires made by the modules that came in through PnP keep going
+ * through PnP, so anything they load later still resolves.
+ *
+ * Point Cypress at this package rather than at cypress-multi-reporters:
+ *
+ *   reporter: require.resolve('@overleaf/cypress-pnp-reporter')
+ */
+
+const Module = require('node:module')
+const path = require('node:path')
+
+function loadReporters() {
+  const MultiReporters = require('cypress-multi-reporters')
+
+  // MultiReporters resolves the reporters named in reporterEnabled when a run
+  // starts, long after PnP is gone. It reads mocha's own registry before
+  // falling back to require(), so register mocha-junit-reporter there while it
+  // can still be found.
+  require('mocha').reporters[
+    'mocha-junit-reporter'
+  ] = require('mocha-junit-reporter')
+
+  return MultiReporters
+}
+
+if (process.versions.pnp) {
+  module.exports = loadReporters()
+} else {
+  const stockLoad = Module._load
+  const stockResolveFilename = Module._resolveFilename
+  const stockFindPath = Module._findPath
+
+  require(path.join(__dirname, '..', '..', '.pnp.cjs')).setup()
+
+  const pnpResolveFilename = Module._resolveFilename
+  const cachedBefore = new Set(Object.keys(Module._cache))
+  const loadedViaPnp = new Set()
+
+  try {
+    module.exports = loadReporters()
+  } finally {
+    for (const filename of Object.keys(Module._cache)) {
+      if (!cachedBefore.has(filename)) loadedViaPnp.add(filename)
+    }
+
+    // Leave the patched fs in place: the reporters live in zip archives, so
+    // anything they read lazily still has to go through it.
+    Module._load = stockLoad
+    Module._findPath = stockFindPath
+    Module._resolveFilename = function (request, parent, ...args) {
+      if (!parent || !loadedViaPnp.has(parent.filename)) {
+        return stockResolveFilename.call(this, request, parent, ...args)
+      }
+      const filename = pnpResolveFilename.call(this, request, parent, ...args)
+      loadedViaPnp.add(filename)
+      return filename
+    }
+  }
+}

libraries/cypress-pnp-reporter/package.json

@@ -0,0 +1,30 @@
+{
+  "name": "@overleaf/cypress-pnp-reporter",
+  "description": "cypress-multi-reporters, loadable from Cypress's Electron process, which has no Yarn PnP",
+  "main": "index.cjs",
+  "files": [
+    "index.cjs"
+  ],
+  "author": "Overleaf (https://www.overleaf.com)",
+  "private": true,
+  "scripts": {
+    "lint": "eslint --cache --cache-location ../../.cache/eslint/ --ext .cjs,.js,.jsx,.mjs,.ts --max-warnings 0 .",
+    "lint:fix": "eslint --cache --cache-location ../../.cache/eslint/ --fix --ext .cjs,.js,.jsx,.mjs,.ts .",
+    "test": "yarn run lint && yarn run types:check && yarn run test:unit",
+    "test:ci": "yarn run test:unit",
+    "test:unit": "mocha --exit test/**/*.{js,cjs}",
+    "types:check": "tsc --noEmit"
+  },
+  "dependencies": {
+    "cypress-multi-reporters": "^2.0.5"
+  },
+  "devDependencies": {
+    "@types/node": "^24.5.2",
+    "chai": "^4.3.6",
+    "eslint": "^10.4.0",
+    "mocha": "^11.1.0",
+    "mocha-junit-reporter": "^2.2.1",
+    "mocha-multi-reporters": "^1.5.1",
+    "typescript": "^5.0.4"
+  }
+}

libraries/cypress-pnp-reporter/test/mocha-multi-reporters.cjs

@@ -0,0 +1,9 @@
+module.exports = {
+  reporterEnabled: 'spec, mocha-junit-reporter',
+  mochaJunitReporterReporterOptions: {
+    mochaFile: `reports/junit-mocha-${process.env.TEST_NAME_PATTERN}.xml`,
+    includePending: true,
+    jenkinsMode: true,
+    output: true,
+  },
+}

libraries/cypress-pnp-reporter/test/setup.js

@@ -0,0 +1 @@
+// no setup yet

libraries/cypress-pnp-reporter/test/unit/cypress-pnp-reporter-test.cjs

@@ -0,0 +1,52 @@
+'use strict'
+
+const { expect } = require('chai')
+const Mocha = require('mocha')
+const JUnitReporter = require('mocha-junit-reporter')
+
+const MultiReporters = require('../../index.cjs')
+
+function buildReporters(reporterEnabled) {
+  const runner = new Mocha.Runner(new Mocha.Suite(''))
+  const reporter = new MultiReporters(runner, {
+    reporterOptions: { reporterEnabled },
+  })
+  return reporter._reporters
+}
+
+describe('cypress-pnp-reporter', function () {
+  it('exports cypress-multi-reporters', function () {
+    expect(MultiReporters).to.equal(require('cypress-multi-reporters'))
+  })
+
+  it('registers mocha-junit-reporter on mocha', function () {
+    expect(Mocha.reporters['mocha-junit-reporter']).to.equal(JUnitReporter)
+  })
+
+  it('builds the reporters used across the Cypress suites', function () {
+    const reporters = buildReporters('spec, mocha-junit-reporter')
+
+    expect(reporters).to.have.length(2)
+    expect(reporters[0]).to.be.an.instanceOf(Mocha.reporters.spec)
+    expect(reporters[1]).to.be.an.instanceOf(JUnitReporter)
+  })
+
+  it('prefers the reporter registered on mocha over resolving it by name', function () {
+    // Cypress builds the reporters after the shim has handed module resolution
+    // back, so mocha's registry is the only place mocha-junit-reporter can
+    // still be found. Everything above this still passes if
+    // cypress-multi-reporters resolves it by name instead, so pin the lookup
+    // order with a reporter that is only reachable through the registry.
+    class Sentinel {}
+    const registered = Mocha.reporters['mocha-junit-reporter']
+    Mocha.reporters['mocha-junit-reporter'] = Sentinel
+
+    try {
+      expect(buildReporters('mocha-junit-reporter')[0]).to.be.an.instanceOf(
+        Sentinel
+      )
+    } finally {
+      Mocha.reporters['mocha-junit-reporter'] = registered
+    }
+  })
+})

libraries/cypress-pnp-reporter/tsconfig.json

@@ -0,0 +1,4 @@
+{
+  "extends": "../../tsconfig.backend.json",
+  "include": ["**/*.js", "**/*.cjs", "**/*.ts"]
+}

services/web/cypress.config.ts

@@ -1,13 +1,12 @@
 import { defineConfig } from 'cypress'
 import { webpackConfig } from './cypress/support/webpack.cypress'
 
-// Cypress's Electron process uses PackherdModuleLoader which doesn't go through
+// Cypress builds reporters in its Electron process, which has no Yarn PnP.
-// PnP hooks. Reporter packages are materialized into /tmp/cypress-reporter by
+// @overleaf/cypress-pnp-reporter loads cypress-multi-reporters there.
-// scripts/materialize-cypress-reporter.js at container startup.
 let reporterOptions = {}
 if (process.env.CI) {
   reporterOptions = {
-    reporter: '/tmp/cypress-reporter/node_modules/cypress-multi-reporters',
+    reporter: require.resolve('@overleaf/cypress-pnp-reporter'),
     reporterOptions: {
       configFile: 'cypress/cypress-multi-reporters.json',
     },
@@ -29,7 +28,16 @@ export default defineConfig({
       webpackConfig,
     },
     setupNodeEvents(on, config) {
-      //
+      on('before:browser:launch', (browser, launchOptions) => {
+        if (browser.family === 'chromium') {
+          // Cypress adds this for docker's default 64MB /dev/shm; the compose services raise shm_size instead, so Chrome's shared memory stays in RAM.
+          const noDevShm = launchOptions.args.indexOf('--disable-dev-shm-usage')
+          if (noDevShm !== -1) {
+            launchOptions.args.splice(noDevShm, 1)
+          }
+        }
+        return launchOptions
+      })
     },
     specPattern:
       process.env.CYPRESS_SPEC_PATTERN ||

services/web/package.json

@@ -40,9 +40,8 @@
     "type-check:backend": "tsc -p tsconfig.backend.json --noEmit",
     "extract-translations": "i18next-scanner",
     "convert-themes": "node frontend/js/features/source-editor/themes/scripts/convert.js",
-    "materialize-reporter": "node ../../bin/shared/materialize-cypress-reporter.js /tmp/cypress-reporter",
+    "cypress:open-ct": "OVERLEAF_CONFIG=$(pwd)/config/settings.webpack.js cypress open --component",
-    "cypress:open-ct": "yarn run materialize-reporter && OVERLEAF_CONFIG=$(pwd)/config/settings.webpack.js cypress open --component",
+    "cypress:run-ct": "OVERLEAF_CONFIG=$(pwd)/config/settings.webpack.js cypress run --component --browser chrome",
-    "cypress:run-ct": "yarn run materialize-reporter && OVERLEAF_CONFIG=$(pwd)/config/settings.webpack.js cypress run --component --browser chrome",
     "cypress:docker:open-ct": "USER_UID=$(id -u) USER_GID=$(id -g) docker compose -f docker-compose.cypress.yml run --rm cypress yarn run cypress:open-ct",
     "cypress:docker:run-ct": "USER_UID=$(id -u) USER_GID=$(id -g) docker compose -f docker-compose.cypress.yml run --rm cypress yarn run cypress:run-ct --browser chrome",
     "lezer-latex:generate": "node scripts/lezer-latex/generate.mjs",
@@ -228,6 +227,7 @@
     "@lezer/lr": "1.4.7",
     "@lezer/markdown": "1.6.3",
     "@overleaf/codemirror-tree-view": "^0.1.3",
+    "@overleaf/cypress-pnp-reporter": "workspace:*",
     "@overleaf/dictionaries": "https://github.com/overleaf/dictionaries/archive/refs/tags/v0.0.3.tar.gz",
     "@overleaf/eslint-plugin": "workspace:*",
     "@overleaf/ranges-tracker": "workspace:*",
@@ -332,7 +332,6 @@
     "csv-parse": "^6.2.1",
     "csv-stringify": "^6.7.0",
     "cypress": "15.20.1",
-    "cypress-multi-reporters": "^2.0.5",
     "cypress-plugin-tab": "^1.0.5",
     "d3": "^3.5.16",
     "daterangepicker": "2.1.27",

services/web/webpack-plugins/serve-built-assets.js

@@ -0,0 +1,95 @@
+/**
+ * Serve each built asset from a prepared response, reused until it is rebuilt.
+ *
+ * webpack-dev-middleware redoes the same work on every request: it looks the
+ * file up in the in-memory filesystem, reads it, and hashes the whole body to
+ * produce a strong ETag. Most of that output does not change between builds, so
+ * it is recomputed for an answer already known - and under CI ~30 browsers
+ * request the same chunks over and over.
+ *
+ * Only what the compilation emitted is served from here, taken from its own
+ * asset list rather than an extension allowlist: a build emits .js and .css but
+ * also .bcmap, .dic, .aff, .pfb and .woff2, and anything else reaching this
+ * middleware belongs to somebody else.
+ *
+ * Two things a build does that this has to respect:
+ *
+ * - It reports `done` more than once - once for the compilation and once for the
+ *   assets-manifest plugin's own compilation, which has two assets in it - and
+ *   an incremental `done` reports only the assets it rewrote, 16 rather than 630.
+ *   So asset names only ever accumulate, and the gate on a running build is a
+ *   counter rather than a flag, which the manifest's `done` would reopen while
+ *   the compilation it belongs to is still running.
+ * - While a rebuild is running, webpack-dev-middleware holds requests until it
+ *   finishes. This sits in front of it, so it steps aside for that window and
+ *   drops the cache when a rebuild starts. Whether a response can be kept is
+ *   decided when it ends rather than when it began: an epoch, bumped on every
+ *   `invalid`, names the build its bytes were read from, and only a response
+ *   from the current build with no rebuild in flight is stored. Nothing a
+ *   rebuild has invalidated can land behind it, so every entry in the cache
+ *   belongs to the build that is current and none has to be evicted by name.
+ */
+module.exports = function serveBuiltAssets(devServer) {
+  const cache = new Map()
+  const known = new Set()
+  const compiler = devServer.compiler
+  let building = 0
+  let epoch = 0
+
+  compiler.hooks.invalid.tap('serve-built-assets', () => {
+    building++
+    epoch++
+    cache.clear()
+  })
+
+  compiler.hooks.done.tap('serve-built-assets', stats => {
+    building = Math.max(0, building - 1)
+    for (const asset of stats.toJson({ all: false, assets: true }).assets) {
+      if (asset.name) known.add('/' + asset.name)
+    }
+  })
+
+  return function serveBuiltAsset(req, res, next) {
+    if (building > 0) return next()
+    if (req.method !== 'GET' && req.method !== 'HEAD') return next()
+    // Ranges are served from the file itself; this only deals in whole bodies.
+    if (req.headers.range) return next()
+
+    const url = req.url.split('?')[0]
+    if (!known.has(url)) return next()
+
+    const hit = cache.get(url)
+    if (hit) {
+      if (req.headers['if-none-match'] === hit.headers.etag) {
+        res.statusCode = 304
+        return res.end()
+      }
+      for (const [name, value] of Object.entries(hit.headers)) {
+        res.setHeader(name, value)
+      }
+      res.statusCode = 200
+      return req.method === 'HEAD' ? res.end() : res.end(hit.body)
+    }
+
+    // Miss: let webpack-dev-middleware answer, and keep what it produced.
+    const readEpoch = epoch
+    const chunks = []
+    const write = res.write.bind(res)
+    const end = res.end.bind(res)
+    res.write = (chunk, ...rest) => {
+      if (chunk) chunks.push(Buffer.from(chunk))
+      return write(chunk, ...rest)
+    }
+    res.end = (chunk, ...rest) => {
+      if (chunk && typeof chunk !== 'function') chunks.push(Buffer.from(chunk))
+      const headers = res.getHeaders()
+      // Kept only while the build these bytes came from is the current one: a rebuild starting mid-response leaves them one behind.
+      const current = building === 0 && epoch === readEpoch
+      if (current && res.statusCode === 200 && headers.etag && chunks.length) {
+        cache.set(url, { body: Buffer.concat(chunks), headers })
+      }
+      return end(chunk, ...rest)
+    }
+    next()
+  }
+}

yarn.lock

@@ -7138,6 +7138,21 @@ __metadata:
   languageName: node
   linkType: hard
 
+"@overleaf/cypress-pnp-reporter@workspace:*, @overleaf/cypress-pnp-reporter@workspace:libraries/cypress-pnp-reporter":
+  version: 0.0.0-use.local
+  resolution: "@overleaf/cypress-pnp-reporter@workspace:libraries/cypress-pnp-reporter"
+  dependencies:
+    "@types/node": "npm:^24.5.2"
+    chai: "npm:^4.3.6"
+    cypress-multi-reporters: "npm:^2.0.5"
+    eslint: "npm:^10.4.0"
+    mocha: "npm:^11.1.0"
+    mocha-junit-reporter: "npm:^2.2.1"
+    mocha-multi-reporters: "npm:^1.5.1"
+    typescript: "npm:^5.0.4"
+  languageName: unknown
+  linkType: soft
+
 "@overleaf/dictionaries@https://github.com/overleaf/dictionaries/archive/refs/tags/v0.0.3.tar.gz":
   version: 0.0.3
   resolution: "@overleaf/dictionaries@https://github.com/overleaf/dictionaries/archive/refs/tags/v0.0.3.tar.gz"
@@ -7707,16 +7722,15 @@ __metadata:
     "@cypress/code-coverage": "npm:^4.0.2"
     "@cypress/grep": "npm:^6.0.2"
     "@isomorphic-git/lightning-fs": "npm:^4.6.0"
+    "@overleaf/cypress-pnp-reporter": "workspace:*"
     "@testing-library/cypress": "npm:^10.1.0"
     "@types/uuid": "npm:^9.0.8"
     adm-zip: "npm:^0.6.0"
     cypress: "npm:15.20.1"
-    cypress-multi-reporters: "npm:^2.0.5"
     eslint: "npm:^10.4.0"
     isomorphic-git: "npm:^1.33.1"
     mailtrap: "npm:^4.3.0"
     mocha: "npm:^11.7.5"
-    mocha-junit-reporter: "npm:^2.2.1"
     pdfjs-dist: "npm:^5.1.91"
     typescript: "npm:^5.0.4"
     uuid: "npm:^9.0.1"
@@ -7728,19 +7742,18 @@ __metadata:
   resolution: "@overleaf/server-ce-test@workspace:server-ce/test"
   dependencies:
     "@isomorphic-git/lightning-fs": "npm:^4.6.0"
+    "@overleaf/cypress-pnp-reporter": "workspace:*"
     "@overleaf/promise-utils": "workspace:*"
     "@overleaf/validation-tools": "workspace:*"
     "@testing-library/cypress": "npm:10.1.0"
     "@types/uuid": "npm:^9.0.8"
     adm-zip: "npm:^0.6.0"
     cypress: "npm:15.20.1"
-    cypress-multi-reporters: "npm:^2.0.5"
     eslint: "npm:^10.4.0"
     express: "npm:4.22.1"
     isomorphic-git: "npm:^1.33.1"
     js-yaml: "npm:^4.1.1"
     mocha: "npm:^11.7.5"
-    mocha-junit-reporter: "npm:^2.2.1"
     pdfjs-dist: "npm:^5.1.91"
     typescript: "npm:^5.0.4"
     uuid: "npm:^9.0.1"
@@ -7956,6 +7969,7 @@ __metadata:
     "@node-saml/passport-saml": "patch:@node-saml/passport-saml@npm%3A5.1.0#~/.yarn/patches/@node-saml-passport-saml-npm-5.1.0-5d7df575fa.patch"
     "@overleaf/access-token-encryptor": "workspace:*"
     "@overleaf/codemirror-tree-view": "npm:^0.1.3"
+    "@overleaf/cypress-pnp-reporter": "workspace:*"
     "@overleaf/dictionaries": "https://github.com/overleaf/dictionaries/archive/refs/tags/v0.0.3.tar.gz"
     "@overleaf/eslint-plugin": "workspace:*"
     "@overleaf/fetch-utils": "workspace:*"
@@ -8097,7 +8111,6 @@ __metadata:
     csv-parse: "npm:^6.2.1"
     csv-stringify: "npm:^6.7.0"
     cypress: "npm:15.20.1"
-    cypress-multi-reporters: "npm:^2.0.5"
     cypress-plugin-tab: "npm:^1.0.5"
     d3: "npm:^3.5.16"
     daterangepicker: "npm:2.1.27"