a648015db8
* Merge all .prettierignore files into top-level config * Merge all .prettierrc files into top-level config * Replace service-specific glob patterns in package.json format scripts with `prettier .` * Add template files with Jinja2, Go template, envsubst, and Handlebars syntax to .prettierignore * Ignore GitHub templates * Ignore PUG templates to format them separately with `format:pug` * Encourage double quotes for YAML, YML files * Move prettier for PUG source format script to the root * Move prettier for styles source format script to the root * Remove prettier for jenkins files from web * Remove prettier source format script from all services * Make .prettierrc more readable * Update format scripts by file type * Organise `.prettierignore` * Add `--cache` flag to prettier scripts for faster runs * Format all files with prettier * Format all or format services * Remove `format`/`format:fix` scripts from services since now it runs from root `package.json` * Avoid conlficts with yamllint configuration * Remove `--cache` flag from prettier scripts * Update all service Makefiles to use root-level prettier configuration * Update all Jenkinsfile to use root-level prettier configuration * Ignore auto-generated files by build_scripts * Update package-lock.json * Update root Makefile format targets * Update SP Jenkinsfile format target * Update E2E Makefile format script * Udpate `format_js` to work in both local and CI env * Add docker-mailtrap to .prettierignore docker-mailtrap is a third-party git-ignored directory used for testing * Added Docker env detection to prevent nested Docker spawning * Ignore handlebars templates * Add cryptographic files and test output to `.prettierignore` * Add terraform modules to `.gitignore` * Remove prettier-plugin-groovy * Use npx directly instead of Docker for local formatting for faster formatting * Auto-generate Makefiles * Revert "Remove prettier-plugin-groovy" This reverts commit 194a33589a2e1e4d2225d10c67e9f025e4222025. * Mount monorepo root in RUN_LINT_FORMAT for prettier config access * Prettier ignores all `node_modules` by default regardless of location * Show only changed files in format output * Ignore LICENSE files * Enable prettier on rendered build_scripts outputs * Ignoring all the template folders by prettier * Remove the public/minjs entry since it does not exist * Remove all non-existent paths * Sync `.prettierignore` with ignored files by `.gitignore` and `.dockerignore` files * Revert "Auto-generate Makefiles" This reverts commit c0233e490de1bc95fe437219d65e0b66e0331ec9. * Revert "Use npx directly instead of Docker for local formatting for faster formatting" This reverts commit 1d2b2cf1a6c6974c76885852a90dd55e84167e41. * Ignore dashboard JSON files * Ignore files generated by bin/update_build_scripts * Remove unsupported file types from `.prettierignore` * Ignore test fixture generated files * Ignore README file types by prettier * Ignore generate snapshots by prettier * Allow to format generated bin/update_build_scripts by prettier * Ensure build script outputs prettier-compatible tsconfig.json * Fix build script output to match prettier formatting - Fix Jinja2 whitespace in docker-compose templates - Change YAML quotes from single to double * Don't read cryptographic files by prettier * Ignore google verification files by prettier * Revert npx prettier formatting * Ignore domain verification files * Show only changed files in format output * Make `.github` prettier * Allow all files to be formatted in jobs by prettier * Allow server-ce/server-pro files to be formatted by prettier * Ignore more folders in clsi, filestory, git-bridge by prettier * Update build script with `RUN_LINTING_CI_MONOREPO` * Ignore docker-mailtrap and downloads in server-ce by prettier * Restore prettier configs and prettierignore for V1 since it has its own prettier (an older version) * Source format GitOrigin-RevId: 637adc3cc422d1f20c86d6ebc8ec514d60758287
Lezer-LaTeX, a LaTeX Parser
Lezer-LaTeX is a LaTeX parser implemented with lezer, the parser system used by CodeMirror 6.
The parser is written in a "grammar" file, (and a "tokens" file with custom tokenizer logic) which is then compiled by @lezer/generator into a parser module and a "terms" module. The parser module is then loaded by the CodeMirror 6 in the web frontend codebase.
Important files
-
Source files:
./latex.grammar: The grammar file, containing the specification for the parser./tokens.mjs: The custom tokenizer logic, required by some rules in the grammar
-
Generated files:
./latex.mjs: The generated parser./latex.terms.mjs: The generated terms file- (these files are ignored by git, eslint, and prettier)
-
Scripts:
web/scripts/lezer-latex/generate.js: A script which runs the generator on the grammar, producing the generated parser/terms filesweb/scripts/lezer-latex/run.mjs: A script that runs the parser against a supplied file, and prints the tree to the terminal
-
Webpack plugins:
web/webpack-plugins/lezer-grammar-compiler.js: A webpack plugin that calls the generator as part of the webpack build. In dev, it will automatically re-build the parser when the grammar file changes.
NPM tasks
-
lezer-latex:generate: Generate the parser files from the grammar- (Calls
lezer-latex/generate.js) - This should be run whenever the grammar changes
- (Calls
-
lezer-latex:run: Run the parser against a file- (Calls
lezer-latex/run.js)
- (Calls
Generating the parser
From the monorepo root:
# automatic (on changes)
make install
# manually
bin/npm -w services/web run 'lezer-latex:generate'
Tests
Unit tests for the parser live in web/test/unit/src/LezerLatex. There are three kinds of test, in three subdirectories:
corpus/: A set of tests using lezer's test framework, consisting of example text and the expected parse treeexamples/: A set of realistic LaTeX documents. These tests pass if the files parse with no errorsregressions/: Likeexamples/, these are expected to parse without error, but they are not realistic documents.
These tests run as part of test_frontend. You can run these tests alone by invoking:
make test_unit MOCHA_GREP='lezer-latex'
Trying the parser
While developing the parser, you can run it against a file by calling the lezer-latex:run task. There are
some example files in the test suite, at web/test/unit/src/LezerLatex/examples/.
For example:
bin/npm -w services/web run 'lezer-latex:run' web/test/unit/src/LezerLatex/examples/amsmath.tex
If you omit the file path, the default file (examples/demo.tex) will be run.
Integration into web
The web frontend imports the parser (from latex.mjs), in frontend/js/features/source-editor/languages/latex/index.ts.
The parser is then plugged in to the CM6 language system.
The web build
In web/Dockerfile, we have a RUN command that calls lezer-latex:generate as part of the build. This is necessary to ensure the parser is built before the CI tests run (notably: we can't do the build during the tests, because we can't write to disk during that stage of CI).