agw/jasp-desktop-bin
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Keep infrastructure details out of the repository

Browse Source 
The README, the publish script and the workflow named the tarball host, its
address and filesystem paths on the web host and the runner. Hosts and paths
now come only from repository variables, and the workflow shares the work
tree and the build cache through the named Docker volumes jasp-build and
jasp-cache instead of host paths. (The runner does not expand ${{ vars }} in
container.volumes, so a variable could not have replaced the host path there.)

Also drop the unused scripts/gen-tarball-index.py, and give each module build
its own process under a time limit: a module once spun for eight hours in
jaspModuleTools' bundling step and stalled the whole build.

Claude-Session: https://claude.ai/code/session_01GGe7kUhooMFuzY6GaDb2w9
This commit is contained in:
agw
2026-10-01 17:05:00 +02:00
parent e8c3d775aa
commit aa08dc108e
6 changed files with 45 additions and 227 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitea/workflows/build.yml
+11 -10
View File
@@ -35,7 +35,7 @@ env:
TARBALL_HOST: ${{ vars.TARBALL_HOST }} TARBALL_HOST: ${{ vars.TARBALL_HOST }}
TARBALL_PATH: ${{ vars.TARBALL_PATH }} TARBALL_PATH: ${{ vars.TARBALL_PATH }}
TARBALL_BASE_URL: ${{ vars.TARBALL_BASE_URL }} TARBALL_BASE_URL: ${{ vars.TARBALL_BASE_URL }}
WORK_DIR: /srv/jasp-build WORK_DIR: /work
jobs: jobs:
check: check:
@@ -62,15 +62,16 @@ jobs:
# full build runs for hours. The act_runner timeout and Gitea's # full build runs for hours. The act_runner timeout and Gitea's
# ENDLESS_TASK_TIMEOUT must both allow for this (see README). # ENDLESS_TASK_TIMEOUT must both allow for this (see README).
timeout-minutes: 900 timeout-minutes: 900
# `docker run -v ${WORK_DIR}:/work` below talks to the runner host's docker, # The build step's `docker run` talks to the runner's docker daemon, not to
# so it mounts the *host's* WORK_DIR. Without this bind the job container # this job container, so the two only share files through a Docker volume.
# has its own private copy: the scripts it prepares never reach the build # Without it the scripts prepared here never reach the build and the
# (which then ran a stale script left on the host) and the tarball the # tarball the build writes is invisible to the publish steps. Named volumes
# build writes is invisible to the publish steps. # rather than host paths: the runner does not expand ${{ vars }} in
# container.volumes, and a host path would put runner internals in the repo.
container: container:
image: docker.gitea.com/runner-images:ubuntu-latest image: docker.gitea.com/runner-images:ubuntu-latest
volumes: volumes:
- /srv/jasp-build:/srv/jasp-build - jasp-build:/work
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
@@ -81,7 +82,7 @@ jobs:
- name: Prepare work directory - name: Prepare work directory
run: | run: |
# WORK_DIR is a bind mount now, so empty it rather than removing it. # WORK_DIR is a volume mount point, so empty it rather than removing it.
sudo find "${WORK_DIR}" -mindepth 1 -delete sudo find "${WORK_DIR}" -mindepth 1 -delete
sudo mkdir -p "${WORK_DIR}/out" sudo mkdir -p "${WORK_DIR}/out"
sudo cp -a build packaging scripts "${WORK_DIR}/" sudo cp -a build packaging scripts "${WORK_DIR}/"
@@ -93,8 +94,8 @@ jobs:
run: | run: |
docker rm -f jasp-build 2>/dev/null || true docker rm -f jasp-build 2>/dev/null || true
docker run --rm --name jasp-build \ docker run --rm --name jasp-build \
-v "${WORK_DIR}:/work" \ -v jasp-build:/work \
-v /srv/jasp-cache:/cache \ -v jasp-cache:/cache \
-e JASP_TAG="${{ needs.check.outputs.jasp_tag }}" \ -e JASP_TAG="${{ needs.check.outputs.jasp_tag }}" \
-e OUT_DIR=/work/out \ -e OUT_DIR=/work/out \
-e JOBS=5 \ -e JOBS=5 \
.gitignore
+2
View File
@@ -1,3 +1,5 @@
release-meta.json release-meta.json
__pycache__/ __pycache__/
*.pyc *.pyc
PKGBUILD
build-info.json
README.md
+15 -17
View File
@@ -96,30 +96,27 @@ Set these in the Gitea repo before the first run.
| Name | Example | Meaning | | Name | Example | Meaning |
| --- | --- | --- | | --- | --- | --- |
| `AUR_MAINTAINER` | `Asger Geel Weirsoe <asger at weircon dot dk>` | Written into the PKGBUILD header. The guidelines suggest obfuscating the address against spam harvesting. | | `AUR_MAINTAINER` | `Asger Geel Weirsoe <asger at weircon dot dk>` | Written into the PKGBUILD header. The guidelines suggest obfuscating the address against spam harvesting. |
| `TARBALL_HOST` | `192.168.10.148` | CT148, where tarballs are uploaded | | `TARBALL_HOST` | `files.example.org` | SSH host the tarballs are uploaded to |
| `TARBALL_PATH` | `/srv/tarballz` | Directory on that host | | `TARBALL_PATH` | `/path/to/tarballs` | Directory on that host that the web server publishes |
| `TARBALL_BASE_URL` | `https://asger.weirsoe.dk/tarballz` | Public URL the PKGBUILD downloads from | | `TARBALL_BASE_URL` | `https://asger.weirsoe.dk/tarballz` | Public URL the PKGBUILD downloads from |
| `PUBLISH` | `true` | Master switch. Unless it is `true`, a run only builds and verifies, leaving the tarball in `/srv/jasp-build/out`: no upload, no `state.json` commit, no AUR push. | | `PUBLISH` | `true` | Master switch. Unless it is `true`, a run only builds and verifies, leaving the tarball in the `jasp-build` volume under `out/`: no upload, no `state.json` commit, no AUR push. |
**Secrets** (Settings → Actions → Secrets): **Secrets** (Settings → Actions → Secrets):
| Name | Meaning | | Name | Meaning |
| --- | --- | | --- | --- |
| `GH_TOKEN` | *Optional.* A GitHub PAT, passed to the build as `GITHUB_PAT`. Not used for publishing — several JASP R modules install from GitHub and anonymous API rate limits can fail the build. | | `GH_TOKEN` | *Optional.* A GitHub PAT, passed to the build as `GITHUB_PAT`. Not used for publishing — several JASP R modules install from GitHub and anonymous API rate limits can fail the build. |
| `TARBALL_SSH_KEY` | Private key authorised on CT148 for uploading tarballs | | `TARBALL_SSH_KEY` | Private key authorised on `TARBALL_HOST` for uploading tarballs |
| `AUR_SSH_PRIVATE_KEY` | Private SSH key registered with your AUR account. Copy the value already set on `agw/pi3-smart-workspace`. | | `AUR_SSH_PRIVATE_KEY` | Private SSH key registered with your AUR account. Copy the value already set on `agw/pi3-smart-workspace`. |
| `STATE_PUSH_TOKEN` | Gitea token with write access to this repo, for committing `state.json`. (Gitea reserves the `GITEA_` secret prefix.) | | `STATE_PUSH_TOKEN` | Gitea token with write access to this repo, for committing `state.json`. (Gitea reserves the `GITEA_` secret prefix.) |
### Where the tarballs live ### Where the tarballs live
Tarballs are served from `https://asger.weirsoe.dk/tarballz/`, backed by Tarballs are served from `https://asger.weirsoe.dk/tarballz/`. They are kept
`/srv/tarballz` on CT148. outside the website's own web root: the site deploy replaces its web root
atomically, so anything stored inside it would vanish on the next deploy. The
They are **not** stored under `/var/www/asger`. The `asger-web` deploy replaces web server exposes the tarball directory through a separate `/tarballz/`
that entire directory atomically — it unpacks into `.new`, swaps it in and location instead, with gzip off so makepkg gets a plain byte stream.
deletes the old one — so anything kept inside it would vanish on the next site
deploy. nginx exposes `/srv/tarballz` through a `location /tarballz/` alias
instead, with `gzip off` so makepkg gets a plain byte stream.
Note that `asger.weirsoe.dk` 301-redirects to the IDN form Note that `asger.weirsoe.dk` 301-redirects to the IDN form
`asger.xn--weirse-fya.dk`. makepkg follows redirects, so the readable hostname is `asger.xn--weirse-fya.dk`. makepkg follows redirects, so the readable hostname is
@@ -135,17 +132,18 @@ pointing at a missing or truncated file.
The build runs for hours and needs disk. On the Gitea runner host: The build runs for hours and needs disk. On the Gitea runner host:
- `runner.timeout` in `/var/lib/gitea-worker/config.yaml` must exceed the build - `runner.timeout` in the runner's `config.yaml` must exceed the build time
time (set to `16h`). (set to `16h`).
- Gitea's own job timeout (`[actions] ENDLESS_TASK_TIMEOUT` in `app.ini`, default - Gitea's own job timeout (`[actions] ENDLESS_TASK_TIMEOUT` in `app.ini`, default
3h) must match, otherwise Gitea kills the job at 3h regardless of the runner. 3h) must match, otherwise Gitea kills the job at 3h regardless of the runner.
- Roughly 40–60 GB of free disk for the build tree, the bundled R and the - Roughly 40–60 GB of free disk for the build tree, the bundled R and the
staged install. staged install.
- `/srv/jasp-cache` on the runner host, mounted into the build as `/cache`. - The Docker volume `jasp-cache`, mounted into the build as `/cache`. It holds ccache and the compiled R package cache (renv), and is what keeps a
It holds ccache and the compiled R package cache (renv), and is what keeps a
rebuild from recompiling 1000+ R packages from scratch. It is never wiped by rebuild from recompiling 1000+ R packages from scratch. It is never wiped by
the workflow; entries whose system libraries have gone missing after an Arch the workflow; entries whose system libraries have gone missing after an Arch
update are purged at the start of each build. Delete it to force a clean build. update are purged at the start of each build. `docker volume rm jasp-cache`
forces a clean build. A second volume, `jasp-build`, holds the work tree and
output; the workflow empties it at the start of every run.
- **Swap.** `rstanarm` (required by `jaspMixedModels`) and `OpenMx` generate - **Swap.** `rstanarm` (required by `jaspMixedModels`) and `OpenMx` generate
translation units where a single `cc1plus` wants 4–8 GB. Without swap an translation units where a single `cc1plus` wants 4–8 GB. Without swap an
over-commit is an instant OOM kill, which surfaces only as a module that never over-commit is an instant OOM kill, which surfaces only as a module that never
build/build-jasp.sh
+10 -1
View File
@@ -422,7 +422,16 @@ for attempt in 1 2 3; do
done done
[[ ${#MODULE_PATHS[@]} -eq 0 ]] && break [[ ${#MODULE_PATHS[@]} -eq 0 ]] && break
log "module build pass ${attempt}/3: ${#MODULE_PATHS[@]} modules left" log "module build pass ${attempt}/3: ${#MODULE_PATHS[@]} modules left"
Rscript /work/build/build-modules.R "${MODULE_PATHS[@]}" || true # One R process per module under a time limit. A module once spun at 100%
# CPU for 8 hours inside jaspModuleTools' bundling step, after it had
# installed fine, and stalled the whole build; the same module bundled in two
# minutes on the previous run. A timed-out module is simply retried in the
# next pass.
for path in "${MODULE_PATHS[@]}"; do
timeout --kill-after=60 "${MODULE_TIMEOUT:-5400}" \
Rscript /work/build/build-modules.R "$path" \
|| log "$(basename "$path") did not finish (exit $?), retrying next pass"
done
built=0 built=0
for module in "${ALL_MODULES[@]}"; do for module in "${ALL_MODULES[@]}"; do
[[ -f "${MODULE_BUNDLES}/${module}.JASPModule" ]] && built=$((built + 1)) [[ -f "${MODULE_BUNDLES}/${module}.JASPModule" ]] && built=$((built + 1))
scripts/gen-tarball-index.py
-192
View File
@@ -1,192 +0,0 @@
#!/usr/bin/env python3
"""Render the index page for https://asger.weirsoe.dk/tarballz/.
Reads the sidecar .json files that publish-tarball.sh writes next to each
tarball (on stdin, one JSON object per line) and emits a standalone HTML page.
The theme tokens are inlined rather than linked to the main site's stylesheet:
that file is content-hashed (_astro/_slug_.<hash>.css) and its name changes on
every site rebuild, so a link would silently rot. Inlining costs a few hundred
bytes and cannot break.
"""
import html
import json
import os
import sys
from datetime import datetime, timezone
GITEA_URL = "https://gitea.weircon.dk/agw/jasp-desktop-bin"
AUR_URL = "https://aur.archlinux.org/packages/jasp-desktop-bin"
PAGE = """<!doctype html>
<html lang="da" data-theme="dark">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Tarballs · Asger Weirsøe</title>
<meta name="description" content="Prækompilerede JASP-binærer til Arch Linux, bygget automatisk og udgivet på AUR som jasp-desktop-bin.">
<meta name="robots" content="noindex">
<link rel="icon" type="image/svg+xml" href="https://asger.weirsoe.dk/favicon.svg">
<script>
// Same localStorage key the main site uses, so the chosen theme carries over.
// Inline and before paint, otherwise the page flashes the wrong theme.
try {
var t = localStorage.getItem('theme');
if (!t) t = matchMedia('(prefers-color-scheme: light)').matches ? 'light' : 'dark';
document.documentElement.dataset.theme = t;
} catch (e) {}
</script>
<style>
:root, :root[data-theme=dark] {
color-scheme: dark;
--bg:#0b0f17; --surface:#121a27; --surface-2:#182232;
--border:#243044; --text:#e6e9ef; --muted:#97a3b6; --accent:#38bdf8;
}
:root[data-theme=light] {
color-scheme: light;
--bg:#f6f8fb; --surface:#fff; --surface-2:#eef2f8;
--border:#dde4ee; --text:#0f1729; --muted:#54617a; --accent:#0284c7;
}
:root[data-theme=terminal] {
color-scheme: dark;
--bg:#050a06; --surface:#0a140c; --surface-2:#102012;
--border:#1d3a22; --text:#4bff7e; --muted:#2fae57; --accent:#7dffa8;
}
* { box-sizing: border-box; }
body {
margin:0; background:var(--bg); color:var(--text);
font-family:"Inter Variable", ui-sans-serif, system-ui, -apple-system, sans-serif;
-webkit-font-smoothing:antialiased; line-height:1.6;
}
code, .mono { font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace; }
a { color:var(--accent); text-decoration:none; }
a:hover { text-decoration:underline; }
header { border-bottom:1px solid color-mix(in srgb, var(--border) 70%, transparent); }
.wrap { max-width:48rem; margin:0 auto; padding:0 1.25rem; }
header .wrap { display:flex; align-items:center; justify-content:space-between; padding-top:.875rem; padding-bottom:.875rem; }
.brand { font-weight:700; font-size:1.125rem; letter-spacing:-.025em; color:var(--text); }
.brand span { color:var(--accent); }
main { padding:3rem 0 1rem; }
h1 { font-size:2.25rem; font-weight:700; letter-spacing:-.025em; margin:0; }
.lede { color:var(--muted); margin-top:.75rem; }
.card { border:1px solid var(--border); border-radius:1rem; background:color-mix(in srgb, var(--surface) 60%, transparent); padding:1.25rem; }
.install { margin-top:1.75rem; }
.install code { display:block; background:var(--surface-2); border-radius:.55rem; padding:.6rem .8rem; font-size:.875rem; overflow-x:auto; }
h2 { font-size:1.25rem; font-weight:600; letter-spacing:-.025em; margin:2.5rem 0 1rem; }
ul.releases { list-style:none; margin:0; padding:0; display:grid; gap:.75rem; }
ul.releases li { border:1px solid var(--border); border-radius:1rem; background:color-mix(in srgb, var(--surface) 60%, transparent); padding:1rem 1.25rem; }
.row { display:flex; flex-wrap:wrap; align-items:baseline; gap:.6rem; justify-content:space-between; }
.ver { font-weight:600; font-size:1.05rem; }
.meta { color:var(--muted); font-size:.8rem; }
.sha { color:var(--muted); font-size:.75rem; word-break:break-all; margin-top:.5rem; }
.empty { color:var(--muted); }
footer { border-top:1px solid color-mix(in srgb, var(--border) 70%, transparent); margin-top:4rem; }
footer .wrap { padding-top:1.5rem; padding-bottom:2rem; color:color-mix(in srgb, var(--muted) 80%, transparent); font-size:.8rem; }
footer .note { font-size:.75rem; }
@media (max-width:480px) { h1 { font-size:1.75rem; } }
</style>
</head>
<body>
<header><div class="wrap">
<a class="brand" href="https://asger.weirsoe.dk/">Asger Weirsøe<span>.</span></a>
<nav class="meta"><a href="https://asger.weirsoe.dk/projekter">Projekter</a></nav>
</div></header>
<main class="wrap">
<h1>Tarballs</h1>
<p class="lede">
Prækompilerede <a href="https://jasp-stats.org/">JASP</a>-binærer til Arch Linux.
JASP udgiver kun installere til Windows og macOS, så disse er bygget fra
kildekoden af <a href="{gitea}">en automatisk pipeline</a> og udgivet på AUR som
<a href="{aur}">jasp-desktop-bin</a>.
</p>
<div class="install">
<p class="meta">Du behøver normalt ikke hente herfra manuelt — brug AUR:</p>
<code>yay -S jasp-desktop-bin</code>
</div>
<h2>Tilgængelige builds</h2>
{releases}
<p class="meta" style="margin-top:1.5rem">
Hver build indeholder sin egen R og de biblioteker den linker mod, så den ikke
bindes til bestemte versioner af noget på dit system.
Kun de nyeste {keep} builds gemmes.
</p>
</main>
<footer><div class="wrap">
<p class="note mono">bygget automatisk · self-hosted på proxmox · <a href="{gitea}">byggeopskrift</a></p>
<p>© {year} Asger Weirsøe</p>
</div></footer>
</body>
</html>
"""
def human_size(n: int) -> str:
for unit in ("B", "KB", "MB", "GB"):
if n < 1024 or unit == "GB":
return f"{n:.1f} {unit}".replace(".0 ", " ")
n /= 1024
return f"{n} B"
def main() -> int:
entries = []
for line in sys.stdin:
line = line.strip()
if not line:
continue
try:
entries.append(json.loads(line))
except json.JSONDecodeError:
print(f"warning: skipping unparsable sidecar line", file=sys.stderr)
# Newest first.
entries.sort(key=lambda e: e.get("built", ""), reverse=True)
if entries:
items = []
for e in entries:
name = html.escape(e["tarball"])
ver = html.escape(e.get("jasp_version", "?"))
built = e.get("built", "")
try:
built_h = datetime.fromisoformat(built).strftime("%d-%m-%Y")
except ValueError:
built_h = html.escape(built)
size = human_size(int(e.get("size_bytes", 0)))
sha = html.escape(e.get("sha256", ""))
r = html.escape(e.get("bundled_r_version", ""))
qt = html.escape(e.get("qt6_minor", ""))
items.append(
f' <li>\n'
f' <div class="row">\n'
f' <span class="ver">JASP {ver}</span>\n'
f' <a href="{name}" download>Hent · {size}</a>\n'
f' </div>\n'
f' <div class="meta">{built_h} · x86_64 · bundlet R {r} · bygget mod Qt {qt}</div>\n'
f' <div class="sha mono">sha256 {sha}</div>\n'
f' </li>'
)
releases = '<ul class="releases">\n' + "\n".join(items) + "\n </ul>"
else:
releases = '<p class="empty">Ingen builds tilgængelige lige nu.</p>'
sys.stdout.write(
PAGE.format(
releases=releases,
gitea=GITEA_URL,
aur=AUR_URL,
keep=len(entries) if entries else 3,
year=datetime.now(timezone.utc).year,
)
)
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/publish-tarball.sh
+7 -7
View File
@@ -2,16 +2,16 @@
# Publish the built tarball to asger.weirsoe.dk, which is what the AUR PKGBUILD # Publish the built tarball to asger.weirsoe.dk, which is what the AUR PKGBUILD
# downloads from. # downloads from.
# #
# TARBALL_SSH_KEY private key authorised on the web host (required) # TARBALL_SSH_KEY private key authorised on the web host (required)
# TARBALL_HOST web host, e.g. 192.168.10.148 (required) # TARBALL_HOST web host to upload to over SSH (required)
# TARBALL_PATH directory on that host (required) # TARBALL_PATH directory on that host (required)
# KEEP_RELEASES how many tarballs to retain (default 3) # KEEP_RELEASES how many tarballs to retain (default 3)
# $1 path to the tarball to upload # $1 path to the tarball to upload
# #
# The files live in /srv/tarballz, deliberately NOT under /var/www/asger: the # TARBALL_PATH is deliberately outside the website's own web root: the site
# asger-web deploy replaces that directory atomically (build into .new, swap, # deploy replaces its web root atomically, so anything stored inside it would
# delete .old), so anything stored inside it disappears on the next site deploy. # disappear on the next deploy. The web server publishes the directory through
# nginx exposes /srv/tarballz via a `location /tarballz/` alias. # a separate /tarballz/ location.
set -euo pipefail set -euo pipefail