Files
pw/public
Asger Geel Weirsøe 2cc6baa43f Keep crawlers off the tarball archives, not the tarball index
SEO crawlers were downloading the release archives (up to about 2 GB each)
even though they are served noindex. Disallow only *.tar.zst and *.zip under
/tarballz/; the index page and the PKGBUILDs stay crawlable. Repeated in every
group, because a crawler with its own group ignores User-agent: *.

Claude-Session: https://claude.ai/code/session_01GGe7kUhooMFuzY6GaDb2w9
2026-10-05 21:30:00 +02:00
..