agw/pw
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Keep crawlers off the tarball archives, not the tarball index

Browse Source 
SEO crawlers were downloading the release archives (up to about 2 GB each)
even though they are served noindex. Disallow only *.tar.zst and *.zip under
/tarballz/; the index page and the PKGBUILDs stay crawlable. Repeated in every
group, because a crawler with its own group ignores User-agent: *.

Claude-Session: https://claude.ai/code/session_01GGe7kUhooMFuzY6GaDb2w9
This commit is contained in:
Asger Geel Weirsøe
2026-10-05 21:30:00 +02:00
parent 58d68aa95f
commit 2cc6baa43f
1 changed files with 29 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
public/robots.txt
+29
View File
@@ -1,43 +1,72 @@
# Asger Weirsøe — asger.weirsoe.dk
# Alle crawlere er velkomne, inkl. AI/LLM-crawlere (vi vil gerne findes og citeres).
# Undtagelse: selve arkivfilerne under /tarballz/ (op til ~2 GB stykket). Indeks og
# PKGBUILD'er må gerne crawles. Står i hver gruppe, fordi en crawler med sin egen
# gruppe ignorerer User-agent: *.
User-agent: *
Allow: /
Disallow: /tarballz/*.tar.zst
Disallow: /tarballz/*.zip
# Eksplicit velkomst til AI/LLM-crawlere
User-agent: GPTBot
Allow: /
Disallow: /tarballz/*.tar.zst
Disallow: /tarballz/*.zip
User-agent: OAI-SearchBot
Allow: /
Disallow: /tarballz/*.tar.zst
Disallow: /tarballz/*.zip
User-agent: ChatGPT-User
Allow: /
Disallow: /tarballz/*.tar.zst
Disallow: /tarballz/*.zip
User-agent: ClaudeBot
Allow: /
Disallow: /tarballz/*.tar.zst
Disallow: /tarballz/*.zip
User-agent: Claude-Web
Allow: /
Disallow: /tarballz/*.tar.zst
Disallow: /tarballz/*.zip
User-agent: anthropic-ai
Allow: /
Disallow: /tarballz/*.tar.zst
Disallow: /tarballz/*.zip
User-agent: PerplexityBot
Allow: /
Disallow: /tarballz/*.tar.zst
Disallow: /tarballz/*.zip
User-agent: Perplexity-User
Allow: /
Disallow: /tarballz/*.tar.zst
Disallow: /tarballz/*.zip
User-agent: Google-Extended
Allow: /
Disallow: /tarballz/*.tar.zst
Disallow: /tarballz/*.zip
User-agent: Applebot-Extended
Allow: /
Disallow: /tarballz/*.tar.zst
Disallow: /tarballz/*.zip
User-agent: CCBot
Allow: /
Disallow: /tarballz/*.tar.zst
Disallow: /tarballz/*.zip
User-agent: cohere-ai
Allow: /
Disallow: /tarballz/*.tar.zst
Disallow: /tarballz/*.zip
Sitemap: https://asger.xn--weirse-fya.dk/sitemap-index.xml