Asger Geel Weirsøe
2cc6baa43f
SEO crawlers were downloading the release archives (up to about 2 GB each) even though they are served noindex. Disallow only *.tar.zst and *.zip under /tarballz/; the index page and the PKGBUILDs stay crawlable. Repeated in every group, because a crawler with its own group ignores User-agent: *. Claude-Session: https://claude.ai/code/session_01GGe7kUhooMFuzY6GaDb2w9
73 lines
1.6 KiB
Plaintext
73 lines
1.6 KiB
Plaintext
# Asger Weirsøe — asger.weirsoe.dk
|
|
# Alle crawlere er velkomne, inkl. AI/LLM-crawlere (vi vil gerne findes og citeres).
|
|
# Undtagelse: selve arkivfilerne under /tarballz/ (op til ~2 GB stykket). Indeks og
|
|
# PKGBUILD'er må gerne crawles. Står i hver gruppe, fordi en crawler med sin egen
|
|
# gruppe ignorerer User-agent: *.
|
|
User-agent: *
|
|
Allow: /
|
|
Disallow: /tarballz/*.tar.zst
|
|
Disallow: /tarballz/*.zip
|
|
|
|
# Eksplicit velkomst til AI/LLM-crawlere
|
|
User-agent: GPTBot
|
|
Allow: /
|
|
Disallow: /tarballz/*.tar.zst
|
|
Disallow: /tarballz/*.zip
|
|
|
|
User-agent: OAI-SearchBot
|
|
Allow: /
|
|
Disallow: /tarballz/*.tar.zst
|
|
Disallow: /tarballz/*.zip
|
|
|
|
User-agent: ChatGPT-User
|
|
Allow: /
|
|
Disallow: /tarballz/*.tar.zst
|
|
Disallow: /tarballz/*.zip
|
|
|
|
User-agent: ClaudeBot
|
|
Allow: /
|
|
Disallow: /tarballz/*.tar.zst
|
|
Disallow: /tarballz/*.zip
|
|
|
|
User-agent: Claude-Web
|
|
Allow: /
|
|
Disallow: /tarballz/*.tar.zst
|
|
Disallow: /tarballz/*.zip
|
|
|
|
User-agent: anthropic-ai
|
|
Allow: /
|
|
Disallow: /tarballz/*.tar.zst
|
|
Disallow: /tarballz/*.zip
|
|
|
|
User-agent: PerplexityBot
|
|
Allow: /
|
|
Disallow: /tarballz/*.tar.zst
|
|
Disallow: /tarballz/*.zip
|
|
|
|
User-agent: Perplexity-User
|
|
Allow: /
|
|
Disallow: /tarballz/*.tar.zst
|
|
Disallow: /tarballz/*.zip
|
|
|
|
User-agent: Google-Extended
|
|
Allow: /
|
|
Disallow: /tarballz/*.tar.zst
|
|
Disallow: /tarballz/*.zip
|
|
|
|
User-agent: Applebot-Extended
|
|
Allow: /
|
|
Disallow: /tarballz/*.tar.zst
|
|
Disallow: /tarballz/*.zip
|
|
|
|
User-agent: CCBot
|
|
Allow: /
|
|
Disallow: /tarballz/*.tar.zst
|
|
Disallow: /tarballz/*.zip
|
|
|
|
User-agent: cohere-ai
|
|
Allow: /
|
|
Disallow: /tarballz/*.tar.zst
|
|
Disallow: /tarballz/*.zip
|
|
|
|
Sitemap: https://asger.xn--weirse-fya.dk/sitemap-index.xml
|