User-agent: * Sitemap: https://www.yourtier.com/sitemap.xml ############## # custom # ############## # Affiliate redirects (monetized, nothing to index behind them) Disallow: /r/ Disallow: /api/r/ # AJAX partials of the infinite scroll (HTML fragments, no layout) Disallow: /*fragment=1 # NOTE: the deep pages of the catalogue listing (page 2 and beyond) used to be # blocked here. That is exactly what FROZE them in the index: a URL blocked in # robots.txt is never re-fetched, so the crawler can never read the noindex # that would get it dropped, and Search Console reported them as "indexed # though blocked by robots.txt". They now serve their own # `X-Robots-Tag: noindex, follow` (mark_listing_noindex on `page > 1`, # controllers/games.py), which is readable and therefore actionable. The crawl # cost is real but bounded, and discovery of the pages worth indexing never # depended on them: sitemap-games.xml enumerates exactly the indexable set and # IndexNow pings each new one (models/indexnow.py). # NOTE — the catalogue facets (?genre= ?sort= ?view= ?platform= ?year= ?mode= # ?perspective= ?engine= ?mp= ?q= ?game= ?type= ?category=) are deliberately # NOT listed here. They are handled by an `X-Robots-Tag: noindex, follow` # served by the listings themselves (mark_listing_noindex, controllers/ # _common.py). Blocking them would be counter-productive right now: Google has # already collected ~3.2k of these URLs, and a blocked URL is never re-fetched, # so it can never read the directive that would get it dropped. Once those # buckets are empty, moving them here would save the recurring crawl. # Odoo internals Disallow: /website/translations ################## # Ad crawlers # ################## # AdSense reads the page to target the ads it serves on it. It must keep # access to everything the "User-agent: *" group blocks for crawl-budget # reasons — the catalogue facets carry ads (tw_ads_enabled on page 1), and a # page Mediapartners-Google cannot fetch gets untargeted or blank ads. # Google documents this group as the way to split "do not index" from # "do not monetize". User-agent: Mediapartners-Google Allow: / Disallow: /r/ Disallow: /api/r/ Disallow: /*fragment=1 Disallow: /website/translations Disallow: /web/login Disallow: /web/signup Disallow: /web/reset_password Disallow: /my/ ################## # AI crawlers # ################## # Tier welcomes AI assistants and LLM crawlers. # LLM site guide: https://www.yourtier.com/llms.txt # LLM full rankings data: https://www.yourtier.com/llms-full.txt # Retrieval / search bots — power AI answers and citations User-agent: OAI-SearchBot Allow: / Disallow: /r/ Disallow: /api/r/ Disallow: /*fragment=1 Disallow: /website/translations Disallow: /web/login Disallow: /web/signup Disallow: /web/reset_password Disallow: /my/ User-agent: ChatGPT-User Allow: / Disallow: /r/ Disallow: /api/r/ Disallow: /*fragment=1 Disallow: /website/translations Disallow: /web/login Disallow: /web/signup Disallow: /web/reset_password Disallow: /my/ User-agent: Claude-SearchBot Allow: / Disallow: /r/ Disallow: /api/r/ Disallow: /*fragment=1 Disallow: /website/translations Disallow: /web/login Disallow: /web/signup Disallow: /web/reset_password Disallow: /my/ User-agent: Claude-User Allow: / Disallow: /r/ Disallow: /api/r/ Disallow: /*fragment=1 Disallow: /website/translations Disallow: /web/login Disallow: /web/signup Disallow: /web/reset_password Disallow: /my/ User-agent: PerplexityBot Allow: / Disallow: /r/ Disallow: /api/r/ Disallow: /*fragment=1 Disallow: /website/translations Disallow: /web/login Disallow: /web/signup Disallow: /web/reset_password Disallow: /my/ User-agent: Perplexity-User Allow: / Disallow: /r/ Disallow: /api/r/ Disallow: /*fragment=1 Disallow: /website/translations Disallow: /web/login Disallow: /web/signup Disallow: /web/reset_password Disallow: /my/ # Google's non-search crawler (research and development fetches). It honours # its own token, and the access log of 2026-09-05 showed it -- not Googlebot -- # behind every /r/ fetch from a 66.249 address. User-agent: GoogleOther Allow: / Disallow: /r/ Disallow: /api/r/ Disallow: /*fragment=1 Disallow: /website/translations Disallow: /web/login Disallow: /web/signup Disallow: /web/reset_password Disallow: /my/ # Training bots User-agent: GPTBot Allow: / Disallow: /r/ Disallow: /api/r/ Disallow: /*fragment=1 Disallow: /website/translations Disallow: /web/login Disallow: /web/signup Disallow: /web/reset_password Disallow: /my/ User-agent: ClaudeBot Allow: / Disallow: /r/ Disallow: /api/r/ Disallow: /*fragment=1 Disallow: /website/translations Disallow: /web/login Disallow: /web/signup Disallow: /web/reset_password Disallow: /my/ User-agent: Google-Extended Allow: / Disallow: /r/ Disallow: /api/r/ Disallow: /*fragment=1 Disallow: /website/translations Disallow: /web/login Disallow: /web/signup Disallow: /web/reset_password Disallow: /my/ User-agent: Applebot-Extended Allow: / Disallow: /r/ Disallow: /api/r/ Disallow: /*fragment=1 Disallow: /website/translations Disallow: /web/login Disallow: /web/signup Disallow: /web/reset_password Disallow: /my/ User-agent: CCBot Allow: / Disallow: /r/ Disallow: /api/r/ Disallow: /*fragment=1 Disallow: /website/translations Disallow: /web/login Disallow: /web/signup Disallow: /web/reset_password Disallow: /my/ # Meta's crawler is the one that actually hurts: on 2026-08-12 it produced # every 502 in the Caddy log, firing bursts of parallel requests at localised # catalogue URLs (/es/games//players, /fr/lists?game=…) until the Odoo # workers timed out. We do not block it — welcoming AI crawlers is a # deliberate GEO decision and this is one of them — we ask it to slow down. # Crawl-delay is non-standard (Bing honours it, Google ignores it, Meta does # not document its stance) and costs nothing to send. It belongs in THIS # group, not a second one: two groups for the same user-agent is ambiguous # under RFC 9309, and a robot that picks the first would have lost the # Disallow rules. If the delay proves insufficient, the real fix is rate # limiting at the edge — which this Caddy build cannot do without an xcaddy # rebuild (no rate_limit module). User-agent: meta-externalagent Crawl-delay: 5 Allow: / Disallow: /r/ Disallow: /api/r/ Disallow: /*fragment=1 Disallow: /website/translations Disallow: /web/login Disallow: /web/signup Disallow: /web/reset_password Disallow: /my/