# Crawl policy for this site.
#
# One policy for every crawler: the public marketing pages are open; the app
# routes and per-doc pages are closed. Search engines AND AI answer engines
# (ChatGPT / OAI-SearchBot, PerplexityBot, ClaudeBot, GPTBot, Google-Extended)
# are welcomed under this single `*` group, so they can index the marketing
# pages for AEO while still respecting the /d/ (user docs) and app-route
# disallows below.
#
# We deliberately do NOT add per-bot `Allow: /` groups. A crawler that finds a
# group matching its own name ignores `*` entirely, which would (a) let that bot
# crawl the private paths and (b) collide with the managed block Cloudflare can
# prepend, producing contradictory duplicate groups.
#
# NOTE: Cloudflare's "AI Crawl Control" / managed robots.txt may prepend
# conflicting `Disallow: /` rules for GPTBot/ClaudeBot/Google-Extended ahead of
# this file. To let AI / answer engines in, allow them there (and set the
# content signal ai-input=yes); this file alone cannot override that block.
User-agent: *
Allow: /$
Allow: /upgrade
Allow: /terms-of-service
Allow: /privacy-policy
Allow: /cookies
Disallow: /d/
Disallow: /f/
Disallow: /home
Disallow: /profile
Disallow: /device
Disallow: /reset-password

Sitemap: https://usecomposer.md/sitemap.xml