# Commercial Garage Doorman | www.commercialgaragedoorman.ca
#
# Public content site. Nothing here is disallowed to anyone. Every AI crawler is named one by
# one so the permission is explicit rather than inherited, and because robots.txt resolves the
# most specific matching group: a named Allow survives anything that later adds a Disallow to *.
#
# Each AI company splits its crawlers across three jobs. All three are answered yes on purpose:
#   may this content train models?                 yes
#   may an AI answer engine surface and cite it?   yes
#   may a live user's question fetch it?           yes
#
# Every token below was read from that vendor's own documentation on 2026-09-10, cited inline.
# Crawlers with no published token (ByteDance's Bytespider, xAI's Grok fetcher, DeepSeek) are
# deliberately absent: a bot that documents no user-agent cannot be addressed in this file, and
# inventing a name for it would be guesswork.

User-agent: *
# Cloudflare Content Signals (contentsignals.org). Advisory, not enforcement: robots.txt states
# a preference and cannot compel anyone. It declares what may happen to the content AFTER a
# crawler is let in. Placed inside this group because that is where Cloudflare's own example
# puts it; at the foot of the file it would bind to whichever group came last instead.
#   search    = building a search index, returning links and short excerpts
#   ai-input  = feeding a live model, which is what a cited AI answer actually is
#   ai-train  = training or fine-tuning models
#   use       immediate (reuse nothing) | reference (index, excerpt, LINK BACK) | full
#             (summarize and reproduce). reference is the level that asks for the citation,
#             which is the point; full permits reproduction without a link back.
Content-Signal: search=yes, ai-input=yes, ai-train=yes, use=reference
Allow: /

# ---- OpenAI ---------------- developers.openai.com/api/docs/bots
# OAI-SearchBot is the token that decides ChatGPT search visibility, not GPTBot. OpenAI: "Sites
# that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers."
# ChatGPT-User is user-triggered and OpenAI says robots.txt "may not apply" to it.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: OAI-AdsBot
User-agent: ChatGPT-User
Allow: /

# ---- Anthropic ------------- support.claude.com/en/articles/8896518
# The only vendor here that honours robots.txt on its user-triggered fetcher as well.
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

# ---- Google ---------------- developers.google.com/search/docs/crawling-indexing/google-common-crawlers
# Google-Extended governs Gemini training only. Google: "Google-Extended does not impact a
# site's inclusion in Google Search nor is it used as a ranking signal in Google Search."
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-News
User-agent: Googlebot-Video
User-agent: Google-Extended
User-agent: Google-InspectionTool
User-agent: Google-CloudVertexBot
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: Storebot-Google
Allow: /

# ---- Microsoft ------------- bing.com/webmasters/help/webmaster-guidelines-30fba23a
# Bing matters twice: its own results, and ChatGPT's web search runs on it. No Bing index
# means no ChatGPT citation, whatever the OpenAI tokens above say.
User-agent: bingbot
User-agent: BingPreview
User-agent: msnbot
User-agent: msnbot-media
Allow: /

# ---- Perplexity ------------ docs.perplexity.ai/guides/bots
# Perplexity states Perplexity-User "generally ignores robots.txt rules" because a user asked
# for the fetch. Named anyway so the intent is on the record either way.
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# ---- Meta ------------------ developers.facebook.com/docs/sharing/webmasters/web-crawlers/
# facebookexternalhit is the link-preview fetcher: it is what renders the share card when a
# page is posted to Facebook or Instagram, and this company posts its own jobs to both.
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: meta-externalads
User-agent: meta-externalfetcher
User-agent: facebookexternalhit
Allow: /

# ---- Apple ----------------- support.apple.com/en-us/119829
# Applebot-Extended controls Apple Intelligence training only and does not crawl. Apple:
# pages that disallow it "can still be included in search results".
User-agent: Applebot
User-agent: Applebot-Extended
Allow: /

# ---- Mistral --------------- docs.mistral.ai/robots
User-agent: MistralAI-Index
User-agent: MistralAI-Training
User-agent: MistralAI-User
Allow: /

# ---- Amazon ---------------- developer.amazon.com/amazonbot
# Also an IndexNow participant, so a deploy ping reaches it directly.
User-agent: Amazonbot
Allow: /

# ---- Common Crawl ---------- commoncrawl.org/ccbot
# The open corpus much of the open-model ecosystem is built from. Blocking it has a wider
# effect than blocking any single company's crawler.
User-agent: CCBot
Allow: /

# ---- Other search engines --
# Yandex, Seznam, Naver and Yep are the remaining IndexNow participants; DuckDuckGo leans on
# Bing's index but runs its own fetcher.
User-agent: YandexBot
User-agent: SeznamBot
User-agent: Yeti
User-agent: DuckDuckBot
Allow: /

Sitemap: https://www.commercialgaragedoorman.ca/sitemap.xml