# InsuranceMonster - robots.txt # We WANT AI assistants to read and cite us, so nothing is blocked from # crawling except the two form endpoints. # # THERE IS DELIBERATELY ONLY ONE User-agent GROUP. This file used to carry a # "*" group that disallowed /md/ and /llms-full.txt, plus eight named groups # that allowed them back in. That design was wrong twice over: # # 1. A named group is INDEPENDENT - an agent that matches its own group # ignores the "*" rules entirely and inherits nothing. So the file's # access policy was "blocked, unless your product name appears in this # list", and every agent not on the list - including every agent that # does not exist yet - got the blocked path. Verified 2026-08-17: an # ordinary fetch of /md/california-auto-insurance and /llms-full.txt came # back robots-disallowed while the HTML pages fetched normally, i.e. the # two URLs llms.txt advertises as the cheap way to read us were the two a # model could not reach. # # 2. The list was already stale. Naming an agent is a maintenance promise we # will not keep. Crawler names split and get renamed constantly - the # training crawler and the live user-initiated fetcher are separate names # for every major vendor, and the old list had one of each pair at best. # # So: allow everyone to CRAWL, and stop duplicate content at the INDEX layer # instead, which is the layer it was always an issue on. See the /md/ note # below. Do not reintroduce a named group to grant access - "*" already grants # it. A named group is only ever needed to take access AWAY from one agent. # # HOST NOTE: apex (no www) is canonical - it matches every , # the sitemap, and the schema. Do NOT change the Sitemap line to www: that host # does not serve this site. # THE /md/ CORPUS: every page is also published as plain Markdown at # /md/, advertised in llms.txt, and all of it is bundled into # /llms-full.txt. It is the SAME CONTENT as the HTML page, and the HTML page is # the one we want ranked. # # That duplication is handled by "X-Robots-Tag: noindex" response headers on # /md/* and /llms-full.txt (set in web.config), NOT by blocking the crawl: # - robots.txt governs CRAWLING. Blocking the crawl stopped models from # reading the corpus, which is the entire reason it is published. # - X-Robots-Tag governs INDEXING. noindex keeps the plain-text twin out of # every search index, so it can never split ranking signals with the HTML # page, while leaving it perfectly readable to anything that fetches it. # It is noindex and not "noindex, nofollow" on purpose: each .md file opens # with a "> Source: " line and links back to the HTML page, and we # want those links followed. # # The header is the load-bearing half of this. If web.config's # blocks for md and llms-full.txt are ever reverted (they are the sections most # suspected in the 500.19 history - see the notes in that file), the corpus # becomes indexable duplicate content. Restore the header rather than # re-blocking the crawl here. User-agent: * Allow: / Disallow: /lead.asp Disallow: /thank-you.html # TO OPT OUT OF MODEL TRAINING while keeping answer-engine citation # eligibility intact, append the block below. Being in the training corpus is # how a brand gets recalled without a live search, so allowing it is a business # decision we have made deliberately - but it is reversible, and this is how: # # User-agent: GPTBot # User-agent: ClaudeBot # User-agent: Google-Extended # User-agent: Applebot-Extended # Disallow: / # # Those four are training/dataset agents only. Blocking them has NO effect on # whether ChatGPT, Claude, Gemini, or Siri can find and cite us live - that is # handled by an entirely different set of agents (OAI-SearchBot and # Claude-SearchBot index for citation; ChatGPT-User, Claude-User, and # Perplexity-User fetch a page live when a prompt needs it; Googlebot and # Bingbot feed AI Overviews and Copilot from the ordinary search index). Do not # add any of THOSE to a Disallow block: that is what makes us ineligible to be # cited at all. # # robots.txt changes take roughly 24 hours to propagate through OpenAI's systems. Sitemap: https://insurancemonster.com/sitemap.xml # Structured summary of this site for LLMs (llmstxt.org format). Also linked # from the footer of every page so a crawler finds it by following links, not # just by guessing the well-known path. llms.txt itself is indexable and in the # sitemap - it is curated commentary, not a duplicate of any page. # https://insurancemonster.com/llms.txt