Published November 13, 2026

What is Common Crawl, and does it affect whether AI knows my business?

Common Crawl is a nonprofit that copies a large sample of the public web every month and gives the archive away for free. Many AI models were trained on filtered versions of that archive, so if your website has never been in it, those models had less chance to learn about your business before anyone asked them a question. It matters most for what an AI "already knows." Live AI search runs on separate crawlers, so being in Common Crawl helps, but it doesn't guarantee anything.

What this means

Common Crawl describes itself as a 501(c)(3) nonprofit that provides a copy of the internet to researchers, companies, and individuals at no cost (Common Crawl, n.d.). Its crawler is called CCBot. It checks a site's robots.txt file first, then fetches the pages it's allowed to fetch. It doesn't run JavaScript, and it doesn't use cookies.

A Mozilla Foundation study found that Common Crawl became one of the most important sources of training data for generative AI after 2020. It also found that developers don't train on the raw archive. They train on filtered versions of it (Baack & Mozilla Insights, 2024). So your website reaches an AI model in stages: CCBot crawls your page, the page lands in the archive, it survives a developer's filters, and it ends up in that developer's training set. Your page can drop out at any of those stages.

A crawler's link-graph visualization of etsy.com, with a central node branching out to clusters of linked pages shown as gray, orange, and blue circles

Photo: "Site Graph of Etsy.com" by Noah Sussman, CC BY 2.0

Who this applies to

This applies to any business that wants AI assistants to describe it accurately without having to search the web first. That includes local practices, contractors, law firms, and service businesses whose customers ask ChatGPT or Gemini open-ended questions. It matters most for businesses whose sites were built on platforms or CDNs that may block AI crawlers by default. Many owners on those platforms have never seen the setting.

How we'd evaluate it

In June 2026, Common Crawl published a free field guide called The AI Visibility Audit. It lays out a five-check process that takes about 90 minutes with free tools, and it warns that CDN and firewall defaults can silently block AI crawlers (Burns, 2026). When we check a site, we look at four things:

Available options

Benefits, limitations, and tradeoffs

The upside is reach. Mohanadasan (2026) puts it this way: a GPTBot visit serves one company, while a CCBot visit ends up with everyone who later trains on the open web. The tradeoff is control. Your public pages can be copied and reused by anyone who downloads the archive. Some major publishers have decided that cost is too high. Mozilla notes that the New York Times and Facebook block Common Crawl from most or all of their pages (Baack & Mozilla Insights, 2024). Neither choice is wrong. The real problem is a block nobody chose on purpose.

There's also a hard limit. Being in Common Crawl doesn't put you in any particular model. Training sets are filtered, and most frontier labs stopped disclosing their training mixes around 2023 (Mohanadasan, 2026).

What we know

Next steps

Start by opening your robots.txt file (yourdomain.com/robots.txt) and searching it for "CCBot." Next, check your CDN or website builder for a setting that blocks AI bots or AI crawlers, and decide on purpose whether to leave it on. Then make sure your sitemap is listed in robots.txt and that your key service pages show their text without needing JavaScript. Crawls run monthly, so any fix you make today won't show up until a later crawl (Mohanadasan, 2026).

Orlando considerations

Many Central Florida small businesses run their sites on hosted builders or behind a CDN that a web vendor set up years ago, so a default AI-crawler block is easy to miss. Bilingual businesses should also note Common Crawl's English skew (Baack & Mozilla Insights, 2024; Burns, 2026). Spanish-language pages that are well linked and readable without JavaScript give crawlers the best chance to find them.

Frequently asked questions

Is Common Crawl the same thing as ChatGPT search or Google AI Overviews?

No. Common Crawl is a free archive of web pages that AI developers filter and use as training data. Live AI search features use their own crawlers and indexes. OpenAI, for example, uses OAI-SearchBot for ChatGPT search and GPTBot for training, and controls them separately.

Should I block CCBot?

That's a business decision. Blocking keeps your future pages out of an archive that many AI training sets start from. Allowing it means anyone who downloads the data may copy and reuse your public pages. Whichever you choose, make sure it's a deliberate choice and not a platform default nobody noticed.

If my site is in Common Crawl, will AI models know about my business?

Not necessarily. AI builders filter Common Crawl before training, most frontier labs no longer disclose their training mixes, and Common Crawl itself is a sample of the web, not a full copy. Being in the archive makes inclusion possible, not certain.

How long does a fix take to show up?

Common Crawl publishes new crawls about once a month, so a fix made today can only appear in a future crawl. Whether it then reaches a given AI model depends on when that model's developer next trains on newer data, and that timing is up to the developer.

References

Baack, S., & Mozilla Insights. (2024, February 6). Training data for the price of a sandwich: Common Crawl's impact on generative AI. Mozilla Foundation. https://www.mozillafoundation.org/en/research/library/generative-ai-training-data/common-crawl/

Burns, S. (2026, June 1). Introducing the AI Visibility Audit. Common Crawl. https://commoncrawl.org/blog/introducing-the-ai-visibility-audit

Common Crawl. (n.d.). Frequently asked questions. Common Crawl. Retrieved November 13, 2026, from https://commoncrawl.org/faq

Mohanadasan, S. (2026, August 10). Common Crawl published a manual for being visible to AI, I automated it. Search Engine Journal. https://www.searchenginejournal.com/common-crawl-published-a-manual-for-being-visible-to-ai-i-automated-it/584466/

OpenAI. (n.d.). Overview of OpenAI crawlers. OpenAI. Retrieved November 13, 2026, from https://developers.openai.com/api/docs/bots

This article is for general informational purposes and isn't a guarantee of placement or performance in any AI system. Common Crawl, ChatGPT, Gemini, Claude, Perplexity, Google AI, and similar systems are operated by third parties outside AnswerFoundry's control, and their behavior changes without notice. Results vary by business, market, and competition.

Last updated: November 13, 2026

Not sure whether your site is open to the crawlers that feed AI, or blocked by a setting nobody remembers turning on? That's one of the first things an AI visibility audit checks.

Find out what AI says about your business.

The audit takes two weeks. The blind spot lasts as long as you let it.

Get Your Free Answer Snapshot