Why Can't ChatGPT Read My Website? How to Diagnose an AI Crawler Block

AI & Automation · 7 min read · November 8, 2026

By Heath Daniel, Founder & AI Platform Architect, HD Connex · Published: November 8, 2026

Direct answer: Usually one of three things: your host or firewall returns an error to the AI crawler while serving browsers normally, your pages build their content in JavaScript that the crawler never runs, or the HTML loads but contains no plain answer to the question being asked. Each has a different fix and a different owner.

If an AI assistant recommended a competitor and never mentioned you, the useful question is not "how do I rank in ChatGPT." It is narrower: can the thing read your site at all? When we measured Volusia County business websites the way AI crawlers actually fetch them, the sites that came back with nothing usable fell into a small number of causes, and most of the owners had no idea anything was wrong. Their site loaded perfectly on their phone.

There are three causes worth checking, in this order, because each makes the next one pointless until it is fixed.

Cause One: Your Host Is Returning an Error to the Crawler

An AI crawler does not browse your site the way you do. It makes one plain request, identifies itself with a bot user-agent such as GPTBot or ClaudeBot, does not run JavaScript, and gives up quickly if something goes wrong. That single request is enough to trip a lot of modern hosting.

The usual culprits are a bot-fighting setting at the CDN, a firewall rule that treats unfamiliar user-agents as hostile, or a rate limiter that answers an unknown client with "too many requests" on its first visit. None of those are decisions anyone made about AI. They are defaults, and they are invisible to you, because you arrive in a browser from a residential connection and sail straight through.

The tell is a mismatch in the HTTP status code: a browser request gets a 200 and a bot request on the same URL gets a 403, 404, 429, or 5xx. That is a hosting problem, not a content problem. No amount of rewriting your services page will change it.

Cause Two: Your Page Is Built in JavaScript the Crawler Never Runs

The second group of failures returns a perfectly healthy 200 that happens to be nearly empty. The HTML shell arrives, the browser runs the scripts, the scripts assemble the headings and paragraphs, and a human sees a finished page. A crawler that does not execute JavaScript sees the shell.

This is common on template platforms, page builders, and single-page applications never configured to pre-render. It is also the failure mode people are least likely to suspect, because every tool they check their own site with runs JavaScript.

There is rarely a middle ground here. Either your words are in the HTML that comes back on the first request, or almost none of them are.

Cause Three: The HTML Is Readable, but It Does Not Answer Anything

The third case looks fine on every technical check. The crawler fetches the page, gets plenty of text, and still has nothing to quote, because the page never states anything a person asked.

Assistants answer questions. Someone types what a drainage fix typically runs, whether anyone nearby handles after-hours calls, or which neighborhoods a firm covers. If your pages speak only in adjectives — trusted, premium, results-driven — there is no sentence to lift. Pages that get quoted tend to name the service area in words, carry a phone number in the HTML rather than inside an image or a script-injected widget, give price ranges or at least how pricing is structured, and answer real questions in question-and-answer form.

Structured data belongs in this layer too. LocalBusiness markup tells a machine what kind of business you are and where, and question-and-answer markup hands it an extractable pair. In the Volusia County sites we measured, question-and-answer markup was the widest open gap in the county: nearly none of those businesses had any.

How to Tell Which One You Have, in Two Minutes

You need two requests to the same URL, and from each one you need two things: the HTTP status code and the body. A word count alone cannot tell a block apart from an empty shell, because an error page is also short.

1. Fetch your homepage as a bot. From a terminal: curl -sL -A "GPTBot/1.0" -o bot.html -w "%{http_code}\n" yoursite.com. That prints the status code and saves what came back. Use your full address including the scheme if your site does not redirect plain http to the secure version, or you will be testing an origin nobody visits.
2. Fetch it again as a browser, using a complete desktop user-agent rather than a short one, because a bare stub gets challenged by some of the same rules you are trying to measure: curl -sL -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" -o browser.html -w "%{http_code}\n" yoursite.com.
3. Compare the two status codes first. Browser 200 and bot anything else means Cause One, at your host. Two 200s means your plumbing answered, and the question moves to what it answered with. If both come back non-200, nothing is diagnosable yet — load the page in an actual browser, and if it works there, your host is rejecting this whole class of request rather than AI specifically. That is still a host conversation, just a broader one.
4. If both returned 200, look at the bodies: wc -w bot.html browser.html. Treat the numbers as a hint, not a verdict — a raw word count includes tags, inline scripts, and embedded JSON, so a framework shell with a large data blob can score higher than it deserves. Open bot.html and look for your own sentences. If your headings and service copy are missing, that is Cause Two — unless what you find instead is a security-check or "verifying you are human" page, which is Cause One wearing a 200. If your copy is all there and still nothing gets quoted, you are looking at Cause Three.
5. Repeat it for the two or three pages you would actually want quoted — a service page, your main service-area page, your contact page. Homepages and inner pages do not always behave the same way, especially on platforms that cache them differently.

If a page looks right on your screen and wrong in that bot request, trust the bot request. That is the version the assistant sees.

What to Ask Your Host, and How to Know It Worked

Cause One is a conversation rather than a project, but it goes badly if you open with "please allow AI." Be specific: ask them to confirm whether bot management, a WAF rule, or rate limiting is challenging or rejecting requests from GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and CCBot, and ask them to allow those user-agents at the edge rather than only in robots.txt. Robots.txt is a request a well-behaved crawler reads; a firewall is a door. Fixing the file while the door stays shut changes nothing.

One name belongs in a different sentence. Google-Extended is not a crawler and has no user-agent of its own — Google's documentation describes it as a robots.txt control token that governs whether content Google has already crawled may be used in Gemini and Vertex AI grounding and training. There is nothing for a firewall to allow. It lives in your robots.txt or nowhere.

Then re-run step 1. A fixed block shows up as a 200 where you had an error, usually within minutes of the rule change. That is the whole verification, and worth doing yourself rather than taking "we've allowed it" on trust.

Cause Two is a platform setting or a build change. Most frameworks and site builders have a pre-render, static-export, or server-rendering option, and turning it on is the whole fix. Where the platform has no such option, the honest answer is that the site needs rebuilding on something that ships HTML.

Cause Three is writing and markup, and you can start it this week without touching your stack. Take the questions you answer on the phone every day and answer them on the page in the words customers use. Our county study sets out the rest of that checklist in more detail.

A Note on llms.txt

Adding an llms.txt file to summarize your business for AI systems is a reasonable finishing touch, and in our county measurement a surprising number of local sites already publish one. At least one publishes it on a site that returns nothing without JavaScript — a polite note left for a visitor who already gave up at the door. Fix the fetch first, then the content, then leave the note.

Why This Matters More in a Local Market

An assistant asked about a national brand can describe it without reading its site, because the whole web has written about it. A roofer in Port Orange or a brokerage in Ormond Beach has a thinner trail. There is usually some of it — a business profile, review sites, directory listings — and an assistant will work from those happily enough. The problem is what lives only on your own pages: your real service area, your hours, what you charge, which jobs you take. A directory entry can confirm you exist. It cannot say any of that. When the crawler cannot read your site, those details are simply absent from the pool the answer gets built from.

Getting It Checked

If you want this measured rather than guessed, HD Connex is a Daytona Beach agency that builds websites, local SEO, and AI agents for Volusia County businesses, and the county measurement above is our own work. Every engagement starts with a free consult and a fixed quote before any build, so nothing is committed while you work out which of the three causes you have. Call (386) 318-2214 or book a consult, and bring the two status codes if you have already run them — it shortens the conversation considerably.

Topics: AI Search, GEO, GPTBot, Volusia County, Technical SEO

Related: Volusia County AI Visibility Study · Agentic Websites · Local SEO for Small Business · Daytona Beach Web Design · Book a Free Consult · Industry solutions (Daytona Beach) · Website design packages

← Back to blog · Industry solutions