Home / What Steeplejack checks
AI search readiness check
Can AI search tools and assistants reach a site at all? Steeplejack checks robots.txt rules for their crawlers, whether the server actually lets them in, and the signals that help them read a page.
Sites often block AI crawlers without anyone deciding to: a firewall rule, a CDN setting or a robots.txt copied from elsewhere. Blocking training crawlers can be a perfectly good choice, so Steeplejack treats search crawlers and training crawlers separately, and reports a deliberate block as a note rather than a fault.
These checks tell you whether a site can be read by AI tools. They don’t measure whether those tools recommend it, and nothing can promise that they will.
What the free survey checks
AI search and assistant crawlers allowed in robots.txt
AI assistants and AI search (ChatGPT search, Claude, Perplexity) can only cite pages their crawlers may fetch. Blocking their search and user agents keeps you out of AI answers.
AI training and AI search crawlers treated deliberately
AI crawlers come in two kinds: training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) and search/assistant crawlers that fetch pages to answer questions. Treating them separately lets you opt out of training while staying visible in AI answers.
AI crawlers can actually fetch the page
Firewalls and CDN bot protection can refuse AI crawlers even when robots.txt allows them, so the site never appears in AI answers. (The scan sends each crawler's user agent from its own IP, so a CDN that verifies bot IPs may treat the real crawlers differently.)
Content available without JavaScript
Most AI crawlers don't run JavaScript. Content that only appears after scripts run is invisible to them, and slower for search engines to index.
Not yet run on the hosted survey, which has no browser to compare the rendered page with.
Schema markup (structured data) present and valid
Schema markup (structured data, usually written as a JSON-LD script block using the schema.org vocabulary) tells search engines and AI tools facts about your organisation and each page, such as your name, logo, address, articles or products, so they don't have to guess. It's what powers rich results like star ratings, breadcrumbs and FAQs in Google.
llms.txt present
llms.txt is a proposed file pointing AI tools to your key content. Few sites have one and major AI crawlers rarely read it yet, so it's a low-priority extra.
Planned for Pro and Expert
These levels aren’t available yet. See the plans.
Pro
- Page uses semantic HTML
- Content is well structured and readable