SteeplejackBot
If you’ve seen SteeplejackBot in your logs, someone asked Steeplejack to survey your website: usually a web agency or freelancer checking a site they look after.
- What it requests
- Your homepage,
/robots.txt,/sitemap.xml,/llms.txtand/.well-known/security.txt; thehttp://and www (or non-www) versions of your address, to see where they redirect; up to three sitemaps listed by your sitemap index or robots.txt, and a sample of about ten pages from them (skipping anything robots.txt disallows for SteeplejackBot); up to 15 links from your homepage (skipping anything robots.txt disallows, and anything like a log-out or basket link); one made-up address, to see that it returns a 404; your social share image, favicon and touch icon; up to ten of the largest CSS and JavaScript files your homepage loads, to see how they’re compressed; and one copy of the homepage under each major AI crawler’s user agent to see whether your firewall lets them in. It also reads your public DNS records (including DNSSEC, through Cloudflare’s public resolver) makes a few ordinary TLS handshakes, and asks your domain’s registry when the registration ends. - What it doesn’t do
- Submit forms, try to log in, or crawl your whole site. A free survey only requests the homepage and those small samples, and doesn’t guess at hidden files. A Pro survey adds the pages described below.
- Pro surveys
- A Pro survey also looks at up to ten of your main pages, chosen from your navigation, footer and sitemap, one at a time with a gap between requests. Each is requested by SteeplejackBot, skipping anything robots.txt disallows for it, then loaded in a browser and in Lighthouse, which identify as ordinary Chrome. Checks for exposed files or admin panels only run with the site owner’s permission.
- How often
- Only when someone starts a survey. It doesn’t revisit on its own.
- User agent
SteeplejackBot/0.1 (+https://steeplejack.mikestreety.co.uk/bot)- Where from
- While Steeplejack is in early testing, surveys run on Netlify, so requests come from Netlify’s cloud addresses rather than a fixed IP address. Fixed addresses will be published here once surveys move to their own server. Performance is measured by Google PageSpeed Insights, which loads your homepage from Google’s servers as Chrome Lighthouse.
- Opting out
- To ask for your site to be excluded, email steeplejack@mikestreety.co.uk from an address at that domain, or explain how you’re connected to the site. Once it’s on the exclusion list, new surveys of it are refused and its results are no longer shown, including to people already tracking it.
- Problems
- If SteeplejackBot is causing a problem, or you think a survey went beyond what’s described here, email steeplejack@mikestreety.co.uk with the site, the time and, if you can, the requests from your logs. Surveys of that site will be paused while it’s looked at. Security issues with Steeplejack itself go to security@mikestreety.co.uk.