First accessible, then readable. Then citable.

Many people talk about content for AI systems – hardly anyone talks about the technical basis. The best content is no use if AI crawlers cannot reach, render, or process it.

Many people talk about content for AI systems – but hardly anyone talks about the technical basis. However, the following applies: the best content is useless if AI crawlers cannot reach, render, or process it.

This article goes through the technical requirements step by step – with concrete checks you can implement directly.

AI providers use their own crawlers with their own names. The most important ones are: GPTBot and OAI-SearchBot (OpenAI/ChatGPT), ClaudeBot (Anthropic/Claude), PerplexityBot (Perplexity) and Google-Extended – the latter controls usage for Google's AI models, not normal search.

Access is controlled via your website's robots.txt file. The most common error is not intentional blocking, but unintentional: security plugins, firewalls, or blanket robots.txt rules block AI crawlers without anyone noticing.

Test step: Open your-company.io/robots.txt and search for crawler names and general disallow rules. Additionally, check server logs or the CDN dashboard to see if requests from these bots are being blocked.

AI crawlers primarily process the delivered HTML text. And therein lies a problem that affects many modern websites: the major AI crawlers do not execute JavaScript. Vercel and MERJ found this in their investigation „The Rise of the AI Crawler" (December 2024) analysed crawler traffic within the Vercel network, including 569 million requests from OpenAI's GPTBot and 370 million from Anthropic's ClaudeBot within a month. The finding: these bots request JavaScript files, but do not execute them, and therefore do not read client-side rendered content. Google's Gemini is an exception because it uses Googlebot's rendering infrastructure. Content that is only loaded in the browser later is invisible to the other systems – even if visitors see it perfectly. The status is December 2024, and the field is moving fast: which bots render may change.

Also critical are content that is only in images or graphics, exclusively found in PDF brochures, or hidden behind logins, forms, and cookie walls.

Test step: Call up your most important pages with JavaScript disabled or look at the page source code. Anything not written as text there doesn't exist for many AI crawlers.

Schema markup in JSON-LD format describes to machines what a page is about and how elements relate to each other. Particularly relevant in B2B: Organization (company data, logo, contact), Service/Product (services), FAQPage (frequently asked questions) and Article (blog posts with date and author).

It is debated how strongly structured data directly influences citations. However, HubSpot's State of AEO 2026 found a measurable correlation: pages that combined a Q&A section with FAQ markup achieved higher citation rates on Gemini, Google AI Mode, and Perplexity. The effort is minimal, the risk zero.

Test step: Test your most important pages with Google's Rich Results Test or the Schema validator (validator.schema.org).

AI crawlers also have limited resources. Websites that load slowly or frequently throw errors are scanned less often and less completely. The basics: load time under two to three seconds, clean internal linking, an up-to-date XML sitemap referenced in robots.txt, and no 404 errors or redirect chains on central pages.

Furthermore, there's the consistency of your company data online. AI systems compare information from various sources to recognise entities: is the company on the website the same as on the LinkedIn profile and in the business directory? Contradictory details weaken this trust – and therefore the likelihood of being used as a source.

Test step: Compares company name, service description, and contact details on the website, LinkedIn, Google Business Profile, and the top two to three industry directories. Unifies discrepancies, removes outdated entries.

A mechanical engineering company was wondering why it wasn't appearing in any AI responses, despite having a website with good content. A technical check revealed three blockers: a security plugin was blocking all AI crawlers at the firewall level – there was nothing about this in the robots.txt. The key service descriptions were exclusively in PDF brochures. And the company data contradicted the entries in two industry portals.

After the cleanup – firewall exceptions for the main AI crawlers, performance content as regular HTML pages with FAQ markup, unified directory entries – the content was indexed by Perplexity and ChatGPT within a few weeks. In the monthly prompt test, the company appeared in provider lists for the first time.

AI readiness doesn't start with content, but with technology. The good news: the check takes an afternoon, most fixes can be implemented quickly – and without this foundation, any content optimisation will go to waste. First accessible, then readable, then citable. In that order.

Read on This is how your company will be recommended by ChatGPT – the GEO guide

Reflects the real prompts from ChatGPT, Perplexity & Google AI around technical AI readiness.