To better understand content, LLMs use mechanisms similar to traditional search engines, but they also come with their own specific characteristics.

In this article, I’ll walk you through the steps required to make your website more visible to LLMs and, of course, to increase your chances of being mentioned or cited by them.

Allow AI Crawlers to Access Your Content

Just like traditional search engines (Google, Bing, etc.), LLMs need access to your website in order to crawl it and even consider it in the first place.

The following bots are important, each serving a specific role in accessing and processing content correctly:

• OAI-SearchBot – An OpenAI bot used for indexing and search.
  
• GPTBot – Another OpenAI bot, used for collecting public data to train language models.
  
• ChatGPT-User – Identifies traffic coming from real ChatGPT users.
  
• PerplexityBot – A bot developed by Perplexity.ai that retrieves information to answer user questions in real time.
  
• Perplexity-User – The user-agent used when a Perplexity user opens a link from an AI-generated answer.
  
• ClaudeBot – The user-agent used by Anthropic for automated crawling and site indexing for Claude.
  
• Claude-User – The user-agent triggered when a Claude user opens a link from an AI-generated answer.
  
• Claude-SearchBot – Used by Claude to search for and evaluate web pages as potential sources before citing them.
  

By allowing access to all these bots, you significantly increase your real chances of being discovered and cited.

P.S. Not all of the bots listed above directly influence whether your site appears in AI citations (for example, GPTBot is theoretically used only for training purposes). That said, I wouldn’t bet my house on the idea that if OpenAI doesn’t use your site for training, it couldn’t indirectly affect your perceived credibility or “score” when it comes to citations.

Add Relevant Schema Markup to Your Templates

Structured markup acts as a native language for LLMs and AI Overviews, enabling fast, accurate, and structured interpretation of on-page data.

LLMs rely heavily on understanding your site’s structure, what your website is about, how it’s organized, and how individual sections relate to one another. They aim to narrow down as precisely as possible to the most relevant pages and even specific sections within those pages, so they can return the exact piece of information your site provides to the user asking a question.

To support this, you need to make things easier for the LLM, which means simplifying the structure of each page using schema.org markup.

Depending on the type of page, you should implement the appropriate structured data:

Examples:

• Blog article – https://schema.org/Article
  
• Service page – https://schema.org/Service
  
• Product page – https://schema.org/Product
  
• Pages with a clear FAQ section – https://schema.org/FAQPage

etc.

P.S. You don’t need to fill out every single schema property, that’s a quick way to overcomplicate things and potentially break database connections. The best approach is to keep it simple.

For example, for the https://schema.org/Service schema, you can safely use just:

• name
  
• serviceType
  
• description
  

The schema will still be valid and correctly understood by LLMs, which is the ultimate goal.

Structure Your Headings for Maximum Clarity

Well-structured headings help AI understand hierarchy and context.

Just like in traditional SEO, heading structure is important, arguably even more so here. As mentioned earlier, LLMs rely heavily on structural clarity. A correct heading hierarchy for each page type should follow a clean structure like:

<h1>Main Heading</h1>
  <h2>Subheading</h2>
    <h3>Sub-subheading</h3>
  <h2>Subheading</h2>
    <h3>Sub-subheading</h3>
    <h3>Sub-subheading</h3>

Prioritize Speed That AI Actually Cares About

First, it’s important to note that classic speed metrics aren’t interpreted by LLMs in the same way they are by traditional search engines.

For example, metrics like LCP (Largest Contentful Paint) or other purely visual UX indicators don’t really matter for LLMs, because the user interaction happens on the AI platform, not on your website. Your site’s primary role is to be crawlable and easy for LLMs to understand.

So what does matter when auditing performance for LLMs?

TTFB (Time to First Byte) / Server Response Time

As the name suggests, this metric measures how long it takes for the server to start responding.

In short: you need a fast server.

FCP (First Contentful Paint)

This is the moment when the first visible piece of content appears (text, image, SVG).

FCP matters for LLMs because it signals that the site delivers readable content quickly; a sign that the information is accessible, stable, and worthy of being read and cited.

Make Sure Key Content is in the Raw HTML

This step checks whether key elements, such as the main title, featured image, and actual page content – are loaded directly in the HTML or injected later via JavaScript.

You need to ensure that:

• The H1 heading
  
• The main body content
  
• The first image
  

are all visible directly in the page source, without relying on JavaScript injection.

This is crucial for LLMs, as they process vast amounts of information and need fast, reliable access to content. Pure HTML is the fastest and safest way for them to extract information correctly.

JavaScript-injected content can still work, but as of writing this article, LLMs behave similarly to how search engines did about 14 years ago, strongly favoring clean HTML over JS-rendered content.

Don’t Rely on JavaScript for Core Content

Here, the page is analyzed with JavaScript enabled and disabled.

Any content that disappears when JavaScript is turned off is, in practice, invisible to LLMs.

Keep Your Brand Identity Consistent Everywhere

Just like in classic SEO, being visible and cited by LLMs requires you to be verified as a real-world entity.

That means consistent NAP information everywhere you appear online – from your Google Business Profile to your official website, partners, and press releases.

Inconsistencies or missing information lead to entity fragmentation, making it difficult for LLMs to confidently determine that all mentions refer to the same brand.

The post How to Make Your Website Visible to LLMs  appeared first on VladSand.

But one very important fact is often overlooked: the biggest issue right now for websites, when it comes to indexing and being processed by LLMs (Large Language Models), is not about optimization. The real problem is that LLMs simply cannot “see” most websites.

Why AI Crawlers Can’t Access Many Websites

This happens because, although we in the SEO world are keeping up with trends, in this case with AI and LLMs, some web hosting providers have unfortunately remained stuck in the “Middle Ages” when it comes to AI.

Even at the time of writing this article, they are still blocking AI bots in the same way they block aggressive, invasive bots such as MJ12bot, SemrushBot, LinkpadBot (the typical bots used by SEO and online marketing tools).

How to Check if Your Server is Blocking LLM and AI Bots

First, we need to identify which LLM bots are relevant at the moment.

Among the most relevant bots currently are:

• OpenAI bots (GPTBot, ChatGPT-User & OAI-SearchBot)
  
• Perplexity (PerplexityBot, Perplexity-User)
  
• Anthropic (ClaudeBot, Claude-User, Claude-SearchBot)
  

Practical Methods to Test Whether Bots Are Being Blocked

Method 1: Using Screaming Frog to Test AI Bot Access

This is the simplest method. You just need a Screaming Frog license, and in Configuration > User Agent, you can select the bot you want to check to see whether your site (or server) is blocking it:

If you receive a 406 error message, it means the server is blocking the ChatGPT bot from crawling your site.

If the bot starts crawling your site and returns status 200 for the pages it visits, that means the server is not blocking GPTBot.

Here I’ll make a quick note regarding what OpenAI states on its official site: their three main bots (GPTBot, OAI-SearchBot, and ChatGPT-User) each have different roles.

Specifically:

• GPTBot – is used for training AI models (OpenAI states it does not affect appearance in search results).
  
• OAI-SearchBot – is used for indexing for ChatGPT Search. This is theoretically the most important if you want your site to be crawled, processed, and shown as a source in ChatGPT.
  
• ChatGPT-User – this bot needs actual access when users click on links or perform live requests, such as “check if there are blue balloons on website.com”. If this bot is blocked, GPT won’t be able to visit the page.
  

As for the other bots, Claude and Perplexity, you can check them in Screaming Frog in the same way as OpenAI’s bots.

Method 2: Testing AI Bots with curl (HEAD Request)

This method is more direct and doesn’t require any special software. Open either the Mac Terminal or Windows Command Prompt and run the following header checks for each bot:

OpenAI – GPTBot

curl -I --ssl-no-revoke -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot" https://example.com

OAI-SearchBot

curl -I --ssl-no-revoke -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" https://example.com

ChatGPT-User

curl -I --ssl-no-revoke -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" https://example.com

PerplexityBot

curl -I --ssl-no-revoke -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://www.perplexity.ai/bot)" https://example.com

Perplexity-User

curl -I --ssl-no-revoke -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user" https://example.com

ClaudeBot

curl -I --ssl-no-revoke -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +https://www.anthropic.com/claude)" https://example.com

Claude-User

curl -I --ssl-no-revoke -A "Mozilla/5.0 (compatible; Claude-User/1.0; +https://www.anthropic.com/claude)" https://example.com

Claude-SearchBot

curl -I --ssl-no-revoke -A "Mozilla/5.0 (compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude)" https://example.com

To understand what this curl command does and where this information comes from: there’s no magic or hack involved.

The major companies behind these bots (OpenAI, Anthropic, Perplexity, etc.) publicly provide all the information about their crawlers, including their full user-agent strings, exactly as they appear in the HTTP headers of real requests.

If you’re wondering about the “revoke” part, the –ssl-no-revoke flag is included because some Windows versions throw SSL errors when they cannot check the certificate’s revocation status.

Case Study: Real Bot Access Test on VladSand.com

OAI-SearchBot gets through (status 200):

As you can see, the GPT crawler that indexes content has access and returns a 200 status.

ChatGPT-User (status 406) – blocked, unfortunately, in my case:

ChatGPT-User is blocked by the server, by the hosting company (I’m currently in a battle with them to explain why it needs to be unblocked), but hopefully they will eventually fix this.

How to Unblock AI and LLM Bots on Your Website

If you’re using shared hosting (like most websites), the only way to unblock these bots is to contact your hosting provider and ask them to whitelist the relevant User-Agents or IP ranges. Fortunately, this is usually a quick fix once they understand the issue.

Most major AI companies also publish their official IP ranges, which can be useful if your firewall or hosting provider uses IP-based filtering. Here are the official resources:

GPTBot: https://openai.com/gptbot.json

OAI-SearchBot: https://openai.com/searchbot.json

ChatGPT-User: https://openai.com/chatgpt-user.json

PerplexityBot: https://www.perplexity.ai/perplexitybot.json

Perplexity-User: https://www.perplexity.ai/perplexity-user.json

*Anthropic (Claude) has not provided official details about the IP ranges it uses.

If you’re on a dedicated server or VPS, you can manually adjust the server configuration, for example, by adding specific ModSecurity rules or firewall exceptions to explicitly allow AI crawlers.

Other Common Reasons AI Bots Are Blocked

Here I focused on blocking at the server level, because in most cases this is where access for these bots is cut off. However, they can also be blocked in other ways, sometimes without you even realizing it:

• through the robots.txt file
  
• through external firewalls (Cloudflare, Wordfence, etc.)
  
• through security plugins
  
• or implicitly through IP blocking (if your hosting provider blocks certain countries, the IPs of OpenAI / Perplexity / Anthropic may be rejected even if the User-Agent is valid)

The post Is Your Server Blocking AI Bots and LLM Crawlers? appeared first on VladSand.