But one very important fact is often overlooked: the biggest issue right now for websites, when it comes to indexing and being processed by LLMs (Large Language Models), is not about optimization. The real problem is that LLMs simply cannot “see” most websites.

Why AI Crawlers Can’t Access Many Websites

This happens because, although we in the SEO world are keeping up with trends, in this case with AI and LLMs, some web hosting providers have unfortunately remained stuck in the “Middle Ages” when it comes to AI.

Even at the time of writing this article, they are still blocking AI bots in the same way they block aggressive, invasive bots such as MJ12bot, SemrushBot, LinkpadBot (the typical bots used by SEO and online marketing tools).

How to Check if Your Server is Blocking LLM and AI Bots

First, we need to identify which LLM bots are relevant at the moment.

Among the most relevant bots currently are:

• OpenAI bots (GPTBot, ChatGPT-User & OAI-SearchBot)
  
• Perplexity (PerplexityBot, Perplexity-User)
  
• Anthropic (ClaudeBot, Claude-User, Claude-SearchBot)
  

Practical Methods to Test Whether Bots Are Being Blocked

Method 1: Using Screaming Frog to Test AI Bot Access

This is the simplest method. You just need a Screaming Frog license, and in Configuration > User Agent, you can select the bot you want to check to see whether your site (or server) is blocking it:

If you receive a 406 error message, it means the server is blocking the ChatGPT bot from crawling your site.

If the bot starts crawling your site and returns status 200 for the pages it visits, that means the server is not blocking GPTBot.

Here I’ll make a quick note regarding what OpenAI states on its official site: their three main bots (GPTBot, OAI-SearchBot, and ChatGPT-User) each have different roles.

Specifically:

• GPTBot – is used for training AI models (OpenAI states it does not affect appearance in search results).
  
• OAI-SearchBot – is used for indexing for ChatGPT Search. This is theoretically the most important if you want your site to be crawled, processed, and shown as a source in ChatGPT.
  
• ChatGPT-User – this bot needs actual access when users click on links or perform live requests, such as “check if there are blue balloons on website.com”. If this bot is blocked, GPT won’t be able to visit the page.
  

As for the other bots, Claude and Perplexity, you can check them in Screaming Frog in the same way as OpenAI’s bots.

Method 2: Testing AI Bots with curl (HEAD Request)

This method is more direct and doesn’t require any special software. Open either the Mac Terminal or Windows Command Prompt and run the following header checks for each bot:

OpenAI – GPTBot

curl -I --ssl-no-revoke -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot" https://example.com

OAI-SearchBot

curl -I --ssl-no-revoke -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" https://example.com

ChatGPT-User

curl -I --ssl-no-revoke -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" https://example.com

PerplexityBot

curl -I --ssl-no-revoke -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://www.perplexity.ai/bot)" https://example.com

Perplexity-User

curl -I --ssl-no-revoke -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user" https://example.com

ClaudeBot

curl -I --ssl-no-revoke -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +https://www.anthropic.com/claude)" https://example.com

Claude-User

curl -I --ssl-no-revoke -A "Mozilla/5.0 (compatible; Claude-User/1.0; +https://www.anthropic.com/claude)" https://example.com

Claude-SearchBot

curl -I --ssl-no-revoke -A "Mozilla/5.0 (compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude)" https://example.com

To understand what this curl command does and where this information comes from: there’s no magic or hack involved.

The major companies behind these bots (OpenAI, Anthropic, Perplexity, etc.) publicly provide all the information about their crawlers, including their full user-agent strings, exactly as they appear in the HTTP headers of real requests.

If you’re wondering about the “revoke” part, the –ssl-no-revoke flag is included because some Windows versions throw SSL errors when they cannot check the certificate’s revocation status.

Case Study: Real Bot Access Test on VladSand.com

OAI-SearchBot gets through (status 200):

As you can see, the GPT crawler that indexes content has access and returns a 200 status.

ChatGPT-User (status 406) – blocked, unfortunately, in my case:

ChatGPT-User is blocked by the server, by the hosting company (I’m currently in a battle with them to explain why it needs to be unblocked), but hopefully they will eventually fix this.

How to Unblock AI and LLM Bots on Your Website

If you’re using shared hosting (like most websites), the only way to unblock these bots is to contact your hosting provider and ask them to whitelist the relevant User-Agents or IP ranges. Fortunately, this is usually a quick fix once they understand the issue.

Most major AI companies also publish their official IP ranges, which can be useful if your firewall or hosting provider uses IP-based filtering. Here are the official resources:

GPTBot: https://openai.com/gptbot.json

OAI-SearchBot: https://openai.com/searchbot.json

ChatGPT-User: https://openai.com/chatgpt-user.json

PerplexityBot: https://www.perplexity.ai/perplexitybot.json

Perplexity-User: https://www.perplexity.ai/perplexity-user.json

*Anthropic (Claude) has not provided official details about the IP ranges it uses.

If you’re on a dedicated server or VPS, you can manually adjust the server configuration, for example, by adding specific ModSecurity rules or firewall exceptions to explicitly allow AI crawlers.

Other Common Reasons AI Bots Are Blocked

Here I focused on blocking at the server level, because in most cases this is where access for these bots is cut off. However, they can also be blocked in other ways, sometimes without you even realizing it:

• through the robots.txt file
  
• through external firewalls (Cloudflare, Wordfence, etc.)
  
• through security plugins
  
• or implicitly through IP blocking (if your hosting provider blocks certain countries, the IPs of OpenAI / Perplexity / Anthropic may be rejected even if the User-Agent is valid)

The post Is Your Server Blocking AI Bots and LLM Crawlers? appeared first on VladSand.

In today’s article, we’ll focus on filters that already exist on your website, but you’re not sure which ones to make indexable, without harming your site’s indexing and without risking to damage its SEO structure.

Why Choosing Filters to Index Is So Difficult

Choosing which filters to index can be a tricky exercise because there are multiple factors to consider when deciding which filters should be indexable.

One of the most common issues is platform limitations. Some e-commerce platforms, whether mainstream or custom, are configured in a way that makes filters either impossible to index correctly or impossible to index at all.

Common Indexing Challenges for Filters

JavaScript-Generated Filters

We’re not talking here about how URLs are generated, but about the fact that, when a filter is applied, nothing changes in the browser’s address bar. No new URL is created; everything happens through JavaScript on the page.

In this situation, filters have no SEO value and don’t contribute to organic visibility. They only partially help improve the site’s UX.

Either All Filters Are Indexed or None

This is a very common situation: the platform generates filter URLs, but it either indexes all the filters on the site (or within a specific category) or none of them.

This can harm your site more than it helps. Imagine your website has 100 categories with 30 filters each. Essentially, each category and every filter gets indexed individually.

You might think: “What’s wrong with that? I’ll have more indexed filters, more ranking opportunities, more URLs.”

The Risks of Indexing Every Filter

• You will index filters with little or no content (e.g., 1–2 products). This creates thin content across your site, which could result in tens of thousands of thin pages. This can lead to algorithmic penalties, such as those from Google Panda.
  
• You’ll end up indexing filters with no SEO value, such as price, free shipping, new arrivals, discount only, in stock, out of stock, etc. Nobody searches for these filters, so they generate extra content with zero search intent. Search engines may interpret these pages as low-value, which can also negatively affect your crawl budget (the number of pages Google is willing and able to crawl on your site within a specific time period).
  

Lack of Customization for Filter Pages

Let’s assume you have filter URLs and can choose which ones to index, but you cannot customize them. This means the filter pages cannot have their own visible titles (H1) or unique meta titles in the format “category + filter” or “filter + category.”

If you can’t do this, all your indexable filter pages will end up having the same visual title and meta title as the parent category.

Example:
Instead of having pages like:

• example.com/women-shoes/filter/red → Red Women Shoes
  
• example.com/women-shoes/filter/blue → Blue Women Shoes
  
• example.com/women-shoes/filter/black → Black Women Shoes
  

You’ll have all these URLs using the same meta title and visual title as the main category page:

example.com/women-shoes → Women Shoes

This leads to keyword cannibalization between the category and its filters, resulting in a complete SEO mess.

Indexing Multiple Filter Combinations

Another issue that can occur is the platform not offering an option to deindex combinations of two or more filters applied at the same time.

Ideally, you should have the option to index a maximum of two combined filters that make sense together.

Valid example:

• example.com/t-shirts/filter/blue/xxl/ → XXL Blue T-Shirts
  

This is a common and correct scenario for indexing two combined filters.

Invalid examples:

• Size + size: t-shirts/filter/xs/xxl/
  
• Size + color + size: t-shirts/filter/xs/blue/xxl/
  

These combinations don’t make sense, have nearly zero search intent, and can generate millions (or even hundreds of millions) of useless pages.

A site that realistically has only a few thousand pages could end up with millions of indexable, irrelevant pages, leading to a potential SEO collapse.

How to Choose the Right Filters to Index

Let’s assume all the above issues have been solved, you have a good developer who can customize everything, or your platform has all the necessary plugins and features to perfectly handle filter functionality.

For a new site, even with the entire infrastructure in place, the ideal approach is to start with all filters set to “noindex.” If these are not noindex and you don’t start from this calming fact, that’s okay, at least from this point on you’ll be able to decide which filters to index and apply noindex to the ones that shouldn’t be.

The Simple Way – Keyword Research

You can use keyword research tools such as Ahrefs, SEO Monitor, Ubersuggest, and others.

These tools are helpful when you start from a single keyword (for example, the category name) and they provide relevant keywords associated with it.

However, most of the time, these tools focus on main category keywords. They can help you:

• Create new categories
  
• Optimize existing categories
  
• Occasionally discover good filter ideas with sufficient search volume
  

Identify Target Filters on Your Platform

This is the part we’ll focus on. Suppose you have an e-commerce site with a wide variety of products. You’ve set up all necessary filters in the database (color, sizes, etc.), and now you want to identify exactly which filters should be indexed.

This method is simple and can be applied using basic tools to help you find the right filters to index.

Steps:

1. Open the Keyword Combiner tool.

2. Open the category you want to analyze. Example: “laptops.”

3. In Keyword Combiner, add both the singular and plural forms: laptop / laptops.

4. Use a tool to extract anchor texts from URLs, such as Magic Anchor & URL Grabber, Linkclump (or other).

5. On the category page, extract the filter anchors (brand, type, operating system, etc.)

and paste them into Keyword Combiner, in filter section:

Then you generate the words and all the combinations of category (singular and plural) + filter (facet):

6. You can repeat this process for multiple categories until you’ve covered them all.

This method is useful when you don’t have backend database access. If you do have access, you can directly extract category and filter names from the database and combine them in Keyword Combiner without manually going through each category.

Once you’ve generated category + filter combinations, use a keyword research tool (Ahrefs, SEO Monitor, etc.) to check the monthly search volume for each combination. This will give you a clear picture of which existing filters on your site have meaningful search demand and therefore should be indexed.

P.S.:

Even if you’ve identified the right filters and have the necessary infrastructure to index and customize them, you need to follow one key rule: if a filter URL lists only one product, that page should remain “noindex.”

Once a second product is added, the “noindex” tag should be dynamically removed, and the filter URL should become indexable.

For maximum SEO impact, it’s also recommended to add descriptive, targeted content (LSI – Latent Semantic Indexing ) to the first page of each filter, but not to its paginated pages.

The fewer filters you index, the richer the descriptive text should be, to compensate for the lack of product content.

The post How to Decide Which Filters (Facets) to Index on an E-Commerce Website appeared first on VladSand.

Questioning the real value of the Disavow File

Let’s get back to the subject, because we strayed a bit with history.

The disavow file, at least for me, has never had any clear value. I mean, it only helped me and seemed to make sense once in its entire history of use, namely around 2012, after Google Penguin came into the SEO world. 

(A small side note for those who don’t know what Google Penguin is: it’s a Google update that basically acts as a link police. Sites that use spammy backlink methods fall into its filter and get penalized, either physically with a direct message in Google Webmaster Tools, or algorithmically, without you realizing it, but you can tell something negative is happening to your site, by an instant drop in organic traffic)

A single successful case

Returning to the topic for the second time: the only time it seemed to make sense to create a disavow file was after Google Penguin was rolled out and I had a client, a gift-basket site, which until Penguin had only directory links, packages of hundreds of links bought all at once on a single anchor, etc. (basically, the kind of practices common back then).

This site was penalized (algorithmically, so without a message in GSC), and because it was post-Penguin, I said clearly the problem was there, in the area of spammy links, so I created a disavow file with hundreds of spammy sites linking to this site. The site recovered its lost traffic in about 2 months after uploading the file.

Is it still worth creating a disavow file nowadays?

Normally I would say no, because creating a disavow file can backfire on your site, even if it is done correctly.

You might add sites with a perfect profile for a disavow file: spammy, toxic, etc., but if these sites are old, your backlink profile has only 20 domains and these are 300 domains, Google can be dumb enough to take your disavow file into account (which it should), but the shock for the site of having so many domains disavowed can be so big that the site might simply drop in rankings.

So is a disavow file worth it, or is it too risky?

If you have a site with traffic and it falls into the class described above: very few clean domains, many spammy and toxic ones, but old and well-established, it’s better to leave it alone and not bother creating a disavow file, because anyway, most of the time, Google knows how to figure out which sites are toxic or not.

Can Google simply ignore the disavow file?

Yes, most of the time it doesn’t take it into account. As I said, I believe that in more than 150 cases of creating a disavow file, it only worked once.

When and how to create a disavow file if you really want to take this step?

First of all, for beginners, here are the technical steps:

Create a simple .txt file (UTF-8).

On each line, write:

domain:site-name.com to disavow an entire domain

or the full address of a page if you only want to block a single link.

You can add comments with # (Google ignores them).

Save the file and upload it here:

https://search.google.com/search-console/disavow-links

And so you don’t have to do everything by eye, use tools like Ahrefs, Semrush, Majestic, or even Google Search Console to build your list. For example, in Ahrefs go to Backlink profile and start filtering by DR, anchors, etc.

What should you add to the disavow file?

I’ll start by telling you what not to add.

First of all, do not add domains that simply have a Domain Rating of 0. This is a practice used by many amateurs who have no idea about SEO and don’t understand the concept of SEO or a site’s value. Many people think that if a site with Domain Rating 0 links to your site, it’s bad.

That’s pure nonsense. Let me give you an example:

If you have an online bookstore, and a book enthusiast has a site where they talk about books, but the poor guy created his site just 2 months ago and links to you… Should I put this site in the disavow file? No. Not at all. That link is probably better than one from a Domain Rating 30 news site that has nothing to do with your niche, so don’t make this mistake.

So what kind of sites should you add to the disavow file?

First, look at the TLDs.

If you see TLDs such as: .xyz, .top, .club, .online, .site, .space, .pw, .gq, .cf, .ml, .ga, .tk, .work, .icu, .bid, .men, .win, .click, .loan, .download, .party, .science, .cam, .monster, .fun, .in, .pk, they are most often not natural links to your website, so consider adding them to your disavow list.

Manual inspection of unfamiliar sites

Then go through the sites one by one, even if there are thousands (yes, you heard right, even if there are thousands). Open the sites that don’t look familiar and if you see strange languages, Chinese, Russian, etc., and your site is in English, straight to the disavow file.

Scraper and directory listings

Also, if the sites you open are the kind of site listings that scrape the entire internet and add your site to a list of hundreds or thousands of links, straight to the disavow file.

Sites with adult or shady ads

Next, if you find sites that seem normal but inside have only links and strange banners to adult sites or potency sites, those kinds of ads go straight to the disavow file as well.

Don’t blindly trust “toxic” flags from tools

Another side note for those who add to the disavow list sites that certain tools flag as toxic. Never rely on such tools. I have seen many sites flagged as toxic when that was not the case.

Check anchor text for spam attacks

And last but not least, you need to check the anchors through which those sites link to your site.

There are bad actors, “SEO agencies” that intentionally spam the entire internet by placing fake anchors on decent sites, but anchors that can harm your site, especially through spammy content. For example, see the image below showing the kind of anchors pointing to this site:

Monitor backlinks regularly

If you catch this type of site early, you’ll need to add it to the disavow file. That’s why I recommend checking your backlink profile weekly, for your own site or your clients’ sites. Catching issues early is much better, because as I said earlier, if you wait for years, even a toxic and spammy site can take root and there may be consequences if you intervene too late with a disavow file (assuming Google will even take it into account)

That being said, I believe it’s still better to avoid creating any disavow file and to let old Google separate the good sites from the bad ones, because if we examine 100 successful organic sites right now, I assure you that 99 of them have spammy links and no one has paid them any attention.

The post The Disavow File – Is it really worth creating it? appeared first on VladSand.