To better understand content, LLMs use mechanisms similar to traditional search engines, but they also come with their own specific characteristics.

In this article, I’ll walk you through the steps required to make your website more visible to LLMs and, of course, to increase your chances of being mentioned or cited by them.

Allow AI Crawlers to Access Your Content

Just like traditional search engines (Google, Bing, etc.), LLMs need access to your website in order to crawl it and even consider it in the first place.

The following bots are important, each serving a specific role in accessing and processing content correctly:

• OAI-SearchBot – An OpenAI bot used for indexing and search.
  
• GPTBot – Another OpenAI bot, used for collecting public data to train language models.
  
• ChatGPT-User – Identifies traffic coming from real ChatGPT users.
  
• PerplexityBot – A bot developed by Perplexity.ai that retrieves information to answer user questions in real time.
  
• Perplexity-User – The user-agent used when a Perplexity user opens a link from an AI-generated answer.
  
• ClaudeBot – The user-agent used by Anthropic for automated crawling and site indexing for Claude.
  
• Claude-User – The user-agent triggered when a Claude user opens a link from an AI-generated answer.
  
• Claude-SearchBot – Used by Claude to search for and evaluate web pages as potential sources before citing them.
  

By allowing access to all these bots, you significantly increase your real chances of being discovered and cited.

P.S. Not all of the bots listed above directly influence whether your site appears in AI citations (for example, GPTBot is theoretically used only for training purposes). That said, I wouldn’t bet my house on the idea that if OpenAI doesn’t use your site for training, it couldn’t indirectly affect your perceived credibility or “score” when it comes to citations.

Add Relevant Schema Markup to Your Templates

Structured markup acts as a native language for LLMs and AI Overviews, enabling fast, accurate, and structured interpretation of on-page data.

LLMs rely heavily on understanding your site’s structure, what your website is about, how it’s organized, and how individual sections relate to one another. They aim to narrow down as precisely as possible to the most relevant pages and even specific sections within those pages, so they can return the exact piece of information your site provides to the user asking a question.

To support this, you need to make things easier for the LLM, which means simplifying the structure of each page using schema.org markup.

Depending on the type of page, you should implement the appropriate structured data:

Examples:

• Blog article – https://schema.org/Article
  
• Service page – https://schema.org/Service
  
• Product page – https://schema.org/Product
  
• Pages with a clear FAQ section – https://schema.org/FAQPage

etc.

P.S. You don’t need to fill out every single schema property, that’s a quick way to overcomplicate things and potentially break database connections. The best approach is to keep it simple.

For example, for the https://schema.org/Service schema, you can safely use just:

• name
  
• serviceType
  
• description
  

The schema will still be valid and correctly understood by LLMs, which is the ultimate goal.

Structure Your Headings for Maximum Clarity

Well-structured headings help AI understand hierarchy and context.

Just like in traditional SEO, heading structure is important, arguably even more so here. As mentioned earlier, LLMs rely heavily on structural clarity. A correct heading hierarchy for each page type should follow a clean structure like:

<h1>Main Heading</h1>
  <h2>Subheading</h2>
    <h3>Sub-subheading</h3>
  <h2>Subheading</h2>
    <h3>Sub-subheading</h3>
    <h3>Sub-subheading</h3>

Prioritize Speed That AI Actually Cares About

First, it’s important to note that classic speed metrics aren’t interpreted by LLMs in the same way they are by traditional search engines.

For example, metrics like LCP (Largest Contentful Paint) or other purely visual UX indicators don’t really matter for LLMs, because the user interaction happens on the AI platform, not on your website. Your site’s primary role is to be crawlable and easy for LLMs to understand.

So what does matter when auditing performance for LLMs?

TTFB (Time to First Byte) / Server Response Time

As the name suggests, this metric measures how long it takes for the server to start responding.

In short: you need a fast server.

FCP (First Contentful Paint)

This is the moment when the first visible piece of content appears (text, image, SVG).

FCP matters for LLMs because it signals that the site delivers readable content quickly; a sign that the information is accessible, stable, and worthy of being read and cited.

Make Sure Key Content is in the Raw HTML

This step checks whether key elements, such as the main title, featured image, and actual page content – are loaded directly in the HTML or injected later via JavaScript.

You need to ensure that:

• The H1 heading
  
• The main body content
  
• The first image
  

are all visible directly in the page source, without relying on JavaScript injection.

This is crucial for LLMs, as they process vast amounts of information and need fast, reliable access to content. Pure HTML is the fastest and safest way for them to extract information correctly.

JavaScript-injected content can still work, but as of writing this article, LLMs behave similarly to how search engines did about 14 years ago, strongly favoring clean HTML over JS-rendered content.

Don’t Rely on JavaScript for Core Content

Here, the page is analyzed with JavaScript enabled and disabled.

Any content that disappears when JavaScript is turned off is, in practice, invisible to LLMs.

Keep Your Brand Identity Consistent Everywhere

Just like in classic SEO, being visible and cited by LLMs requires you to be verified as a real-world entity.

That means consistent NAP information everywhere you appear online – from your Google Business Profile to your official website, partners, and press releases.

Inconsistencies or missing information lead to entity fragmentation, making it difficult for LLMs to confidently determine that all mentions refer to the same brand.

The post How to Make Your Website Visible to LLMs  appeared first on VladSand.

It usually comes up when they want to know the total number of links their site has gained through a link-building strategy, or when they start using backlink analysis tools like Ahrefs or Majestic SEO and realize, to their shock, that their competitors have tens or even hundreds of thousands of backlinks, while they have only a few thousand.

The answer here is short and straight to the point:

The number of backlinks does not matter at all. What truly matters is the number of referring domains, the unique websites linking back to you.

Of course, the quality of those domains matters too, as does the type of website and whether it is relevant to your niche, but that is a whole different discussion.

Why the Number of Backlinks Doesn’t Really Matter

Think of it this way:

You get a backlink from Site A, placed inside a guest post. That gives you one link from Domain A, pointing to your website from within an article. Pretty straightforward so far.

Now you get another backlink from Site B, also via a guest post. So far, the setup looks identical to Site A.

But Site B has a completely different way of handling its content and internal structure. For example, your article, the one containing your backlink in the first paragraph, might be listed in multiple categories. On top of that, Site B may use indexable tags, pagination, archives, and author pages, which could automatically create dozens of additional pages where your backlink shows up multiple times.

Now, do you really think those dozens of extra backlinks, generated this way (which is very common), count for more than a single backlink from a well-structured site that does not multiply its content across pages?

Absolutely not.

Different Anchors, Different Backlinks From The Same Domain

You might be wondering: What if I have multiple backlinks from the same site, but each one uses a different anchor and comes from a different article?

In that case, Google does not completely ignore the second link, but its link equity, meaning the value it passes, is significantly lower. From a single domain, the first link carries the most weight, the second contributes less (but not zero), and by the third or fourth, the value keeps dropping.

That said, take all this with a grain of salt, because there are plenty of nuances and exceptions in how this works.

You might walk away from this thinking you only need one backlink per domain. Not at all. That would mean building a strategy with hundreds of websites, each linking to you only once, and that would not look natural either.

So, keep things balanced and aim for a natural link profile.

Or, to put it simply: buy your links, but do it naturally. 🙂

The post Domains vs. Backlinks: What Really Matters for SEO? appeared first on VladSand.

What Exactly Is SEO Cannibalization?

In case someone has no idea what cannibalization means, let me take a short moment to explain it.

Cannibalization happens when your website ranks for a keyword with a different page than the one that’s actually supposed to rank, the one that’s properly targeted for that keyword.

This usually happens when you have at least two pages that are very similar for various reasons:

• They have similar meta titles
  
• They have similar visual titles (H1 headings)
  
• Or the content itself is very similar
  

When pages share at least two of these elements, they end up cannibalizing each other (meaning they compete against one another for the same keywords). In other words, inside those meta titles, headings, or content, the targeted keywords start overlapping. As a result, search engines don’t know which page to show for a given query.

In most cases, the website ends up downgrading all the pages involved, and effectively, none of them has the full ranking potential anymore. This phenomenon is one of the most common reasons why websites fail to rank properly.

Why No One Talks About External Cannibalization and Why It Matters

There’s another type of cannibalization that’s at least as common as internal content cannibalization. These are called external cannibalizations, meaning issues created when inexperienced SEOs or people who don’t understand how to build a proper link-building infrastructure or strategy end up using the same anchor text for different URLs.

And to be honest, the reason no one talks about it is that most people don’t even take it into consideration, and it simply never crosses their mind that this aspect could actually cause harm. First of all, cannibalization audits are done far too rarely, and secondly, most SEO specialists don’t even realize there could be issues in this area.

What Happens When You Use the Wrong Anchors in Link Building?

Example:
An SEO specialist wants to promote the URL https://site.com/green-laptops/, but on the same website, there’s also the main category: https://site.com/laptops/.

If that person uses the anchor text “laptops” instead of “green laptops” in their link-building strategy, there’s a good chance that https://site.com/green-laptops/ will start ranking for the keyword “laptops” instead of https://site.com/laptops/, or might even completely outrank it.

Of course, there are many nuances to consider here:

• The non-targeted URL (the wrong one) might have multiple incorrect anchors, not just one.
  
• The correct URL (https://site.com/laptops/) may have more targeted anchors than https://site.com/green-laptops/, but if the linking websites pointing to “green laptops” have much higher authority, then even a single misplaced anchor can cause cannibalization and permanently damage your rankings (and I’ll explain below why I said permanently).
  

How Common Is External Cannibalization?

Honestly, very often. I frequently run targeted audits for cannibalization issues, and in many cases, I find this exact problem, wrong anchors, which end up ruining entire sections of a website.

I usually notice this after completing the initial on-site cannibalization checks and seeing that there’s nothing wrong with the content itself. The next step is to analyze the external backlinks for the cannibalizing expressions, and that’s when I find what I don’t want to see: anchors placed incorrectly.

Can You Fix External Cannibalization Once It Happens?

This brings me back to what I said earlier, that a website can be damaged permanently, and here’s why I used those harsh words.

First of all, because once the issue exists externally, it’s no longer in your control. You might have placed an external link (or maybe someone else did it without your consent) using the wrong anchor text. In most cases (and I’ve seen dozens), it’s extremely difficult to convince the site owner or webmaster to change that anchor to the correct URL.

This happens mostly because the sites where links are placed usually do this constantly, it’s part of their business, and they don’t have the time or motivation to stop making money just to fix a link they added two years ago. Of course, I’m not saying it’s impossible; sometimes you do come across webmasters with common sense who’ll actually help you out.

How to Avoid External Cannibalization in the First Place

The main problem occurs because the people creating the link-building strategy don’t know the website well enough and are very superficial when planning it. They don’t perform internal research to see which anchors fit better for certain URLs, or, more precisely, which URLs fit better for specific anchors.

So, My Advice Is:

• Know the website well if you’re an SEO specialist.
  
• If you’re the site owner and you’ve outsourced your link-building strategy to an agency or freelancer, make sure you provide all the necessary details. If you want to promote a specific category, show them the subcategories as well. Don’t leave everything in the hands of so-called “specialists,” because most of them just want to get the job done quickly, collect their money, and go home.
  
• And, of course, last but not least, have some basic SEO common sense when creating a link-building strategy, because as I mentioned above, even a single misplaced anchor can destroy a business.

The post Content Cannibalization vs. External Linking Cannibalization: What’s the Real Difference? appeared first on VladSand.

Of course, there are plenty of tools for every type of need in everyday SEO tasks, from keyword research to ranking, crawling, reporting, and so on.

But today, I will talk from my personal experience and the way I do my work, about Chrome tools that simply help you work more precisely, more confidently, and of course, faster.

I’m not going to present each tool the same way they are already presented in the other 50,000 blog posts out there. Instead, I will go straight to the specific needs in a technical audit and show the best Chrome extension option for each basic need.

Just to clarify, I’m not going into the category of famous extensions from tools like SemRush or Ahrefs, which make our work easier through their Chrome add-ons, but rather stand-alone tools, as I mentioned above, that serve the primary needs of any SEO professional, especially those focusing on technical SEO.

Instantly Check and Trace Redirects

One of the first things you need to look at when performing an audit, or when you want to see how certain URLs behave, is whether they redirect.

Why should you check if a URL redirects?

The reasons are numerous in SEO: to see if there are multiple redirect chains that could cause authority loss, to check whether HTTP redirects to HTTPS, to confirm if the redirect goes to the correct domain version (either non-www or www), or to verify whether URLs with or without a trailing slash redirect correctly, and so on.

The ideal tool for this specific task, in my case, is Redirect Path.

You can see the actual redirect, its type (301, 302, JS, etc.), the number of redirect hops until the final link and its status, detect any infinite redirect loops, or see through its inactive color indicators that no redirection exists.

Check Indexing and Robots Tags Effortlessly

To quickly and visually check page patterns and understand their indexing status, you need an extension that shows you, the moment you open a page, details such as: is it indexed? Is it index, follow or index, nofollow? Is it noindex, or maybe noindex, follow?

Why is it important to know a page’s robots and indexability settings?

There are multiple reasons. For example, to check whether pagination pages are indexable (which they should always be), or to verify whether a page where you’ve purchased a paid link for SEO purposes is marked as “index, nofollow”. That would mean you should get your money back because you’ve just been scammed. You can also instantly detect if important pages are non-indexable.

The tool that has helped me for years without giving me any headaches and that’s always reliable is Seerobots.

Simple and straight to the point, it shows you exactly what you need to know. For example, whether a page has:

<meta name="robots" content="index, follow">
<meta name="robots" content="index, nofollow">
<meta name="robots" content="noindex, follow">
<meta name="robots" content="noindex, nofollow">

See If Your Canonicals Are Pointing the Right Way

When checking a website, you must always be able to instantly see whether a specific page, whether you reached it intentionally, by accident, or as part of a pattern, canonicalizes correctly.

Why are canonical tags so important?

It’s simple: if a page you want to be indexable has a canonical pointing to another page, there’s clearly an issue that needs fixing.

For example, let’s take pagination. Every pagination page should be self-canonicalized, not because it adds SEO value directly, but to show search engines the diversity of products or content available. On many e-commerce sites, there are plenty of cases where pagination pages incorrectly canonicalize to their parent category, which is completely wrong.

The tool I’ve been using for a very long time to help me with this is Inspect Canonical.

With it, you can easily see if a page has a self-canonical, if it canonicalizes to another URL, or if it lacks a canonical altogether. It’s easy to interpret, as it visually displays the canonical status right in your toolbar, using color indicators.

Instantly View Meta Tags, Headings, and On-Page Details

The most important on-page SEO elements, such as meta titles and headings, should be visible instantly without having to dive into the source code.

For these purposes, I use SEO META in 1 CLICK.

This extension falls under the general SEO utilities category, as it doesn’t only show the meta title, H1 heading, and meta description of a page.

It also lists all the headers, all images and their corresponding alt texts, every internal and external link found in the source code, and the Open Graph tags for the page.

Detect Nofollow Links Directly on the Page

It’s absolutely necessary to be able to identify, at any moment, which links on your page are marked as nofollow.

Having this detail at hand allows you to instantly see if you or your platform mistakenly marked important links as nofollow, check which external links (especially pattern-based ones) on a client’s site are nofollow or dofollow, and verify during link building whether webmasters actually posted your links as dofollow or not.

The best and most stable tool for this purpose is Nofollow.

Once enabled, nofollow links appear on the page with a red dashed outline, making them stand out immediately.

Check Your Site’s Rendering Without JavaScript

Nowadays, as more and more websites rely heavily on JavaScript, it’s essential to see how a webpage and especially a page pattern appears when JS is disabled.

Viewing how a page renders with JavaScript turned off helps you ask the right questions:
– Can Google see all the content on this page?
– Can the FAQ section built in JS be read by search engines?

Then, of course, you move forward with testing whether search engines can actually crawl and read the page.

The tool I use for this purpose is Web Developer.

I should mention that I personally use this tool only for this specific reason, even though it’s extremely powerful. You can disable not just JavaScript, but also notifications, pop-ups, and cookies. You can even view image alt texts and image dimensions directly on the page in real time. It truly lives up to its name, being a complete tool for web developers. But, as I said, the way I use it is much simpler.

Instantly Count Words in Any Text

Almost every day you come across a text where you can’t really tell by eye how many words it has, and you just end up guessing.

To stop guessing and instantly see the exact word count of any text, Word Counter is the perfect extension.

Over time, I’ve tried several similar extensions, but unfortunately, most of them became deprecated and stopped working.

For now, it’s exactly what I need.

It’s super easy to use: just select the text you want to count, and it instantly displays the number of words.

Copy Texts or URLs from Anchors

One of the most useful extensions is the one that can extract URLs from anchors, menus, or any link text found on a page.

Why do you need to extract URLs and anchors?

If you want to do keyword research and don’t want to manually write down all the category names from a menu, subcategories, or even product names from a page, you’ll need a tool that can extract anchors automatically.

If you want to crawl certain URLs from a page to check their status or meta titles, you’ll also want to be able to do it instantly with a single extension.

One such tool I used to rely on was Link Clump, but it stayed deprecated for quite a while, and I couldn’t find anything similar. So I created my own tool to extract anchors and URLs, called Magic Anchor & URL Grabber.

It’s very easy to use: copy the anchor text instantly by pressing Z + Left Click (hold Z and click), and copy the URL behind the anchor by pressing U + Left Click (hold U and click). You can also bulk select either anchors or URLs.

It’s tested on Windows (not on Mac yet), and personally, it hasn’t disappointed me once.

Instantly Test Schema Markup and Structured Data

How many times have you been too lazy to open Google’s Rich Results Test or schema.org just to check the schema markup of a page?

I’ve been there a million times. Every single time, I’d reluctantly open five tabs at once for different URLs just to see what schema each had.

Honestly, to my shame, it never even crossed my mind that there could be a tool that takes you directly to those sites. I discovered Rich Results – Structured Data Test Plugin quite recently, and I’ve been using it constantly ever since.

It does exactly what it should: it takes you straight to the Schema Validator site with your URL already entered and ready for testing.

Mapping the Web Technologies Used

Last but not least, I use Wappalyzer.

It’s an extremely useful extension that, with a single click, shows a wide range of interesting details. You can instantly see which frameworks are being used (like React, Angular, or Laravel), what web server runs in the background (Apache, Nginx, LiteSpeed), or what programming language powers the site.

Moreover, it can tell you if a site is built on an e-commerce platform such as Shopify, WooCommerce, or Magento, what hosting services it uses (Amazon AWS, Google Cloud, DigitalOcean), and even what payment systems are integrated, like Stripe or PayPal.

If you’re curious about the marketing side, Wappalyzer also shows tracking tools (Hotjar, Facebook Pixel, Google Tag Manager), email marketing platforms (Mailchimp, Klaviyo), and even advertising or live chat systems (Intercom, Tawk.to).

I hope this was helpful and that I gave you at least two or three tool ideas you didn’t already know about.

The post 10 Must-Have Chrome Extensions Every SEO Expert Can’t Live Without appeared first on VladSand.

But one very important fact is often overlooked: the biggest issue right now for websites, when it comes to indexing and being processed by LLMs (Large Language Models), is not about optimization. The real problem is that LLMs simply cannot “see” most websites.

Why AI Crawlers Can’t Access Many Websites

This happens because, although we in the SEO world are keeping up with trends, in this case with AI and LLMs, some web hosting providers have unfortunately remained stuck in the “Middle Ages” when it comes to AI.

Even at the time of writing this article, they are still blocking AI bots in the same way they block aggressive, invasive bots such as MJ12bot, SemrushBot, LinkpadBot (the typical bots used by SEO and online marketing tools).

How to Check if Your Server is Blocking LLM and AI Bots

First, we need to identify which LLM bots are relevant at the moment.

Among the most relevant bots currently are:

• OpenAI bots (GPTBot, ChatGPT-User & OAI-SearchBot)
  
• Perplexity (PerplexityBot, Perplexity-User)
  
• Anthropic (ClaudeBot, Claude-User, Claude-SearchBot)
  

Practical Methods to Test Whether Bots Are Being Blocked

Method 1: Using Screaming Frog to Test AI Bot Access

This is the simplest method. You just need a Screaming Frog license, and in Configuration > User Agent, you can select the bot you want to check to see whether your site (or server) is blocking it:

If you receive a 406 error message, it means the server is blocking the ChatGPT bot from crawling your site.

If the bot starts crawling your site and returns status 200 for the pages it visits, that means the server is not blocking GPTBot.

Here I’ll make a quick note regarding what OpenAI states on its official site: their three main bots (GPTBot, OAI-SearchBot, and ChatGPT-User) each have different roles.

Specifically:

• GPTBot – is used for training AI models (OpenAI states it does not affect appearance in search results).
  
• OAI-SearchBot – is used for indexing for ChatGPT Search. This is theoretically the most important if you want your site to be crawled, processed, and shown as a source in ChatGPT.
  
• ChatGPT-User – this bot needs actual access when users click on links or perform live requests, such as “check if there are blue balloons on website.com”. If this bot is blocked, GPT won’t be able to visit the page.
  

As for the other bots, Claude and Perplexity, you can check them in Screaming Frog in the same way as OpenAI’s bots.

Method 2: Testing AI Bots with curl (HEAD Request)

This method is more direct and doesn’t require any special software. Open either the Mac Terminal or Windows Command Prompt and run the following header checks for each bot:

OpenAI – GPTBot

curl -I --ssl-no-revoke -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot" https://example.com

OAI-SearchBot

curl -I --ssl-no-revoke -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" https://example.com

ChatGPT-User

curl -I --ssl-no-revoke -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" https://example.com

PerplexityBot

curl -I --ssl-no-revoke -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://www.perplexity.ai/bot)" https://example.com

Perplexity-User

curl -I --ssl-no-revoke -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user" https://example.com

ClaudeBot

curl -I --ssl-no-revoke -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +https://www.anthropic.com/claude)" https://example.com

Claude-User

curl -I --ssl-no-revoke -A "Mozilla/5.0 (compatible; Claude-User/1.0; +https://www.anthropic.com/claude)" https://example.com

Claude-SearchBot

curl -I --ssl-no-revoke -A "Mozilla/5.0 (compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude)" https://example.com

To understand what this curl command does and where this information comes from: there’s no magic or hack involved.

The major companies behind these bots (OpenAI, Anthropic, Perplexity, etc.) publicly provide all the information about their crawlers, including their full user-agent strings, exactly as they appear in the HTTP headers of real requests.

If you’re wondering about the “revoke” part, the –ssl-no-revoke flag is included because some Windows versions throw SSL errors when they cannot check the certificate’s revocation status.

Case Study: Real Bot Access Test on VladSand.com

OAI-SearchBot gets through (status 200):

As you can see, the GPT crawler that indexes content has access and returns a 200 status.

ChatGPT-User (status 406) – blocked, unfortunately, in my case:

ChatGPT-User is blocked by the server, by the hosting company (I’m currently in a battle with them to explain why it needs to be unblocked), but hopefully they will eventually fix this.

How to Unblock AI and LLM Bots on Your Website

If you’re using shared hosting (like most websites), the only way to unblock these bots is to contact your hosting provider and ask them to whitelist the relevant User-Agents or IP ranges. Fortunately, this is usually a quick fix once they understand the issue.

Most major AI companies also publish their official IP ranges, which can be useful if your firewall or hosting provider uses IP-based filtering. Here are the official resources:

GPTBot: https://openai.com/gptbot.json

OAI-SearchBot: https://openai.com/searchbot.json

ChatGPT-User: https://openai.com/chatgpt-user.json

PerplexityBot: https://www.perplexity.ai/perplexitybot.json

Perplexity-User: https://www.perplexity.ai/perplexity-user.json

*Anthropic (Claude) has not provided official details about the IP ranges it uses.

If you’re on a dedicated server or VPS, you can manually adjust the server configuration, for example, by adding specific ModSecurity rules or firewall exceptions to explicitly allow AI crawlers.

Other Common Reasons AI Bots Are Blocked

Here I focused on blocking at the server level, because in most cases this is where access for these bots is cut off. However, they can also be blocked in other ways, sometimes without you even realizing it:

• through the robots.txt file
  
• through external firewalls (Cloudflare, Wordfence, etc.)
  
• through security plugins
  
• or implicitly through IP blocking (if your hosting provider blocks certain countries, the IPs of OpenAI / Perplexity / Anthropic may be rejected even if the User-Agent is valid)

The post Is Your Server Blocking AI Bots and LLM Crawlers? appeared first on VladSand.

What Is Pattern Content (Pattern Navigation)?

Here, I’m referring to the fact that there are repetitive internal anchors within a site that use internal linking, but have different URLs behind them.

Examples of such areas include:

• Menu areas, where we have the same words but different categories (in the case of e-commerce), for example: “Shoes”, “Jackets”, but the links point to different URLs:

example.com/women-shoes/
example.com/men-shoes/
example.com/kids-shoes/
etc.

• “Read More” buttons, which are mostly found in article sections, but also appear in various areas throughout the site. Just like in the case of menu items with identical names, these buttons have different URLs behind them but use the same anchor text: “Read More”.

Why Repetitive Anchors Can Confuse Search Engines

At first glance, most people would say: “Why would that be a problem? The links are always pointing to different pages anyway.”
Exactly! Even though the links point to different URLs, the anchor text is always the same. The search engine can get confused and may stop giving maximum value to the proper anchor of the linked article. This happens because it sees generic anchors like “Read More” or “Shoes” used across tens of thousands of pages. And as a result, it doesn’t pass the maximum value to the actual landing page behind the link.

How to Solve the Pattern Anchor Issue

The answer is very simple: a HINT. We need to give the search engine a hint about what lies behind the anchor.

Of course, it wouldn’t be practical to have a menu where the parent category “Women” contains all subcategories named “Women Shoes”, “Women Boots”, “Women Pants”, and so on. That would look awkward from a UX perspective, and the menu would seem overloaded.

Creating a Clear Hint for Search Engines

The hint is created by adding a title tag (title="") that contains, in the case of “Read More” anchors, the name of the article or page behind the link.

For those who don’t know, title="" is an attribute that displays extra context to users when they hover over the element.

Although it doesn’t have a direct SEO effect, like actual SEO content or an image’s alt tag, it provides a valuable hint and additional context for search engines, helping them differentiate between the URL behind the anchor and the anchor text itself.

A Correct Way to Implement This on a Blog:

A Correct Way to Implement This in an E-commerce Website:

The post Identical Pattern Anchors – How to Avoid Search Engine Confusion appeared first on VladSand.

In today’s article, we’ll focus on filters that already exist on your website, but you’re not sure which ones to make indexable, without harming your site’s indexing and without risking to damage its SEO structure.

Why Choosing Filters to Index Is So Difficult

Choosing which filters to index can be a tricky exercise because there are multiple factors to consider when deciding which filters should be indexable.

One of the most common issues is platform limitations. Some e-commerce platforms, whether mainstream or custom, are configured in a way that makes filters either impossible to index correctly or impossible to index at all.

Common Indexing Challenges for Filters

JavaScript-Generated Filters

We’re not talking here about how URLs are generated, but about the fact that, when a filter is applied, nothing changes in the browser’s address bar. No new URL is created; everything happens through JavaScript on the page.

In this situation, filters have no SEO value and don’t contribute to organic visibility. They only partially help improve the site’s UX.

Either All Filters Are Indexed or None

This is a very common situation: the platform generates filter URLs, but it either indexes all the filters on the site (or within a specific category) or none of them.

This can harm your site more than it helps. Imagine your website has 100 categories with 30 filters each. Essentially, each category and every filter gets indexed individually.

You might think: “What’s wrong with that? I’ll have more indexed filters, more ranking opportunities, more URLs.”

The Risks of Indexing Every Filter

• You will index filters with little or no content (e.g., 1–2 products). This creates thin content across your site, which could result in tens of thousands of thin pages. This can lead to algorithmic penalties, such as those from Google Panda.
  
• You’ll end up indexing filters with no SEO value, such as price, free shipping, new arrivals, discount only, in stock, out of stock, etc. Nobody searches for these filters, so they generate extra content with zero search intent. Search engines may interpret these pages as low-value, which can also negatively affect your crawl budget (the number of pages Google is willing and able to crawl on your site within a specific time period).
  

Lack of Customization for Filter Pages

Let’s assume you have filter URLs and can choose which ones to index, but you cannot customize them. This means the filter pages cannot have their own visible titles (H1) or unique meta titles in the format “category + filter” or “filter + category.”

If you can’t do this, all your indexable filter pages will end up having the same visual title and meta title as the parent category.

Example:
Instead of having pages like:

• example.com/women-shoes/filter/red → Red Women Shoes
  
• example.com/women-shoes/filter/blue → Blue Women Shoes
  
• example.com/women-shoes/filter/black → Black Women Shoes
  

You’ll have all these URLs using the same meta title and visual title as the main category page:

example.com/women-shoes → Women Shoes

This leads to keyword cannibalization between the category and its filters, resulting in a complete SEO mess.

Indexing Multiple Filter Combinations

Another issue that can occur is the platform not offering an option to deindex combinations of two or more filters applied at the same time.

Ideally, you should have the option to index a maximum of two combined filters that make sense together.

Valid example:

• example.com/t-shirts/filter/blue/xxl/ → XXL Blue T-Shirts
  

This is a common and correct scenario for indexing two combined filters.

Invalid examples:

• Size + size: t-shirts/filter/xs/xxl/
  
• Size + color + size: t-shirts/filter/xs/blue/xxl/
  

These combinations don’t make sense, have nearly zero search intent, and can generate millions (or even hundreds of millions) of useless pages.

A site that realistically has only a few thousand pages could end up with millions of indexable, irrelevant pages, leading to a potential SEO collapse.

How to Choose the Right Filters to Index

Let’s assume all the above issues have been solved, you have a good developer who can customize everything, or your platform has all the necessary plugins and features to perfectly handle filter functionality.

For a new site, even with the entire infrastructure in place, the ideal approach is to start with all filters set to “noindex.” If these are not noindex and you don’t start from this calming fact, that’s okay, at least from this point on you’ll be able to decide which filters to index and apply noindex to the ones that shouldn’t be.

The Simple Way – Keyword Research

You can use keyword research tools such as Ahrefs, SEO Monitor, Ubersuggest, and others.

These tools are helpful when you start from a single keyword (for example, the category name) and they provide relevant keywords associated with it.

However, most of the time, these tools focus on main category keywords. They can help you:

• Create new categories
  
• Optimize existing categories
  
• Occasionally discover good filter ideas with sufficient search volume
  

Identify Target Filters on Your Platform

This is the part we’ll focus on. Suppose you have an e-commerce site with a wide variety of products. You’ve set up all necessary filters in the database (color, sizes, etc.), and now you want to identify exactly which filters should be indexed.

This method is simple and can be applied using basic tools to help you find the right filters to index.

Steps:

1. Open the Keyword Combiner tool.

2. Open the category you want to analyze. Example: “laptops.”

3. In Keyword Combiner, add both the singular and plural forms: laptop / laptops.

4. Use a tool to extract anchor texts from URLs, such as Magic Anchor & URL Grabber, Linkclump (or other).

5. On the category page, extract the filter anchors (brand, type, operating system, etc.)

and paste them into Keyword Combiner, in filter section:

Then you generate the words and all the combinations of category (singular and plural) + filter (facet):

6. You can repeat this process for multiple categories until you’ve covered them all.

This method is useful when you don’t have backend database access. If you do have access, you can directly extract category and filter names from the database and combine them in Keyword Combiner without manually going through each category.

Once you’ve generated category + filter combinations, use a keyword research tool (Ahrefs, SEO Monitor, etc.) to check the monthly search volume for each combination. This will give you a clear picture of which existing filters on your site have meaningful search demand and therefore should be indexed.

P.S.:

Even if you’ve identified the right filters and have the necessary infrastructure to index and customize them, you need to follow one key rule: if a filter URL lists only one product, that page should remain “noindex.”

Once a second product is added, the “noindex” tag should be dynamically removed, and the filter URL should become indexable.

For maximum SEO impact, it’s also recommended to add descriptive, targeted content (LSI – Latent Semantic Indexing ) to the first page of each filter, but not to its paginated pages.

The fewer filters you index, the richer the descriptive text should be, to compensate for the lack of product content.

The post How to Decide Which Filters (Facets) to Index on an E-Commerce Website appeared first on VladSand.

But a link building strategy really has to be, at its core, “a strategy” . Today I will cover the two main areas of a link building strategy, namely: how to choose the URLs and how to create the anchors that are part of the link building strategy.

How to Select the Right URLs

Number of impressions

This is not the most important criterion, but it is the most consistent. It will be part of the other criteria presented when selecting URLs. Choose the URL based on the number of impressions received within a specific time period.

There is no point in adding a URL with very few impressions, unless it is a Hero URL (we will discuss this type of URL below).

URLs with non-brand keywords

For a correct analysis of the link building strategy, the brand name must always (absolutely always) be excluded, because the brand modifies the average position of the URL, and the average position of that URL is not the real one. We target organic traffic on non-branded keywords unaffected by the domain’s own name.

Average position

First, we look at the average position of the URLs we want to include in the link building strategy.

The average position must, of course, correspond to the selected time period. As a rule, we always select a short period. One week, or a maximum of three weeks, is ideal.

It is not correct to choose longer periods such as 3 months, 6 months, or 12 months to see the average position of URLs, because that average is NOT REAL. It is not real simply because it is spread over 12 months. You need the true and latest average position to know the current ranking of that URL.

The average position for selecting URLs in a link building strategy in order to achieve real results is:

• 4–7 to have a chance, after receiving links, to enter the top 3
• 10–12 to have a chance, after receiving links, to enter the top 10.

Value of URLs without backlinks

Consider that you have a URL with an average position of 7 achieved only through its content value, age, etc. Imagine what an additional link to this URL could mean.

Therefore, besides selecting URLs in a link building strategy based only on average position, also include URLs that have potential but no backlinks. Potential here means a decent position and a good number of impressions.

Hero URLs (Strategic Targets)

In addition to URLs that have a position we want to push forward (for example 4–7 or 10–12) or URLs without backlinks, of course a strategy also needs targeted URLs (if applicable).

These are the URLs that are in the client’s focus, for which there is a clear desire to rank.

Naturally, these URLs, as well as the others above that already have backlinks, should be chosen carefully and with a comparative analysis of the competition. For example, if we already have 20 backlinks with relevant anchors for a URL and we see that for the main keyword the competitors ranking above us have only 2 backlinks, it obviously makes no sense to spend resources acquiring more links in that area, because it is clear that backlinks are not the trigger for that type of keyword.

Tools and Methods for Gathering URL Data

You may have wondered by now: “Ok, you talk about average positions, number of impressions, etc., but how can I extract all these data?”

If we are dealing with a small presentation website, you can rely on the well-known Google Search Console to extract these data.

But, as the title of the article suggests, here we are talking about a link building strategy for e-commerce sites, so we will need more advanced tools that can extract URLs with details such as average position, number of impressions, etc., and at the same time allow very easy filtering.

We can use “Search Analytics for Sheets,” a Google Sheets add-on that is very easy to use for extracting data from Google Search Console (you need to be logged in to the site’s Google Search Console account for the tool to pull details from the site’s GSC).

Or we can use Screaming Frog, leveraging the Search Console API, which helps us see in real time, while crawling the site or batches of URLs we provide; data such as number of impressions, average position, CTR, etc., essentially all the data from Google Search Console.

To see the number of backlinks for URLs, tools such as Ahrefs or Majestic SEO can be used. (For example, in Ahrefs you can input batches of URLs and it will tell you how many domains link to each respective URL.)

How to Choose the Right Anchors

We are now at the stage where the URLs have been selected and that part is done. The next step is to choose the anchors. Below is how these anchors should be selected and created.

Where to Find the Best Anchor Opportunities

The safest way to see which anchors a URL needs is again Google Search Console, the fundamental tool of any SEO campaign. We look here, with external tools or manually, at which queries each URL ranks for (of course, as mentioned above, non-brand queries).

Any URL will have from a few to dozens or even thousands of queries it ranks for. Sort by number of impressions and then look at the average position.

Naturally, human judgment is required to exclude queries that are not related to the URL, queries that are too general, or queries clearly assigned to another URL on the site but for which your URL also received impressions.

Be careful: if you use in the link building strategy anchors that should be used for another URL; or if that other URL has the keyword in its meta title and overlaps with that anchor, you can create unintended cannibalization (URLs competing for the same keywords). Cannibalization caused by a poorly executed link building strategy is much harder to fix than internal cannibalization from duplicate content.

Types of Anchors to Use

Anchors with prepositions

It is a must for anchors to include prepositions if they are longer and not “one-word”. Without prepositions you may end up with anchors such as “Muscle building steps” instead of “Steps for building muscle,” or “Online personal brand building steps” instead of “Steps for building a personal brand online.”

As you can see, anchors without prepositions sound forced, unnatural, and clearly crafted for link building purposes. This can trigger algorithmic penalties such as Google’s Penguin update.

Generic anchors

Anchors like “click here,” “read more,” or “this link” are necessary for a natural link building strategy. They should represent about 1 in 10 anchors in a link building strategy, but ideally used only if the URL already has a backlink targeting the main keyword with a focused anchor.

Image anchors

Again, to stay safe and avoid the patterns typical of most link building strategies, it’s good to place links on images as well. Ideally, for maximum effect, the image should have an alt tag containing the main keyword of the URL.

Naked URLs

Naked URLs are simply the URL itself. This type of link should be used for naturalness and to avoid giving search engines any hint that you are deliberately pursuing a link building strategy.

Very natural long anchors

For maximum effect nowadays you should use anchors that target the main keyword but also include a naturally phrased sentence around that keyword.
Example: if you want to target the anchor “cheap laptops” in a link building strategy, you could use the following anchor: “I went specifically to a store to see if they have offers and find cheaper laptops.”

Short anchors

Literally, the use of short anchors such as “cheap laptops,” concise and to the point, but they should not be overused.

Percentage of URLs Targeted in a Link Building Strategy for an E-commerce Site

Category pages

The most important set of URLs on which a link building strategy for an e-commerce site should focus is category pages. Categories contain the most searched and relevant expressions and, of course, where competition exists. Therefore, about 80% of the entire strategy should focus on categories.

P.S.: Links to filters (if you have indexable filters) can also work here. You can treat filters in a link building strategy the same way you treat a category.

Product pages

Product pages are relevant in an e-commerce link building strategy mainly for naturalness. This ensures the site does not receive links only to categories and allows ranking for Hero products you specifically want to promote. A share of about 10% on product URLs is a reasonable percentage.

Links to the homepage

Backlinks to the homepage add naturalness, strengthen the brand, and more recently also help in the LLM area.

Anchors for homepage links should be used like: “website.com,” “Website,” “https://website.com,” “WEBSITE,” or combinations of generic anchors plus brand such as “read more on Website.com”.

Recommended Brand Anchor Percentages

Currently it is very important to have a percentage of URLs containing the brand in the anchors within link building strategies. As noted above in “Choosing the Anchors,” keep in mind that for every type of anchor presented you should naturally include the brand; except for generic anchors and naked URLs.

Brand anchors count heavily for naturalness and brand growth, especially for e-commerce businesses competing in crowded markets. Usually, on forums, blogs, etc., a person who naturally writes an anchor and adds a backlink also includes the brand.

Therefore:

• 20–30% of the anchors in a link building strategy should contain the brand for naturalness and E.E.A.T.
• 50% of all anchors should contain the brand if the goal is ranking in LLMs (Large Language Models), which place great emphasis on brand notoriety and trust.

To see what percentage of anchors on your site contain the brand name, I recommend using Ahrefs in the Backlinks Profile > Anchors section.

There you can filter the anchors containing your site’s name and calculate what percentage of all anchors currently include the brand.

For example, the Amazon site has a total number of anchors of 359,476,711
And a number of anchors containing “Amazon”: 74,715,513 → this results in a percentage of 20.78% of anchors containing the brand.

That is fine. But keep in mind that your site is not a huge brand like Amazon and therefore requires a higher percentage in link building strategies, exactly as mentioned above.

Note: you may see many fake anchors, patterns of anchors created by spam scrapers, so if you notice such patterns, filter them out to reach the most realistic percentage possible.

P.S.:
It should go without saying, BUT any URL included in a link building strategy must be indexable.

What does indexable mean?

• It must not be noindex
• It must not be canonicalized to something else (it should have a self-canonical)
• It must not redirect to something else
• It must not be a soft 404
• It must not be blocked by robots.txt

The post How to Choose URLs and Create Anchors for an E-commerce Link Building Strategy appeared first on VladSand.

Of course, this aspect of the search made life easier for us, especially those of us in online marketing, rather than for regular users, who rarely used this method. And honestly, rarely even click on page 2 of Google unless they really want something.

When did this change happen?

The first changes were noticed around September 10-12, when it was observed that this parameter no longer worked, either functioning intermittently or simply leaving the 10 results on the page without changing anything.

Google confirmed this on September 18 and basically responded to all the dissatisfied users who thought this was an error, clearly stating that it was not a bug but an intentional decision on their part.

Possible reasons why Google removed &num=100

Two major reasons come to mind as an explanation for this decision:

• Mainly because such pages were often used by tools to crawl large result sets, with impressions being generated in a very high percentage by bots.
• To align with the real behavior of users, who typically don’t go beyond the first page in their searches, and because Google wants to provide what it claims, an optimal UX for real users.

What changed after removing &num=100

Improvement of the Average Position in Google Search Console (GSC)

First of all, the average position changed drastically in Google Search Console for all websites that had a decent number of users and pages indexed both on the first page of Google and beyond position 10. (Naturally, for sites ranking mostly beyond position 20, no major changes are noticeable in GSC.)

Here is how the positions shifted, see the example for these 3 sites:

Site 1:

Site 2:

Site 3:

Sharp Drop in Impressions

There were sudden drops in impressions, especially on the desktop version.

SEO Tools and Rank Trackers Thrown Off

As a result of this change, SEO tools that measured average position and impressions were thrown off, showing display issues for rankings and/or missing data.

In addition, SEO tools will now have to spend more and work on new ways to extract this data. The first thing that comes to mind is that now, for 10 pages, they will need to perform 10 separate queries, and that is where the snowball effect begins.

Can you have a position worse than 10

A client asked me if it is still possible to see positions worse than 10 under these circumstances. Of course! The only difference is that now the impressions coming from page 2, page 3, page 4 will be much more natural and realistic. So for clients, this is clearly a win, their real situation will now be presented in a much cleaner and more accurate way.

The post Game Over for Google’s 100 Results – Why Average Position Just Improved appeared first on VladSand.

Questioning the real value of the Disavow File

Let’s get back to the subject, because we strayed a bit with history.

The disavow file, at least for me, has never had any clear value. I mean, it only helped me and seemed to make sense once in its entire history of use, namely around 2012, after Google Penguin came into the SEO world. 

(A small side note for those who don’t know what Google Penguin is: it’s a Google update that basically acts as a link police. Sites that use spammy backlink methods fall into its filter and get penalized, either physically with a direct message in Google Webmaster Tools, or algorithmically, without you realizing it, but you can tell something negative is happening to your site, by an instant drop in organic traffic)

A single successful case

Returning to the topic for the second time: the only time it seemed to make sense to create a disavow file was after Google Penguin was rolled out and I had a client, a gift-basket site, which until Penguin had only directory links, packages of hundreds of links bought all at once on a single anchor, etc. (basically, the kind of practices common back then).

This site was penalized (algorithmically, so without a message in GSC), and because it was post-Penguin, I said clearly the problem was there, in the area of spammy links, so I created a disavow file with hundreds of spammy sites linking to this site. The site recovered its lost traffic in about 2 months after uploading the file.

Is it still worth creating a disavow file nowadays?

Normally I would say no, because creating a disavow file can backfire on your site, even if it is done correctly.

You might add sites with a perfect profile for a disavow file: spammy, toxic, etc., but if these sites are old, your backlink profile has only 20 domains and these are 300 domains, Google can be dumb enough to take your disavow file into account (which it should), but the shock for the site of having so many domains disavowed can be so big that the site might simply drop in rankings.

So is a disavow file worth it, or is it too risky?

If you have a site with traffic and it falls into the class described above: very few clean domains, many spammy and toxic ones, but old and well-established, it’s better to leave it alone and not bother creating a disavow file, because anyway, most of the time, Google knows how to figure out which sites are toxic or not.

Can Google simply ignore the disavow file?

Yes, most of the time it doesn’t take it into account. As I said, I believe that in more than 150 cases of creating a disavow file, it only worked once.

When and how to create a disavow file if you really want to take this step?

First of all, for beginners, here are the technical steps:

Create a simple .txt file (UTF-8).

On each line, write:

domain:site-name.com to disavow an entire domain

or the full address of a page if you only want to block a single link.

You can add comments with # (Google ignores them).

Save the file and upload it here:

https://search.google.com/search-console/disavow-links

And so you don’t have to do everything by eye, use tools like Ahrefs, Semrush, Majestic, or even Google Search Console to build your list. For example, in Ahrefs go to Backlink profile and start filtering by DR, anchors, etc.

What should you add to the disavow file?

I’ll start by telling you what not to add.

First of all, do not add domains that simply have a Domain Rating of 0. This is a practice used by many amateurs who have no idea about SEO and don’t understand the concept of SEO or a site’s value. Many people think that if a site with Domain Rating 0 links to your site, it’s bad.

That’s pure nonsense. Let me give you an example:

If you have an online bookstore, and a book enthusiast has a site where they talk about books, but the poor guy created his site just 2 months ago and links to you… Should I put this site in the disavow file? No. Not at all. That link is probably better than one from a Domain Rating 30 news site that has nothing to do with your niche, so don’t make this mistake.

So what kind of sites should you add to the disavow file?

First, look at the TLDs.

If you see TLDs such as: .xyz, .top, .club, .online, .site, .space, .pw, .gq, .cf, .ml, .ga, .tk, .work, .icu, .bid, .men, .win, .click, .loan, .download, .party, .science, .cam, .monster, .fun, .in, .pk, they are most often not natural links to your website, so consider adding them to your disavow list.

Manual inspection of unfamiliar sites

Then go through the sites one by one, even if there are thousands (yes, you heard right, even if there are thousands). Open the sites that don’t look familiar and if you see strange languages, Chinese, Russian, etc., and your site is in English, straight to the disavow file.

Scraper and directory listings

Also, if the sites you open are the kind of site listings that scrape the entire internet and add your site to a list of hundreds or thousands of links, straight to the disavow file.

Sites with adult or shady ads

Next, if you find sites that seem normal but inside have only links and strange banners to adult sites or potency sites, those kinds of ads go straight to the disavow file as well.

Don’t blindly trust “toxic” flags from tools

Another side note for those who add to the disavow list sites that certain tools flag as toxic. Never rely on such tools. I have seen many sites flagged as toxic when that was not the case.

Check anchor text for spam attacks

And last but not least, you need to check the anchors through which those sites link to your site.

There are bad actors, “SEO agencies” that intentionally spam the entire internet by placing fake anchors on decent sites, but anchors that can harm your site, especially through spammy content. For example, see the image below showing the kind of anchors pointing to this site:

Monitor backlinks regularly

If you catch this type of site early, you’ll need to add it to the disavow file. That’s why I recommend checking your backlink profile weekly, for your own site or your clients’ sites. Catching issues early is much better, because as I said earlier, if you wait for years, even a toxic and spammy site can take root and there may be consequences if you intervene too late with a disavow file (assuming Google will even take it into account)

That being said, I believe it’s still better to avoid creating any disavow file and to let old Google separate the good sites from the bad ones, because if we examine 100 successful organic sites right now, I assure you that 99 of them have spammy links and no one has paid them any attention.

The post The Disavow File – Is it really worth creating it? appeared first on VladSand.