Control Web Crawlers and Crawl Budget in Nuxt · Nuxt SEO

[NuxtSEO](https://nuxtseo.com/ "Home")

- [Modules](https://nuxtseo.com/docs/nuxt-seo/getting-started/introduction)
- [Tools](https://nuxtseo.com/tools)
- [Pro](https://nuxtseo.com/pro)
- [Learn SEO](https://nuxtseo.com/learn-seo/nuxt) [Releases](https://nuxtseo.com/releases)

[1.4K](https://github.com/harlan-zw/nuxt-seo)

[Nuxt SEO on GitHub](https://github.com/harlan-zw/nuxt-seo)

Learn SEO

Master search optimization

Nuxt

 Vue

[SEO Checklist](https://nuxtseo.com/learn-seo/checklist) [Pre-Launch Warmup](https://nuxtseo.com/learn-seo/pre-launch-warmup) [Backlinks & Authority](https://nuxtseo.com/learn-seo/backlinks)

[Mastering Meta](https://nuxtseo.com/learn-seo/nuxt/mastering-meta)

- [Titles](https://nuxtseo.com/learn-seo/nuxt/mastering-meta/titles)
- [Meta Description](https://nuxtseo.com/learn-seo/nuxt/mastering-meta/descriptions)
- [Image Alt Text](https://nuxtseo.com/learn-seo/nuxt/mastering-meta/alt-text)
- [Social Sharing](https://nuxtseo.com/learn-seo/nuxt/mastering-meta/open-graph)
- [Rich Results](https://nuxtseo.com/learn-seo/nuxt/mastering-meta/rich-results)
- [Schema.org](https://nuxtseo.com/learn-seo/nuxt/mastering-meta/schema-org)
- [Twitter Cards](https://nuxtseo.com/learn-seo/nuxt/mastering-meta/twitter-cards)

[ Controlling Crawlers](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers)

- [Robots Txt](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/robots-txt)
- [Sitemaps](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/sitemaps)
- [Robot Meta Tag](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/meta-tags)
- [Canonical Link Tag](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/canonical-urls)
- [HTTP Redirects](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/redirects)
- [Duplicate Content](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/duplicate-content)
- [llms.txt](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/llms-txt)

[ Routes & Rendering](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering)

- [URL Structure](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/url-structure)
- [Pagination](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/pagination)
- [Trailing Slashes](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/trailing-slashes)
- [Query Parameters](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/query-parameters)
- [Hreflang & i18n](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/i18n)
- [404 Pages](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/404-pages)
- [Dynamic Routes](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/dynamic-routes)
- [Internal Linking](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/internal-linking)
- [Programmatic SEO](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/programmatic-seo)
- [Rendering Modes](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/rendering)
- [Security](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/security)

[ Launch & Listen](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen)

- [Getting Indexed](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen/going-live)
- [Google Search Console](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen/search-console)
- [Core Web Vitals](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen/core-web-vitals)
- [Indexing Issues](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen/indexing-issues)
- [SEO Monitoring](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen/seo-monitoring)
- [Site Migration](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen/site-migration)
- [IndexNow](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen/indexnow)
- [Debugging](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen/debugging)
- [AI Search Optimization](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen/ai-optimized-content)

1. [Learn SEO for Nuxt](https://nuxtseo.com/learn-seo)
2.
3. [Controlling Crawlers](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers)

# Control Web Crawlers and Crawl Budget in Nuxt

Manage how search engines crawl and index your Nuxt app. Configure robots.txt, sitemaps, canonical URLs, and redirects for better SEO.

[![Harlan Wilton](https://avatars.githubusercontent.com/u/5326365?v=4)Harlan Wilton](https://x.com/harlan-zw)8 mins read Published Nov 3, 2024 Updated Jan 29, 2026

Web crawlers determine what gets indexed and how often. Controlling them affects your [crawl budget](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget): the number of pages Google will crawl on your site in a given timeframe.

In 2026, crawler control extends to **AI Bot Governance** and the **Agentic Web**. You need to guide LLMs to your best content while protecting your data from unauthorized training.

Most sites don't need to worry about crawl budget. But if you have 10,000+ pages, frequently updated content, or want to manage how AI agents consume your data, crawler control matters.

## [Types of Crawlers](#types-of-crawlers)

**Search engines**: Index your pages for search results

- [Googlebot](https://developers.google.com/search/docs/advanced/crawling/overview-google-crawlers) (28% of bot traffic)
- [Bingbot](https://ahrefs.com/seo/glossary/bingbot)

**Social platforms**: Generate link previews when shared

- [FacebookExternalHit](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/)
- Twitterbot, Slackbot, Discordbot

**AI training & Inference**: Scrape content for model training or real-time answering

- [GPTBot](https://platform.openai.com/docs/bots/overview-of-openai-crawlers) (OpenAI training)
- [ClaudeBot](https://www.anthropic.com) (Anthropic training)
- [PerplexityBot](https://www.perplexity.ai) (Real-time AI search)
- [Google-Extended](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers) (Google's AI training)

**Agentic AI**: Bots that perform actions on behalf of users (e.g., booking, shopping)

- [OpenAI-GPT-4o](https://openai.com)
- [Claude-3.5-Sonnet](https://anthropic.com)

**Malicious**: Ignore robots.txt, spoof user agents, scan for vulnerabilities. Block these at the [firewall level](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/security), not with robots.txt.

## [Control Mechanisms](#control-mechanisms)

| Mechanism | Use When |
| --- | --- |
| [robots.txt](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/robots-txt) | Block site sections, manage crawl budget, block AI crawlers |
| [Sitemaps](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/sitemaps) | Help crawlers discover pages, especially on large sites |
| [Meta robots](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/meta-tags) | Control indexing per page (noindex, nofollow) |
| [Canonical URLs](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/canonical-urls) | Consolidate duplicate content, handle URL parameters |
| [Redirects](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/redirects) | Preserve SEO when moving/deleting pages |
| [llms.txt](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/llms-txt) | Guide AI tools to your documentation (via nuxt-llms) |
| [X-Robots-Tag](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag#xrobotstag) | Control non-HTML files (PDFs, images) |
| [Firewall](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/security) | Block malicious bots at network level |

## [Quick Recipes](#quick-recipes)

**Block page from indexing**: [Full guide](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/meta-tags)

pages/admin.vue

```
<script setup lang="ts">
useSeoMeta({ robots: 'noindex, follow' })
</script>
```

**Block AI training bots**: [Full guide](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/robots-txt)

public/robots.txt

```
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /
```

**Fix duplicate content**: [Full guide](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/canonical-urls)

pages/products/[id].vue

```
<script setup lang="ts">
const route = useRoute()
useHead({
  link: [{ rel: 'canonical', href: \`https://mysite.com/products/${route.params.id}\` }]
})
</script>
```

**Redirect moved page**: [Full guide](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/redirects)

nuxt.config.ts

```
export default defineNuxtConfig({
  routeRules: {
    '/old-url': { redirect: { to: '/new-url', statusCode: 301 } }
  }
})
```

## [When Crawler Control Matters](#when-crawler-control-matters)

Most small sites don't need to optimize crawler behavior. But it matters when:

**Crawl budget concerns**: Sites with 10,000+ pages need Google to prioritize important content. Block low-value pages (search results, filtered products, admin areas) so crawlers focus on what matters.

**Duplicate content**: URLs like `/about` and `/about/` compete against each other. Same with `?sort=price` variations. [Canonical tags](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/canonical-urls) consolidate these.

**Staging environments**: Search engines index any public site they find. Block staging/dev environments in [robots.txt](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/robots-txt) to avoid duplicate content issues.

**AI training opt-out**: [GPTBot was the most-blocked crawler in 2024](https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/). Block AI training bots without affecting search rankings.

**Server costs**: Bots consume CPU. Heavy pages (maps, infinite scroll, SSR) cost money per request. Blocking unnecessary crawlers reduces load.

## [Nuxt SEO Modules](#nuxt-seo-modules)

Nuxt handles crawler control through dedicated modules. Install once, configure in `nuxt.config.ts`, and forget about it.

[Nuxt SEO v5.1.02.6M1.4K The all-in-one module that brings it all together.](https://nuxtseo.com/docs/nuxt-seo/getting-started/introduction) [Robots v6.0.68.8M516 Tame the robots crawling and indexing your site with ease.](https://nuxtseo.com/docs/robots/getting-started/introduction) [Sitemap v8.0.1110M421 Powerfully flexible XML Sitemaps that integrate seamlessly.](https://nuxtseo.com/docs/sitemap/getting-started/introduction) [SEO Utils v8.1.62.2M123 SEO utilities to improve your Nuxt sites discoverability and shareability.](https://nuxtseo.com/docs/seo-utils/getting-started/introduction)

[The 2026 SEO Checklist for Nuxt & Vue Pre-launch setup, post-launch verification, and ongoing monitoring. Interactive checklist with links to every guide.](https://nuxtseo.com/learn-seo/checklist) [Haven't launched yet? Start with the Pre-Launch Warmup](https://nuxtseo.com/learn-seo/pre-launch-warmup)

---

[Twitter Cards Configure Twitter/X Cards to control how your links appear when shared on Twitter with rich previews, images, and metadata.](https://nuxtseo.com/learn-seo/nuxt/mastering-meta/twitter-cards) [Robots Txt](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/robots-txt)

On this page

- [Types of Crawlers](#types-of-crawlers)
- [Control Mechanisms](#control-mechanisms)
- [Quick Recipes](#quick-recipes)
- [When Crawler Control Matters](#when-crawler-control-matters)
- [Nuxt SEO Modules](#nuxt-seo-modules)

[GitHub](https://github.com/harlan-zw/nuxt-seo) [ Discord](https://discord.com/invite/275MBUBvgP)

### [NuxtSEO](https://nuxtseo.com/ "Home")

- [Getting Started](https://nuxtseo.com/docs/nuxt-seo/getting-started/introduction)
- [MCP](https://nuxtseo.com/docs/nuxt-seo/guides/mcp)

Modules

- [Robots](https://nuxtseo.com/docs/robots/getting-started/introduction)
- [Sitemap](https://nuxtseo.com/docs/sitemap/getting-started/introduction)
- [OG Image](https://nuxtseo.com/docs/og-image/getting-started/introduction)
- [Schema.org](https://nuxtseo.com/docs/schema-org/getting-started/introduction)
- [Link Checker](https://nuxtseo.com/docs/link-checker/getting-started/introduction)
- [SEO Utils](https://nuxtseo.com/docs/seo-utils/getting-started/introduction)
- [Site Config](https://nuxtseo.com/docs/site-config/getting-started/introduction)
- [Skew Protection](https://nuxtseo.com/docs/skew-protection/getting-started/introduction)
- [AI Ready](https://nuxtseo.com/docs/ai-ready/getting-started/introduction)

### [NuxtSEO Pro](https://nuxtseo.com/pro "Home")

- [Getting Started](https://nuxtseo.com/pro)
- [Dashboard](https://nuxtseo.com/pro/dashboard)
- [Pro MCP](https://nuxtseo.com/docs/nuxt-seo-pro/mcp/installation)

### [Learn SEO](https://nuxtseo.com/learn-seo "Learn SEO")

Nuxt

- [Mastering Meta](https://nuxtseo.com/learn-seo/nuxt/mastering-meta)
- [Controlling Crawlers](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers)
- [Launch & Listen](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen)
- [Routes & Rendering](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering)
- [Staying Secure](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/security)

Vue

- [Vue SEO Guide](https://nuxtseo.com/learn-seo/vue)
- [Mastering Meta](https://nuxtseo.com/learn-seo/vue/mastering-meta)
- [Controlling Crawlers](https://nuxtseo.com/learn-seo/vue/controlling-crawlers)
- [SPA SEO](https://nuxtseo.com/learn-seo/vue/spa)
- [SSR Frameworks](https://nuxtseo.com/learn-seo/vue/ssr-frameworks)
- [SEO Checklist](https://nuxtseo.com/learn-seo/checklist)
- [Pre-Launch Warmup](https://nuxtseo.com/learn-seo/pre-launch-warmup)
- [Backlinks & Authority](https://nuxtseo.com/learn-seo/backlinks)

### [Tools](https://nuxtseo.com/tools "SEO Tools")

- [Social Share Debugger](https://nuxtseo.com/tools/social-share-debugger)
- [Robots.txt Generator](https://nuxtseo.com/tools/robots-txt-generator)
- [Meta Tag Checker](https://nuxtseo.com/tools/meta-tag-checker)
- [HTML to Markdown](https://nuxtseo.com/tools/html-to-markdown)
- [XML Sitemap Validator](https://nuxtseo.com/tools/xml-sitemap-validator)
- [Schema.org Validator](https://nuxtseo.com/tools/schema-validator)
- [Keyword Research Pro](https://nuxtseo.com/tools/keyword-research)
- [SERP Analyzer Pro](https://nuxtseo.com/tools/serp-analyzer)
- [Domain Rankings Pro](https://nuxtseo.com/tools/domain-rankings)

Copyright © 2023-2026 Harlan Wilton - [MIT License](https://github.com/harlan-zw/nuxt-seo/blob/main/license) · [mdream](https://mdream.dev)