Protecting Nuxt Apps from Malicious Crawlers · Nuxt SEO

[NuxtSEO](https://nuxtseo.com/ "Home")

- [Modules](https://nuxtseo.com/docs/nuxt-seo/getting-started/introduction)
- [Tools](https://nuxtseo.com/tools)
- [Pro](https://nuxtseo.com/pro)
- [Learn SEO](https://nuxtseo.com/learn-seo/nuxt) [Releases](https://nuxtseo.com/releases)

[1.4K](https://github.com/harlan-zw/nuxt-seo)

[Nuxt SEO on GitHub](https://github.com/harlan-zw/nuxt-seo)

Learn SEO

Master search optimization

Nuxt

 Vue

[SEO Checklist](https://nuxtseo.com/learn-seo/checklist) [Pre-Launch Warmup](https://nuxtseo.com/learn-seo/pre-launch-warmup) [Backlinks & Authority](https://nuxtseo.com/learn-seo/backlinks)

[Mastering Meta](https://nuxtseo.com/learn-seo/nuxt/mastering-meta)

- [Titles](https://nuxtseo.com/learn-seo/nuxt/mastering-meta/titles)
- [Meta Description](https://nuxtseo.com/learn-seo/nuxt/mastering-meta/descriptions)
- [Image Alt Text](https://nuxtseo.com/learn-seo/nuxt/mastering-meta/alt-text)
- [Social Sharing](https://nuxtseo.com/learn-seo/nuxt/mastering-meta/open-graph)
- [Rich Results](https://nuxtseo.com/learn-seo/nuxt/mastering-meta/rich-results)
- [Schema.org](https://nuxtseo.com/learn-seo/nuxt/mastering-meta/schema-org)
- [Twitter Cards](https://nuxtseo.com/learn-seo/nuxt/mastering-meta/twitter-cards)

[ Controlling Crawlers](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers)

- [Robots Txt](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/robots-txt)
- [Sitemaps](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/sitemaps)
- [Robot Meta Tag](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/meta-tags)
- [Canonical Link Tag](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/canonical-urls)
- [HTTP Redirects](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/redirects)
- [Duplicate Content](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/duplicate-content)
- [llms.txt](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/llms-txt)

[ Routes & Rendering](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering)

- [URL Structure](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/url-structure)
- [Pagination](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/pagination)
- [Trailing Slashes](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/trailing-slashes)
- [Query Parameters](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/query-parameters)
- [Hreflang & i18n](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/i18n)
- [404 Pages](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/404-pages)
- [Dynamic Routes](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/dynamic-routes)
- [Internal Linking](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/internal-linking)
- [Programmatic SEO](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/programmatic-seo)
- [Rendering Modes](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/rendering)
- [Security](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/security)

[ Launch & Listen](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen)

- [Getting Indexed](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen/going-live)
- [Google Search Console](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen/search-console)
- [Core Web Vitals](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen/core-web-vitals)
- [Indexing Issues](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen/indexing-issues)
- [SEO Monitoring](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen/seo-monitoring)
- [Site Migration](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen/site-migration)
- [IndexNow](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen/indexnow)
- [Debugging](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen/debugging)
- [AI Search Optimization](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen/ai-optimized-content)

1. [Learn SEO for Nuxt](https://nuxtseo.com/learn-seo)
2.
3. [Routes And Rendering](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering)
4.
5. [Security](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/security)

# Protecting Nuxt Apps from Malicious Crawlers

Robots.txt is a polite suggestion. Malicious crawlers ignore it. Here's how to actually protect your Nuxt app.

[![Harlan Wilton](https://avatars.githubusercontent.com/u/5326365?v=4)Harlan Wilton](https://x.com/harlan-zw)8 mins read Published Nov 3, 2024 Updated Jan 4, 2026

What you'll learn

- Robots.txt is a suggestion, not security. use server middleware and route rules for actual protection
- Block non-production environments with `X-Robots-Tag: noindex` header via middleware
- Rate limiting and security headers belong in `routeRules`, not robots.txt

[Robots.txt](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/robots-txt) and meta robots tags are polite suggestions. Malicious crawlers ignore them.

You need actual security: block non-production environments, protect development assets, rate limit aggressive crawlers, authenticate sensitive routes, use HTTPS everywhere. Don't rely on robots.txt for sensitive data, IP blocking alone (easily bypassed), or user-agent detection (trivial to fake).

## [Quick Setup](#quick-setup)

The easiest way to handle crawler blocking in Nuxt is with the Robots module:

nuxt.config.ts

```
export default defineNuxtConfig({
  modules: ['@nuxtjs/robots'],
  robots: {
    // Automatically blocks non-production environments
    disallow: process.env.NODE_ENV !== 'production' ? ['*'] : [],
    // Block specific paths
    groups: [
      { userAgent: '*', disallow: ['/admin', '/dashboard'] }
    ]
  }
})
```

[Robots v6.0.68.8M516 Tame the robots crawling and indexing your site with ease.](https://nuxtseo.com/docs/robots/getting-started/introduction)

For additional security beyond crawler blocking, use server middleware and route rules:

server/middleware/security.ts

```
export default defineEventHandler((event) => {
  // Block non-production environments
  if (process.env.NODE_ENV !== 'production') {
    setHeader(event, 'X-Robots-Tag', 'noindex, nofollow')
  }

  // Enforce HTTPS
  const proto = getRequestHeader(event, 'x-forwarded-proto')
  if (proto === 'http') {
    return sendRedirect(event, \`https://${getRequestHost(event)}${event.path}\`, 301)
  }
})
```

nuxt.config.ts

```
export default defineNuxtConfig({
  nitro: {
    routeRules: {
      '/**': {
        headers: {
          'X-Frame-Options': 'DENY',
          'X-Content-Type-Options': 'nosniff',
          'Referrer-Policy': 'strict-origin-when-cross-origin'
        }
      }
    }
  }
})
```

For rate limiting, use a dedicated module like [nuxt-rate-limit](https://github.com/timb-103/nuxt-rate-limit) or [nuxt-api-shield](https://nuxt.com/modules/api-shield):

nuxt.config.ts

```
export default defineNuxtConfig({
  modules: ['nuxt-rate-limit'],
  nuxtRateLimit: {
    routes: {
      '/api/*': { maxRequests: 100, intervalSeconds: 60 }
    }
  }
})
```

Or implement manually:

server/middleware/rate-limit.ts

```
const requestCounts = new Map<string, number>()

export default defineEventHandler((event) => {
  const ip = getRequestIP(event)
  const count = requestCounts.get(ip) || 0

  if (count > 100) {
    throw createError({
      statusCode: 429,
      message: 'Too Many Requests'
    })
  }

  requestCounts.set(ip, count + 1)
})
```

## [Environment Protection](#environment-protection)

### [Development & Staging](#development-staging)

The [Robots module](https://nuxtseo.com/docs/robots/getting-started/introduction) automatically handles non-production blocking. It detects preview/staging environments and adds `noindex` headers:

nuxt.config.ts

```
export default defineNuxtConfig({
  modules: ['@nuxtjs/robots'],
  robots: {
    // Automatically detects non-production environments
    // Or manually control:
    blockNonSeoCrawlers: true,
    disallow: import.meta.env.VERCEL_ENV === 'preview' ? ['*'] : []
  }
})
```

For custom logic or basic auth, use middleware:

server/middleware/block-non-production.ts

```
export default defineEventHandler((event) => {
  const isProd = process.env.NODE_ENV === 'production'
  const isMainDomain = getRequestHost(event) === 'mysite.com'

  if (!isProd || !isMainDomain) {
    setHeader(event, 'X-Robots-Tag', 'noindex, nofollow')

    // Optional: basic auth for staging
    const auth = getRequestHeader(event, 'authorization')
    if (!auth) {
      setResponseStatus(event, 401)
      setHeader(event, 'WWW-Authenticate', 'Basic')
      return 'Authentication required'
    }
  }
})
```

### [Sensitive Routes](#sensitive-routes)

Use the [Robots module](https://nuxtseo.com/docs/robots/getting-started/introduction) to block indexing of sensitive paths:

nuxt.config.ts

```
export default defineNuxtConfig({
  modules: ['@nuxtjs/robots'],
  robots: {
    disallow: ['/admin', '/dashboard', '/user']
  }
})
```

For authentication and custom protection logic, add middleware:

server/middleware/protect-routes.ts

```
export default defineEventHandler((event) => {
  const protectedPaths = ['/admin', '/dashboard', '/user']

  if (protectedPaths.some(path => event.path.startsWith(path))) {
    // Ensure user is authenticated
    if (!event.context.auth?.user) {
      return sendRedirect(event, '/login')
    }

    // Block indexing of protected content
    setHeader(event, 'X-Robots-Tag', 'noindex, nofollow')
  }
})
```

Or use route rules for static protection:

nuxt.config.ts

```
export default defineNuxtConfig({
  nitro: {
    routeRules: {
      '/admin/**': {
        headers: {
          'X-Robots-Tag': 'noindex, nofollow'
        }
      }
    }
  }
})
```

## [Crawler Identification](#crawler-identification)

### [Good vs Bad Crawlers](#good-vs-bad-crawlers)

Identify legitimate crawlers through:

- Reverse DNS lookup
- IP verification
- Behavior patterns
- Request rate

server/utils/verify-crawler.ts

```
import { lookup } from 'node:dns/promises'

export async function isLegitCrawler(ip: string, userAgent: string) {
  // Example: Verify Googlebot
  if (userAgent.includes('Googlebot')) {
    const [hostname] = await lookup(ip)
    return hostname.endsWith('googlebot.com')
  }
  return false
}
```

### [Rate Limiting](#rate-limiting)

Use [nuxt-security](https://nuxt-security.vercel.app/middleware/rate-limiter) for built-in rate limiting:

nuxt.config.ts

```
export default defineNuxtConfig({
  modules: ['nuxt-security'],
  security: {
    rateLimiter: {
      tokensPerInterval: 100,
      interval: 60000, // 1 minute
      headers: true
    }
  }
})
```

Or use [nuxt-rate-limit](https://github.com/timb-103/nuxt-rate-limit) for simpler API-focused rate limiting:

nuxt.config.ts

```
export default defineNuxtConfig({
  modules: ['nuxt-rate-limit'],
  nuxtRateLimit: {
    routes: {
      '/api/*': { maxRequests: 100, intervalSeconds: 60 },
      '/api/auth/*': { maxRequests: 10, intervalSeconds: 60 }
    }
  }
})
```

For custom tiered logic, implement manually:

server/middleware/rate-limit.ts

```
const requestCounts = new Map<string, { count: number, resetAt: number }>()

export default defineEventHandler((event) => {
  const ip = getRequestIP(event)
  const now = Date.now()
  const windowMs = 15 * 60 * 1000 // 15 minutes

  const record = requestCounts.get(ip)

  if (!record || now > record.resetAt) {
    requestCounts.set(ip, { count: 1, resetAt: now + windowMs })
    return
  }

  // Different limits for different paths
  const maxRequests = event.path.startsWith('/api') ? 100 : 1000

  if (record.count > maxRequests) {
    throw createError({
      statusCode: 429,
      message: 'Too Many Requests'
    })
  }

  record.count++
})
```

## [Infrastructure Security](#infrastructure-security)

### [HTTPS Enforcement](#https-enforcement)

Nuxt handles HTTPS redirects via middleware:

server/middleware/https.ts

```
export default defineEventHandler((event) => {
  const proto = getRequestHeader(event, 'x-forwarded-proto')

  if (proto === 'http') {
    return sendRedirect(
      event,
      \`https://${getRequestHost(event)}${event.path}\`,
      301
    )
  }
})
```

### [Security Headers](#security-headers)

Use [nuxt-security](https://nuxt-security.vercel.app/) to automatically configure security headers following OWASP best practices:

nuxt.config.ts

```
export default defineNuxtConfig({
  modules: ['nuxt-security'],
  security: {
    headers: {
      crossOriginResourcePolicy: 'same-origin',
      crossOriginOpenerPolicy: 'same-origin',
      xContentTypeOptions: 'nosniff',
      referrerPolicy: 'strict-origin-when-cross-origin',
      contentSecurityPolicy: {
        'default-src': ['\'self\''],
        'script-src': ['\'self\'', '\'unsafe-inline\'']
      }
    }
  }
})
```

Or configure manually via route rules:

nuxt.config.ts

```
export default defineNuxtConfig({
  nitro: {
    routeRules: {
      '/**': {
        headers: {
          'X-Frame-Options': 'DENY',
          'X-Content-Type-Options': 'nosniff',
          'Referrer-Policy': 'strict-origin-when-cross-origin',
          ...(process.env.NODE_ENV === 'production' && {
            'Content-Security-Policy': 'default-src \'self\';'
          })
        }
      }
    }
  }
})
```

## [Monitoring & Detection](#monitoring-detection)

### [Logging Suspicious Activity](#logging-suspicious-activity)

server/middleware/crawler-monitor.ts

```
export default defineEventHandler((event) => {
  const ua = getRequestHeader(event, 'user-agent')
  const ip = getRequestIP(event)

  // Log suspicious patterns
  if (isSuspiciousPattern(ua, ip)) {
    console.warn(\`Suspicious crawler: ${ip} with UA: ${ua}\`)
    // Consider blocking or rate limiting
  }
})
```

### [Using Web Application Firewalls](#using-web-application-firewalls)

Services like [Cloudflare](https://cloudflare.com) or AWS WAF can:

- Block malicious IPs
- Prevent DDoS attacks
- Filter suspicious requests
- Monitor traffic patterns

**Opinion:** If you're running a small blog, a WAF is overkill. Add it when you're getting attacked.

## [Common Attacks](#common-attacks)

### [Content Scraping](#content-scraping)

Use [nuxt-security](https://nuxt-security.vercel.app/middleware/rate-limiter) rate limiting to prevent automated scraping:

nuxt.config.ts

```
export default defineNuxtConfig({
  modules: ['nuxt-security'],
  security: {
    rateLimiter: {
      tokensPerInterval: 50,
      interval: 60000
    }
  }
})
```

For more control over bot detection and delays:

server/middleware/anti-scraping.ts

```
const requestCounts = new Map<string, number>()

export default defineEventHandler(async (event) => {
  const ip = getRequestIP(event)
  const count = requestCounts.get(ip) || 0

  if (count > 100) {
    throw createError({
      statusCode: 429,
      message: 'Too Many Requests'
    })
  }

  requestCounts.set(ip, count + 1)

  // Add slight delays to automated requests
  const ua = getRequestHeader(event, 'user-agent')
  if (isBot(ua)) {
    await new Promise(r => setTimeout(r, 500))
  }
})
```

### [Form Spam](#form-spam)

Use [nuxt-security](https://nuxt-security.vercel.app/) for XSS validation and request limiting:

nuxt.config.ts

```
export default defineNuxtConfig({
  modules: ['nuxt-security'],
  security: {
    xssValidator: true,
    rateLimiter: {
      tokensPerInterval: 5,
      interval: 60000
    }
  }
})
```

For honeypot fields and custom validation:

server/api/contact.post.ts

```
const submissionCounts = new Map<string, number>()

export default defineEventHandler(async (event) => {
  const body = await readBody(event)
  const ip = getRequestIP(event)

  // Honeypot check
  if (body.website) { // hidden field
    return { success: false }
  }

  // Rate limiting
  const count = submissionCounts.get(ip) || 0
  if (count > 5) {
    throw createError({
      statusCode: 429,
      message: 'Too many attempts'
    })
  }

  submissionCounts.set(ip, count + 1)

  // Process legitimate submission
  // ...
})
```

[The 2026 SEO Checklist for Nuxt & Vue Pre-launch setup, post-launch verification, and ongoing monitoring. Interactive checklist with links to every guide.](https://nuxtseo.com/learn-seo/checklist) [Haven't launched yet? Start with the Pre-Launch Warmup](https://nuxtseo.com/learn-seo/pre-launch-warmup)

---

[Rendering Modes How to use SSR, SSG, Nuxt Islands, and Edge-side rendering to optimize for indexing and INP.](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/rendering) [Launch & Listen Submit sitemap to Search Console, verify indexing, monitor rankings and AI visibility. Complete post-launch SEO workflow for Nuxt in 2026.](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen)

On this page

- [Quick Setup](#quick-setup)
- [Environment Protection](#environment-protection)
- [Crawler Identification](#crawler-identification)
- [Infrastructure Security](#infrastructure-security)
- [Monitoring & Detection](#monitoring-detection)
- [Common Attacks](#common-attacks)

[GitHub](https://github.com/harlan-zw/nuxt-seo) [ Discord](https://discord.com/invite/275MBUBvgP)

### [NuxtSEO](https://nuxtseo.com/ "Home")

- [Getting Started](https://nuxtseo.com/docs/nuxt-seo/getting-started/introduction)
- [MCP](https://nuxtseo.com/docs/nuxt-seo/guides/mcp)

Modules

- [Robots](https://nuxtseo.com/docs/robots/getting-started/introduction)
- [Sitemap](https://nuxtseo.com/docs/sitemap/getting-started/introduction)
- [OG Image](https://nuxtseo.com/docs/og-image/getting-started/introduction)
- [Schema.org](https://nuxtseo.com/docs/schema-org/getting-started/introduction)
- [Link Checker](https://nuxtseo.com/docs/link-checker/getting-started/introduction)
- [SEO Utils](https://nuxtseo.com/docs/seo-utils/getting-started/introduction)
- [Site Config](https://nuxtseo.com/docs/site-config/getting-started/introduction)
- [Skew Protection](https://nuxtseo.com/docs/skew-protection/getting-started/introduction)
- [AI Ready](https://nuxtseo.com/docs/ai-ready/getting-started/introduction)

### [NuxtSEO Pro](https://nuxtseo.com/pro "Home")

- [Getting Started](https://nuxtseo.com/pro)
- [Dashboard](https://nuxtseo.com/pro/dashboard)
- [Pro MCP](https://nuxtseo.com/docs/nuxt-seo-pro/mcp/installation)

### [Learn SEO](https://nuxtseo.com/learn-seo "Learn SEO")

Nuxt

- [Mastering Meta](https://nuxtseo.com/learn-seo/nuxt/mastering-meta)
- [Controlling Crawlers](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers)
- [Launch & Listen](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen)
- [Routes & Rendering](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering)
- [Staying Secure](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/security)

Vue

- [Vue SEO Guide](https://nuxtseo.com/learn-seo/vue)
- [Mastering Meta](https://nuxtseo.com/learn-seo/vue/mastering-meta)
- [Controlling Crawlers](https://nuxtseo.com/learn-seo/vue/controlling-crawlers)
- [SPA SEO](https://nuxtseo.com/learn-seo/vue/spa)
- [SSR Frameworks](https://nuxtseo.com/learn-seo/vue/ssr-frameworks)
- [SEO Checklist](https://nuxtseo.com/learn-seo/checklist)
- [Pre-Launch Warmup](https://nuxtseo.com/learn-seo/pre-launch-warmup)
- [Backlinks & Authority](https://nuxtseo.com/learn-seo/backlinks)

### [Tools](https://nuxtseo.com/tools "SEO Tools")

- [Social Share Debugger](https://nuxtseo.com/tools/social-share-debugger)
- [Robots.txt Generator](https://nuxtseo.com/tools/robots-txt-generator)
- [Meta Tag Checker](https://nuxtseo.com/tools/meta-tag-checker)
- [HTML to Markdown](https://nuxtseo.com/tools/html-to-markdown)
- [XML Sitemap Validator](https://nuxtseo.com/tools/xml-sitemap-validator)
- [Schema.org Validator](https://nuxtseo.com/tools/schema-validator)
- [Keyword Research Pro](https://nuxtseo.com/tools/keyword-research)
- [SERP Analyzer Pro](https://nuxtseo.com/tools/serp-analyzer)
- [Domain Rankings Pro](https://nuxtseo.com/tools/domain-rankings)

Copyright © 2023-2026 Harlan Wilton - [MIT License](https://github.com/harlan-zw/nuxt-seo/blob/main/license) · [mdream](https://mdream.dev)