# Technical SEO audit Human Guide

## What This Is For
Technical SEO audit across 9 categories: crawlability, indexability, security, URL structure, mobile, Core Web Vitals, structured data, JavaScript rendering, and IndexNow protocol. It gives the agent a clearer input/output frame for search and SEO workflows: what context to ask for, what decisions to make, and what usable artifact to return.

Use this as a human-readable version of the Technical SEO audit agent skill. It is meant for marketers, operators, founders, and other non-coders who want the workflow without reading agent-specific implementation instructions.

## When To Use This
- Use this when you need a repeatable process for technical SEO audit.
- Use this when the task needs judgment, examples, constraints, or a clear output format rather than a one-off prompt.
- Use this when you want to hand an AI assistant enough context to produce a usable marketing artifact.

## When Not To Use This
- Do not use this when you only need a quick factual answer.
- Do not use this when the work depends on private data you cannot share with the assistant.
- Do not use this as a replacement for legal, compliance, financial, or medical review.

## What You Need Before Starting
- The goal or business outcome you want.
- The audience, customer segment, or market context.
- Any source material the assistant should respect, such as notes, briefs, examples, URLs, or brand guidance.
- Constraints such as tone, length, channel, deadline, region, or approval requirements.
- A clear definition of what a good final answer should look like.

## Step-By-Step Workflow
1. State the job clearly: "Use the Technical SEO audit guide to help me with..."
2. Add context: audience, goal, offer, channel, source material, and constraints.
3. Ask the assistant to identify missing inputs before producing the final output.
4. Have the assistant follow the skill-specific guidance below.
5. Review the result against the final checklist and ask for revisions where needed.

## Skill-Specific Guidance
- robots.txt: exists, valid, not blocking important resources
- XML sitemap: run `claude-seo run sitemap_discovery.py <url> --json`; require a
- Noindex tags: intentional vs accidental
- Crawl depth: important pages within 3 clicks of homepage
- JavaScript rendering: check if critical content requires JS execution
- Crawl budget: for large sites (>10k pages), efficiency matters
- Google's canonical crawling/robots reference moved to **developers.google.com/crawling** (migrated 2025-11-20); IP-range files relocated to `/crawling/ipranges/` and `googlebot.json` was renamed `common-crawlers.json`.
- Blocking `Google-Extended` prevents Gemini training use but does NOT affect Google Search indexing or AI Overviews (those use `Googlebot`)
- Blocking `GPTBot` prevents OpenAI training but does NOT prevent ChatGPT from citing your content via browsing (`ChatGPT-User`)
- ~3-5% of websites now use AI-specific robots.txt rules
- Canonical tags: self-referencing, no conflicts with noindex
- Duplicate content: near-duplicates, parameter URLs, www vs non-www

## Decision Points And Nuance
The original skill emphasizes: Categories, Crawlability, AI Crawler Management, Indexability, Security, URL Structure, Mobile Optimization & Page Experience, Core Web Vitals, Structured Data, JavaScript Rendering.

Use these questions to steer the work:
- What is the intended audience or buyer?
- What source material must be preserved?
- What should the assistant optimize for: clarity, persuasion, accuracy, speed, creativity, or conversion?
- What examples represent the desired quality bar?
- What should the assistant avoid?

## Common Mistakes
- robots.txt: exists, valid, not blocking important resources
- Crawl depth: important pages within 3 clicks of homepage
- INP replaced FID on March 12, 2024. FID was removed from Chrome's field-data tools (CrUX API, PageSpeed Insights) on September 9, 2024 (Lighthouse is a lab tool that never reported FID). Do NOT reference FID anywhere.
- Surface findings as **opportunities**, not failures; don't gate audits on a
- Every form input must have an associated label:
- 4. Don't cover interactive nodes with transparent overlays
- a hint that an element is actionable. Do not override it to `cursor: default`
- Inverse: do not apply `cursor: pointer` to non-interactive elements — that

## Copy-And-Paste Prompt
```text
Use the Technical SEO audit human guide.

My goal:
[Describe the business outcome]

Audience:
[Describe who this is for]

Context and source material:
[Paste notes, examples, links, or existing copy]

Constraints:
[Tone, length, channel, timeline, must-include items, must-avoid items]

Before producing the final output, ask me for any missing information that would materially improve the result.
```

## Final Checklist
- [ ] The output matches the original goal.
- [ ] The audience and context are reflected in the answer.
- [ ] Important constraints and source material were preserved.
- [ ] The assistant made the relevant decisions explicit.
- [ ] The final artifact is ready to use, review, or hand to the next person.

## Source
This guide was generated from the agricidaniel/claude-seo skill entry for `seo-technical`.

## Source Skill Notes
These notes preserve the nuance from the original skill. Use them as supporting reference when the workflow above feels too generic.

# Technical SEO Audit

## Categories

### 1. Crawlability
- robots.txt: exists, valid, not blocking important resources
- XML sitemap: run `claude-seo run sitemap_discovery.py <url> --json`; require a
  valid entry in `found`, and report stale or unsafe robots.txt declarations
  separately from working fallback locations
- Noindex tags: intentional vs accidental
- Crawl depth: important pages within 3 clicks of homepage
- JavaScript rendering: check if critical content requires JS execution
- Crawl budget: for large sites (>10k pages), efficiency matters
- Googlebot **fetch limits**: Googlebot fetches the first **2MB of HTML** and first **64MB of a PDF** (uncompressed; 15MB is the broader crawler-infra default). Long-standing, not a 2026 change, but inline base64 images, oversized inline CSS/JS, or bloated nav can push critical content/JSON-LD past the cap and out of the index. Keep key content + structured data within the first 2MB.
- Crawl rate **auto-adjusts** (backs off on 5xx/slow responses); there is **no manual crawl-rate control** (the legacy Search Console setting was removed Jan 2024). Influence crawling via sitemaps, server responsiveness, and robots controls.
- Google's canonical crawling/robots reference moved to **developers.google.com/crawling** (migrated 2025-11-20); IP-range files relocated to `/crawling/ipranges/` and `googlebot.json` was renamed `common-crawlers.json`.

#### AI Crawler Management

As of 2025-2026, AI companies actively crawl the web to train models and power AI search. Managing these crawlers via robots.txt is a critical technical SEO consideration.

**Known AI crawlers:**

| Crawler | Company | robots.txt token | Purpose |
|---------|---------|-----------------|---------|
| GPTBot | OpenAI | `GPTBot` | Model training |
| ChatGPT-User | OpenAI | `ChatGPT-User` | Real-time browsing |
| ClaudeBot | Anthropic | `ClaudeBot` | Model training |
| PerplexityBot | Perplexity | `PerplexityBot` | Search index + training |
| Bytespider | ByteDance | `Bytespider` | Model training |
| Google-Extended | Google | `Google-Extended` | Gemini training (NOT search) |
| CCBot | Common Crawl | `CCBot` | Open dataset |

**Key distinctions:**
- Blocking `Google-Extended` prevents Gemini training use but does NOT affect Google Search indexing or AI Overviews (those use `Googlebot`)
- Blocking `GPTBot` prevents OpenAI training but does NOT prevent ChatGPT from citing your content via browsing (`ChatGPT-User`)
- ~3-5% of websites now use AI-specific robots.txt rules

**Example, selective AI crawler blocking:**
```
# Allow search indexing, block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

# Allow all other crawlers (including Googlebot for search)
User-agent: *
Allow: /
```

**Recommendation:** Consider your AI visibility strategy before blocking. Being cited by AI systems drives brand awareness and referral traffic. Cross-reference the `seo-geo` skill for the full AI crawler/fetcher taxonomy.

> **User-triggered fetchers ignore robots.txt by design.** Google now documents **Google-Agent** (Project Mariner, agentic browsing) plus **Google-NotebookLM** and **Google Messages** as *user-triggered* fetchers that **cannot be blocked via robots.txt**. Use server-side access controls instead. By contrast, `Google-Extended` and `Google-CloudVertexBot` obey robots.txt. Emerging: **Web Bot Auth** (RFC 9421) lets bots authenticate cryptographically via a `Signature-Agent` header + key directory at `agent.bot.goog` (used by Google-Agent); reverse-DNS verification remains the fallback.

### 2. Indexability
- Canonical tags: self-referencing, no conflicts with noindex
- Duplicate content: near-duplicates, parameter URLs, www vs non-www
- Thin content: pages below minimum word counts per type
- Pagination: rel=next/prev or load-more pattern
- Hreflang: correct for multi-language/multi-region sites
- Index bloat: unnecessary pages consuming crawl budget

### 3. Security
- HTTPS: enforced, valid SSL certificate, no mixed content
- Security headers:
  - Content-Security-Policy (CSP)
  - Strict-Transport-Security (HSTS)
  - X-Frame-Options
  - X-Content-Type-Options
  - Referrer-Policy
- HSTS preload: check preload list inclusion for high-security sites
- **Back-button hijacking** (spam-policy violation, malicious practices): flag pages that defeat the Back button via `history.pushState`/`replaceState` (including scripts injected by third-party ad/library platforms). Added to Google's spam policies 2026-04-13; **enforcement live since 2026-06-15** (manual actions + automated demotions): treat as Critical.

### 4. URL Structure
- Clean URLs: descriptive, hyphenated, no query parameters for content