# SEO-AI-crawlers (m14) Human Guide

## What This Is For
Audit AI crawler access and citability for a page — confirm retrieval/citation bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are allowed and the Googlebot vs Google-Extended split. It gives the agent a clearer input/output frame for search and SEO workflows: what context to ask for, what decisions to make, and what usable artifact to return.

Use this as a human-readable version of the SEO-AI-crawlers (m14) agent skill. It is meant for marketers, operators, founders, and other non-coders who want the workflow without reading agent-specific implementation instructions.

## When To Use This
- Use this when you need a repeatable process for SEO-AI-crawlers (m14).
- Use this when the task needs judgment, examples, constraints, or a clear output format rather than a one-off prompt.
- Use this when you want to hand an AI assistant enough context to produce a usable marketing artifact.

## When Not To Use This
- Do not use this when you only need a quick factual answer.
- Do not use this when the work depends on private data you cannot share with the assistant.
- Do not use this as a replacement for legal, compliance, financial, or medical review.

## What You Need Before Starting
- The goal or business outcome you want.
- The audience, customer segment, or market context.
- Any source material the assistant should respect, such as notes, briefs, examples, URLs, or brand guidance.
- Constraints such as tone, length, channel, deadline, region, or approval requirements.
- A clear definition of what a good final answer should look like.

## Step-By-Step Workflow
1. State the job clearly: "Use the SEO-AI-crawlers (m14) guide to help me with..."
2. Add context: audience, goal, offer, channel, source material, and constraints.
3. Ask the assistant to identify missing inputs before producing the final output.
4. Have the assistant follow the skill-specific guidance below.
5. Review the result against the final checklist and ask for revisions where needed.

## Skill-Specific Guidance
- **llms.txt / llms-full.txt** (also covers M21): presence at the site root, valid Markdown structure (H1 title, summary blockquote, sectioned link lists), and that linked URLs resolve. Follow `references/ai-crawlers.md`.
- **ADVISORY** (`fixable: advisory`): edge/WAF block for bots that ignore `robots.txt` (e.g. Bytespider) — the tool never writes infra config.
- `node ${CLAUDE_SKILL_DIR}/../../scripts/parse-robots-sitemap.mjs --url <u>` (`robots_parse`) — parses `robots.txt`, resolves the effective directive for each AI user-agent, and confirms it.
- `dom_assert` against the M4 render result for the renderability check (content present in `raw_html`).
- When the required data tier is unavailable (e.g. `robots.txt` unfetchable, no M4 render result), status is `needs_api` — **never** a false `pass`.
- `M14.citation_bots.blocked` — `Disallow: /` reaches `OAI-SearchBot`/`Claude-SearchBot`/`PerplexityBot` (status `fail`, severity 4, `fixable: auto`, axis `ai`, confidence `established`).
- `M14.render.content_js_only` — primary content in `rendered_dom` but absent from `raw_html`, invisible to non-JS AI crawlers (status `warn`, severity 4, `fixable: advisory`, axis `ai`, confidence `directional`).
- `M14.llmstxt.missing` — no `/llms.txt` at site root (status `warn`, severity 1, `fixable: auto`, axis `ai`, confidence `speculative`, scored 0).
- Blocking a **training** bot does NOT block the matching **search** bot — they are separate user-agents (`GPTBot` ≠ `OAI-SearchBot`, `ClaudeBot` ≠ `Claude-SearchBot`). Many "block AI" guides get this wrong.
- `Bytespider` and some agents frequently ignore `robots.txt`; a robots rule is best-effort. Real enforcement needs an edge rule / WAF (advisory).
- `robots.txt` controls **crawling**, not **indexing** — to keep a page out, use a `noindex` meta tag and don't also `Disallow` it.
- Blocking a **training** bot does NOT block the matching **search** bot — they are separate user-agents. Many "block AI" guides get this wrong.

## Decision Points And Nuance
The original skill emphasizes: Audits, Fixes, Verification, Findings, Honesty, Supporting file: references/ai-crawlers.md, User-agent reference, Presets the fixer can generate (choice-gated, opt-in), Honesty notes, Supporting file: schema/finding.schema.json.

Use these questions to steer the work:
- What is the intended audience or buyer?
- What source material must be preserved?
- What should the assistant optimize for: clarity, persuasion, accuracy, speed, creativity, or conversion?
- What examples represent the desired quality bar?
- What should the assistant avoid?

## Common Mistakes
- **ADVISORY** (`fixable: advisory`): edge/WAF block for bots that ignore `robots.txt` (e.g. Bytespider) — the tool never writes infra config.
- **Never fabricate** sitemap URLs, contact emails, or link targets — ask the user or leave a clearly-marked `TODO` placeholder.
- When the required data tier is unavailable (e.g. `robots.txt` unfetchable, no M4 render result), status is `needs_api` — **never** a false `pass`.
- `robots.txt` controls **crawling**, not **indexing** — to keep a page out, use a `noindex` meta tag and don't also `Disallow` it.
- | `Googlebot` | Google | Classic search index (also feeds AI Overviews) | Don't block |
- | `Bingbot` | Microsoft | Bing search (feeds Copilot) | Don't block |
- Search engines — required for ranking
- `robots.txt` controls **crawling**, not **indexing**. To keep a page out of an index use a `noindex` robots meta tag (and don't also `Disallow` it, or the crawler can't see the `noindex`).

## Copy-And-Paste Prompt
```text
Use the SEO-AI-crawlers (m14) human guide.

My goal:
[Describe the business outcome]

Audience:
[Describe who this is for]

Context and source material:
[Paste notes, examples, links, or existing copy]

Constraints:
[Tone, length, channel, timeline, must-include items, must-avoid items]

Before producing the final output, ask me for any missing information that would materially improve the result.
```

## Final Checklist
- [ ] The output matches the original goal.
- [ ] The audience and context are reflected in the answer.
- [ ] Important constraints and source material were preserved.
- [ ] The assistant made the relevant decisions explicit.
- [ ] The final artifact is ready to use, review, or hand to the next person.

## Source
This guide was generated from the hainrixz/claude-seo-ai skill entry for `seo-ai-crawlers`.

## Source Skill Notes
These notes preserve the nuance from the original skill. Use them as supporting reference when the workflow above feels too generic.

# seo-ai-crawlers (M14)

Controls whether AI search engines can crawl and cite the page, and whether they can read it without JS. The training-vs-search-vs-fetch distinction is everything. Reference: `references/ai-crawlers.md`.

## Audits
Working from the PageSnapshot (`rendered_dom` if present, else `raw_html`) plus the site `robots.txt`:
1. **Citation access**: are retrieval/citation bots — `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `Bingbot` — actually allowed (not caught by a broad `Disallow: /` or a wildcard block)? Confirm `Googlebot` is not blocked and the `Googlebot` (search) vs `Google-Extended` (Gemini training control) split is correct.
2. **User-agent classification**: bucket every AI agent in `robots.txt` into **training** (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot), **search/retrieval** (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and **user-triggered fetch** (ChatGPT-User, Claude-User, Perplexity-User). Match user-agents case-insensitively; treat the table in `references/ai-crawlers.md` as a starting set, not exhaustive.
3. **Renderability for non-JS crawlers**: pull the M4 (seo-crawl-render) render result — most AI crawlers do not execute JS. If primary content only appears in `rendered_dom` and is absent from `raw_html`, flag it as invisible to AI retrieval.
4. **llms.txt / llms-full.txt** (also covers M21): presence at the site root, valid Markdown structure (H1 title, summary blockquote, sectioned link lists), and that linked URLs resolve. Follow `references/ai-crawlers.md`.

## Fixes
- **AUTO** (`fixable: auto`): a citation-friendly `robots.txt` preset, **choice-gated** — the user picks `allow-citations` (default: allow search/retrieval, opt out of training), `allow-all`, or `block-all`. Deterministic, additive, verifiable; emitted as a diff for `fix`.
- **AUTO** (`fixable: auto`), **disclosure-gated** and **scored 0**: `llms.txt` / `llms-full.txt`, generated from the site's own structure only on explicit request (`fix --category llms`), shown as a diff before writing. Additive and deterministic, but never sold as proven ranking value — the disclosure that it is low/uncertain impact is shown every time.
- **ADVISORY** (`fixable: advisory`): edge/WAF block for bots that ignore `robots.txt` (e.g. Bytespider) — the tool never writes infra config.
**Never fabricate** sitemap URLs, contact emails, or link targets — ask the user or leave a clearly-marked `TODO` placeholder.

## Verification
- `node ${CLAUDE_SKILL_DIR}/../../scripts/parse-robots-sitemap.mjs --url <u>` (`robots_parse`) — parses `robots.txt`, resolves the effective directive for each AI user-agent, and confirms it.
- `dom_assert` against the M4 render result for the renderability check (content present in `raw_html`).
- When the required data tier is unavailable (e.g. `robots.txt` unfetchable, no M4 render result), status is `needs_api` — **never** a false `pass`.

## Findings
Findings conform to `schema/finding.schema.json`. `evidence.observed` quotes the page/robots line; `verification.reproduce` is the runnable command above; `expected_impact` is banded + confidence-tagged (no naked %). Examples:
- `M14.citation_bots.blocked` — `Disallow: /` reaches `OAI-SearchBot`/`Claude-SearchBot`/`PerplexityBot` (status `fail`, severity 4, `fixable: auto`, axis `ai`, confidence `established`).
- `M14.render.content_js_only` — primary content in `rendered_dom` but absent from `raw_html`, invisible to non-JS AI crawlers (status `warn`, severity 4, `fixable: advisory`, axis `ai`, confidence `directional`).
- `M14.llmstxt.missing` — no `/llms.txt` at site root (status `warn`, severity 1, `fixable: auto`, axis `ai`, confidence `speculative`, scored 0).

## Honesty
- Blocking a **training** bot does NOT block the matching **search** bot — they are separate user-agents (`GPTBot` ≠ `OAI-SearchBot`, `ClaudeBot` ≠ `Claude-SearchBot`). Many "block AI" guides get this wrong.
- `Bytespider` and some agents frequently ignore `robots.txt`; a robots rule is best-effort. Real enforcement needs an edge rule / WAF (advisory).
- `robots.txt` controls **crawling**, not **indexing** — to keep a page out, use a `noindex` meta tag and don't also `Disallow` it.
- `llms.txt` has only **partial** vendor support (Anthropic and Perplexity honor it in retrieval; Google does not use it for AI Overviews/AI Mode; OpenAI uncommitted) and is also useful as IDE/coding-agent context. Still low/uncertain — scored 0, never presented as proven ranking value.

---

## Supporting file: references/ai-crawlers.md

# AI crawlers & robots.txt control (2026)

The key distinction: **training bots** (used to train models) vs **search/retrieval bots** (fetch pages to answer live queries and may cite you) vs **user-triggered fetchers** (fetch a page because a user pasted/asked about it). You can allow citations while controlling training — but only if you target the right user-agents.

## User-agent reference

| User-agent | Operator | Purpose | Controllable via robots.txt |
|---|---|---|---|
| `Googlebot` | Google | Classic search index (also feeds AI Overviews) | Don't block |
| `Google-Extended` | Google | Gemini / Vertex training control | Yes |
| `Bingbot` | Microsoft | Bing search (feeds Copilot) | Don't block |
| `GPTBot` | OpenAI | Model **training** | Yes |
| `OAI-SearchBot` | OpenAI | ChatGPT **search/retrieval** (citations) | Yes — allow for citations |
| `ChatGPT-User` | OpenAI | **User-triggered** fetch | Yes (limited effect) |
| `ClaudeBot` | Anthropic | Model **training** | Yes |
| `Claude-SearchBot` | Anthropic | Claude **search/retrieval** (citations) | Yes — allow for citations |
| `Claude-User` / `Claude-Web` | Anthropic | User-triggered fetch | Yes |
| `PerplexityBot` | Perplexity | Crawl for retrieval (citations) | Yes — allow for citations |
| `Perplexity-User` | Perplexity | User-triggered fetch | Yes |
| `Applebot-Extended` | Apple | Apple Intelligence training control | Yes |
| `Meta-ExternalAgent` | Meta | Training | Yes |
| `Amazonbot` | Amazon | Indexing | Yes |
| `CCBot` | Common Crawl | Public dataset (feeds many models) | Yes |
| `Bytespider` | ByteDance | Training + search; **often ignores robots.txt** | Unreliable — recommend edge/WAF block |

User-agent strings drift; match case-insensitively and treat the table as a starting set. Verify a bot's current docs before making strong claims to a user.

## Presets the fixer can generate (choice-gated, opt-in)

**1. Allow citations, control training (recommended default for most sites):**
```
# Search engines — required for ranking
User-agent: Googlebot
User-agent: Bingbot
Disallow:

# AI search/retrieval — allow so engines can cite you
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: