llms.txt & AI Crawlers

Summary: Publish a curated map of your site for AI answer engines, decide which AI crawlers may read it, and track the content aging out of citation eligibility.

llms.txt & AI Crawlers is a feature in Antimanual that gives you three distinct controls over how AI systems interact with your WordPress site:

1. llms.txt generator — Publishes a curated Markdown index of your best pages at your site root (/llms.txt) so AI answer engines know which of your pages are worth reading.
2. AI Crawler Manager — Lets you set per-bot allow or block rules for known AI crawlers, applied through WordPress’s virtual robots.txt.
3. Citation Risk queue — Scores your published content by how likely an answer engine is to skip it, surfacing the pages that need attention most urgently.

The feature is available in Antimanual Free. One action inside the Citation Risk tab — Schedule refreshes — requires Antimanual Pro with the Auto-Update module enabled.

---

How to Open llms.txt & AI Crawlers

1. In your WordPress admin, go to the Antimanual menu in the left sidebar.
2. Click llms.txt & AI Crawlers.

The page opens with a summary header showing four live stats:

Stat	What it shows
llms.txt	Off, Needs setup, or Live
Pages listed	Number of entries in the current document
Crawlers blocked (or Crawlers (Off))	Number of explicitly blocked bots, or Off if crawler management is disabled
Stale pages	Number of pages past the stale age threshold

A Verify Setup button in the header runs a live loopback check against your llms.txt, llms-full.txt, and robots.txt endpoints and shows an AI Endpoint Verification modal with the result of each check (ok, warning, or error).

The page is organised into three tabs:

• llms.txt
• AI Crawlers
• Citation Risk

---

Tab 1: llms.txt

This tab controls what gets published as /llms.txt and (optionally) /llms-full.txt, shows a live preview, and lets you write optional static files to your web root.

Publishing section

Setting	Description
Serve /llms.txt	Master toggle. When on, Antimanual serves a dynamically generated Markdown document at /llms.txt. Nothing is written to disk unless you use the Static file section below.
Also serve /llms-full.txt	When on (and Serve /llms.txt is also on), a second document is served at /llms-full.txt. This document includes the full text of each listed page so an answer engine can answer without fetching every URL.
Site name	Used as the heading in the document. Leave blank to use your WordPress site title.
One-line summary	Shown as a blockquote under the heading. Leave blank to use your site tagline.
Notes	Optional Markdown text shown before the link sections — for example, licensing terms, contact details, or citation instructions.

Note: If Serve /llms.txt is off, a banner is shown with a Turn it on button that enables and saves the setting in one click.

Note: If your site uses plain (non-pretty) permalinks, a warning is shown. With plain permalinks, your web server may answer /llms.txt itself before WordPress can. Switch to a pretty permalink structure, or use the Static file section to write the file to disk directly.

---

What to include section

Setting	Description
Post types	The post types drawn from to build the document. Only published, password-free content is listed.
Secondary post types	Post types listed under an ## Optional heading — the llms.txt convention for links an agent may skip when it is short on context. Must be a subset of the selected primary post types.
Group sections by	How entries are grouped: Post type, Category, or One list (no grouping).
Order entries by	Sort order within each section: Last updated, Published date, Title, or Menu order.
Max entries in llms.txt	Maximum number of entries in /llms.txt. Range: 1–500. Default: 100.
Max entries in llms-full.txt	Maximum number of entries in /llms-full.txt. Range: 1–200. Default: 50.
Add a short description to each entry	When on, each entry in /llms.txt includes a short description. Uses the hand-written excerpt when one exists, otherwise the opening of the post content.
Description length	Maximum characters for the per-entry description. Range: 40–320. Available when Add a short description to each entry is on.
Body length in llms-full.txt	Maximum characters of body text per page in /llms-full.txt. Range: 500–20,000.
Skip pages marked noindex	When on, pages marked noindex by Antimanual, Yoast SEO, or Rank Math are excluded from the document.
Only list Knowledge Base content	When on, the document is restricted to posts already indexed in your Antimanual Knowledge Base. The current Knowledge Base count is shown in the help text.

To save changes, click Save settings. If you have unsaved changes, a sticky bar also appears at the bottom of the page with Save settings and Discard buttons.

---

Pin and exclude section

Control	Description
Key pages	Pages pinned here appear first in the document under a ## Key pages heading, in the order you add them. Use this for the handful of pages you most want cited. Search for published content by title or ID.
Excluded pages	Pages listed here are never included in the document, regardless of other settings.

---

Preview section

The right column shows a live preview of the document that updates as you change settings (with a short debounce delay). The preview shows:

• A toggle to switch between llms.txt and llms-full.txt previews.
• Download button — downloads the current preview as a local file.
• Rebuild button — drops the document cache and rebuilds both documents from your current content. Any static files written by Antimanual are also updated.
• Entry count, file size, approximate token count, and build timestamp.
• The rendered Markdown document body (truncated for display if very large; the published file is always complete).
• A Public URL field showing the live URL of the document.
• An Open button that opens the live document in a new tab (only active when the document is enabled).

Note: If your document hits the size ceiling, a warning is shown: “This document hit the size ceiling and was cut short. Lower the entry count or the body length.”

---

Static file (optional) section

Antimanual serves both documents through WordPress by default, which is sufficient for most sites. This section lets you write a real file to your web root if a caching layer or your host answers unknown paths before WordPress sees them.

For each document (llms.txt and llms-full.txt):

• The current status is shown: size and write date if a file exists, or “No static file in your site root”.
• A badge shows whether the file is Antimanual managed or an External file (written by something else).
• Write file — writes the current document to your web root. If an external file already exists, a confirmation dialog is shown before overwriting.
• Overwrite — replaces an existing file.
• Remove — deletes an Antimanual-managed file from your web root.

Important: When a static file exists, it takes precedence over virtual endpoints. Antimanual keeps its own managed files up to date on every save, as long as that document is still turned on.

---

How the document is generated and cached

• Documents are generated from your published content and cached in a WordPress option. The cache is invalidated automatically when a post in the selected post types is saved, deleted, trashed, or restored.
• The cache key includes a signature of the current settings, so changing settings always produces a fresh build.
• The document is served through the WordPress parse_request hook (not written to disk unless you use the Static file section).
• A hard cap of 1,500,000 bytes applies to any single document. If a document exceeds this, it is cut at the last complete line and a truncation comment is appended.

---

Tab 2: AI Crawlers

This tab lets you set per-bot Allow or Disallow rules for known AI crawlers. Rules are appended to WordPress’s virtual robots.txt through the standard robots_txt filter — nothing is written to disk.

Crawler rules section

Setting	Description
Manage AI crawlers in robots.txt	Master toggle. When on, Antimanual appends its block to your robots.txt. Bots left on Default are never mentioned, so your existing rules stay untouched.

Presets let you set rules for all crawlers at once:

Preset	Effect
Allow all	Sets every crawler to Allow.
Block training, keep citations	Sets training crawlers to Block, all others to Allow.
Block all	Sets every crawler to Block.
Clear	Resets every crawler to Default.

A filter box lets you search crawlers by name, user-agent string, or vendor.

---

Crawler catalog

Crawlers are grouped into three categories:

Model training

Crawlers that collect pages to train future AI models. Blocking them keeps your writing out of training data and costs you no search visibility.

Crawler	Vendor	User-agent
GPTBot	OpenAI	GPTBot
ClaudeBot	Anthropic	ClaudeBot
Google-Extended	Google	Google-Extended
CCBot	Common Crawl	CCBot
Applebot-Extended	Apple	Applebot-Extended
Meta-ExternalAgent	Meta	meta-externalagent
Bytespider	ByteDance	Bytespider
Amazonbot	Amazon	Amazonbot
cohere-ai	Cohere	cohere-ai
Diffbot	Diffbot	Diffbot
ImagesiftBot	ImageSift	ImagesiftBot
MistralAI	Mistral AI	MistralAI

AI search & citations

Crawlers that build the indexes AI answers cite from. Blocking one removes your site from that product’s answers.

Crawler	Vendor	User-agent
OAI-SearchBot	OpenAI	OAI-SearchBot
Claude-SearchBot	Anthropic	Claude-SearchBot
PerplexityBot	Perplexity	PerplexityBot
YouBot	You.com	YouBot
DuckAssistBot	DuckDuckGo	DuckAssistBot

On-demand user fetches

Crawlers that load a page only when a person asks the assistant about it. Blocking them prevents click-through from inside the assistant.

Crawler	Vendor	User-agent
ChatGPT-User	OpenAI	ChatGPT-User
Claude-User	Anthropic	Claude-User
Perplexity-User	Perplexity	Perplexity-User
Meta-ExternalFetcher	Meta	Meta-ExternalFetcher

Each crawler row shows:

• The crawler label, vendor, and user-agent string.
• A Vendor docs link (where available) to the vendor’s official crawler documentation.
• A description of what blocking the bot actually costs the site.
• A hit badge: No recent visits or the number of hits in the last 30 days plus the time since the last visit.
• A three-option toggle to set the rule: Default, Allow, or Block.

Group actions within each category let you set all crawlers in that group to Allow all, Block all, or Reset at once.

To save, click Save crawler rules. A sticky save bar also appears at the bottom when there are unsaved changes.

---

robots.txt preview section

The right column shows a live preview of the full robots.txt your site would serve right now, updating as you change rules. It shows the number of blocked and explicitly allowed crawlers.

• Open robots.txt — opens your live robots.txt in a new tab.

---

Extra directives section

Setting	Description
Point crawlers at your llms.txt	Adds a comment line with your llms.txt URL to the Antimanual block in robots.txt. Only written while llms.txt is being served.
Add a Sitemap line	Adds a Sitemap: directive. Useful when nothing else on the site declares one. Skip if your SEO plugin already adds one.
Custom directives	A free-text area for additional robots.txt directives, appended verbatim. Only standard robots.txt fields are accepted: User-agent, Allow, Disallow, Crawl-delay, Sitemap, Host, Clean-param, and comments. Other content is stripped on save. Maximum 2,000 characters.

---

Choosing a policy — in-page guidance

The Choosing a policy section on the AI Crawlers tab includes the following notes (shown directly in the UI):

• Blocking training crawlers while allowing search crawlers is the common middle ground: your work stays out of training corpora but can still be cited.
• Google-Extended only governs Gemini and AI Overviews. Blocking it does not affect normal Google Search ranking.
• robots.txt is a request, not enforcement. Crawlers that ignore it need blocking at the server or CDN.
• Leaving a bot on “Default” writes nothing for it, so any rules already in your robots.txt keep working.

---

Important: physical robots.txt file

If your web root contains a physical robots.txt file, your server sends it directly — WordPress never sees the request, so nothing set in this tab takes effect. A red alert is shown in this case. To fix it: delete or rename the physical file, or copy the generated block into it manually.

Important: search engine visibility

If WordPress is set to discourage search engines (Settings → Reading), the AI Crawlers tab shows a warning and per-bot rules are not written. Turn that setting off to manage AI crawlers.

---

Crawler hit tracking

Antimanual passively logs visits from known AI crawlers to your site’s front end. For each bot it records:

• Last seen — timestamp of the most recent visit.
• Hits in the last 30 days — a rolling 30-day visit count.

Hits are sampled at most once per bot per minute to avoid excessive database writes. This data is shown in the crawler list and does not require any configuration.

---

Tab 3: Citation Risk

This tab ranks your published content by how likely an answer engine is to skip it, based primarily on how recently each page was updated.

How risk is calculated

Each page receives a risk score from 0 to 100:

• Age is the dominant factor. Pages updated within the Fresh window receive a score of 0. Pages in the Aging range receive a score between 25 and 60. Pages past the Stale after threshold receive a score between 60 and 85, rising further as they age.
• Not in llms.txt adds 8 points to the score.
• Not in Knowledge Base adds 7 points to the score.
• Thin content (word count below the Thin content under threshold) adds 10 points to the score.

The maximum risk score is 100.

---

Freshness buckets

Pages are classified into one of four buckets, shown as clickable metric cards:

Bucket	Condition
Fresh	Updated within the Fresh window (default: ≤ 90 days)
Aging	Updated between the Fresh and Stale thresholds (default: 91–180 days)
Stale	Not updated since the Stale after threshold (default: > 180 days)
Snoozed	Manually removed from the queue for 90 days
All monitored	Every post across all buckets

The summary counters are cached for 15 minutes and refreshed when you click Rescan.

---

Thresholds section

Setting	Description
Fresh window	Days. Content updated inside this window carries no age risk. Range: 7–365. Default: 90.
Stale after	Days. Past this point a page is treated as out of citation eligibility. Must be at least 7 days beyond the Fresh window. Range: up to 1,095. Default: 180.
Thin content under	Words. Pages shorter than this receive a risk penalty. Set to 0 to disable the thin-content penalty. Range: 0–3,000. Default: 300.
Post types to score	Which post types are included in the Citation Risk queue.

Click Save thresholds to apply. A sticky save bar also appears at the bottom when there are unsaved changes.

---

Refresh queue

The queue lists pages in the selected bucket, ordered by last-modified date (oldest first), 25 per page. Each row shows:

• Risk score — a colored badge (green for low risk, amber for medium, red for high).
• Title — linked to the edit screen or the live URL.
• Days since update, word count, and post type.
• A flag if the page is not in llms.txt or not in Knowledge Base.
• Edit — opens the post editor.
• View — opens the live page in a new tab.
• Snooze 90 days — removes the page from the queue for 90 days. A toast notification appears with an Undo button that remains for 6 seconds.
• Requeue — for snoozed pages, returns them to the active queue.

Rescan forces a fresh scan of the current bucket, clearing the cached summary. Export CSV downloads the current page of results as a CSV file with columns for title, URL, post type, days since update, word count, in llms.txt, in Knowledge Base, and risk score.

---

Schedule refreshes (Antimanual Pro only)

The Schedule refreshes button links to the Auto-Update module. This button is only active when Antimanual Pro is installed and the Auto-Update module is enabled. On Antimanual Free, clicking the button shows a Pro upgrade prompt.

---

Antimanual Free vs. Antimanual Pro

Capability	Free	Pro
Serve /llms.txt and /llms-full.txt	✓	✓
All llms.txt generator settings	✓	✓
AI crawler allow/block rules	✓	✓
robots.txt preview	✓	✓
Citation Risk queue	✓	✓
Verify Setup (loopback check)	✓	✓
Crawler hit tracking (30-day)	✓	✓
Schedule content refreshes from the Citation Risk queue	—	✓ (requires Auto-Update module)

---

Troubleshooting & Notes

llms.txt returns a 404 or the wrong content

• Confirm Serve /llms.txt is on (the header stat will show Live).
• If you are on plain permalinks, switch to a pretty permalink structure or write a static file from the Static file section.
• If a static file exists that was not written by Antimanual, it takes precedence over the virtual endpoint. Check the Static file section for an External file badge.
• Run Verify Setup for a loopback check that confirms the endpoint is reachable and returning a valid document.

AI crawler rules are not appearing in robots.txt

• Confirm Manage AI crawlers in robots.txt is on.
• Check that no physical robots.txt file exists in your web root (an alert is shown on the tab if one is detected).
• Check that Settings → Reading does not have “Discourage search engines” enabled (an alert is shown on the tab if this setting is active).
• If every crawler is set to Default, nothing is written, which is expected behavior.

The Citation Risk queue is empty

• Confirm the desired post types are selected in Thresholds → Post types to score.
• If the Stale bucket is empty, no published content has exceeded the stale threshold yet.
• Click Rescan to force a fresh count.

Static files and cache

• Antimanual automatically updates any static files it wrote whenever you save settings, as long as that document is still enabled.
• The document cache is also dropped automatically when a post in the selected post types is saved, deleted, trashed, or restored.
• To force a manual rebuild, click Rebuild in the Preview section.

Need Further Assistance?

If you encounter any issues or have questions regarding llms.txt & AI Crawlers, please visit our official support channels:

• Support Portal: https://helpdesk.spider-themes.net/ask/