User guide

Product version: 0.1.0 (CS-Cart 4.18+, verified on 4.18.3; PHP 8.1+).

This guide describes implemented P0 MVP behaviour only.

Screen illustrations are available on the add-on listing page. Below is a short text description of what you will see in the admin area.

Product formula: Visible. Controlled. Measured.

1. What it is

RobotsLLM for AI-SEO is a CS-Cart add-on that:

1. Controls AI and SEO crawler access via a smart robots.txt (presets, diff, rollback).
2. Maintains a registry of 31 bots and can fetch a diff from the open ai-robots-txt source (apply is manual only).
3. Generates /llms.txt and per-language variants as a navigation map for AI agents (with curation).
4. Publishes .md mirrors for categories, CMS pages, and products.
5. Protects crawl budget (CS-Cart critical paths, sitemap line, noindex on mirrors).
6. Shows AI bot analytics and AI-referral orders in the admin area.
7. Sanitizes content, scans for PII/secrets, and runs self-audit.

No separate VPS is required. It is a normal PHP add-on + MySQL queue + panel cron.

Serving note. Generated /llms.txt and *.md files live under var/ (CS-Cart denies direct web access to that tree). Public URLs are served by PHP serve (dispatch=robots_llm.serve): the same Content-Type, X-Robots-Tag: noindex, ETag / Last-Modified, and 304 on a matching conditional request. Zero-PHP static delivery is a future optimisation, not 0.1.0 behaviour.

The add-on does not publish robots.txt by itself after install. The policy is applied only with Publish.

2. Installation (zip via admin)

1. Obtain package robots_llm-0.1.0.zip (built with build/package.ps1 or build/package.sh).
2. In CS-Cart: Administration → Add-ons → Manage add-ons → Upload & install (wording may vary by edition).
3. Select the zip and confirm.
4. Activate RobotsLLM if it is not active.
5. Refresh language/template caches if UK/EN strings do not appear.

Package layout (store root):

app/addons/robots_llm/
design/backend/templates/addons/robots_llm/
var/langs/uk/addons/robots_llm.po
var/langs/en/addons/robots_llm.po

Install creates ?:robots_llm_* tables, seeds 31 bots, and prepares var/files/{id}/robots_llm/generated and …/previous (root from CS-Cart fn_get_files_dir_path()). If the store-root .htaccess is writable, the add-on inserts an idempotent # BEGIN robots_llm section - rules send .md / llms.txt to PHP serve, not to a static file under var/.

On screen: the CS-Cart add-on list; a RobotsLLM for AI-SEO row with status Active and the add-on description.

Uninstall / rollback

• robots.txt rollback: Website → RobotsLLM → Crawl policy → rollback to a previous version. The first publish stores a backup of the current CS-Cart robots content.
• Full uninstall: Add-ons → Uninstall. Tables drop via uninstall SQL; generated files and the .htaccess section are removed by the uninstall hook. Before dropping tables the add-on tries to restore the first-run backup into CS-Cart storage. If you need a later version, rollback before uninstall. The add-on never creates or deletes a physical root robots.txt.

3. First run: policy wizard (robots.txt)

Menu: Website → RobotsLLM → Crawl policy.

The wizard asks a few questions (AI search / training / user agents / SEO tools) and selects a preset (or Custom). Then:

1. Review the live preview of generated robots.txt.
2. Review the before → after diff.
3. Optionally use the simulator (“what bot X sees for URL Y”).
4. Save, then Publish (writes via CS-Cart’s storefront robots storage + version ledger). Install does not publish.

On screen (wizard): yes/no questions, a preset dropdown (auto / Growth / Open / Strict / Custom), and the current preset label.

>

On screen (diff): a full robots.txt preview and a unified diff of “active version → new”.

>

On screen (simulator): bot picker, path field, run button, and an Allowed / Blocked badge.

3.1. Presets - honest bot-class table

Custom - manual purpose-class toggles + per-bot overrides in the registry.

Compiler notes (RFC 9309):

• For every bot with its own group, critical Disallows are copied into that group (a dedicated group ignores *).
• Google-Extended / Applebot-Extended are control tokens: full Disallow: / or no group.
• Existing merchant rules are imported as a manual block and are never silently deleted.
• A single Sitemap: line is appended.

3.2. Physical robots.txt in the store root

If a physical robots.txt exists in the storefront root, CS-Cart serves that file - the database row (and the add-on policy) will not go live. The add-on writes only to native storage and does not overwrite or delete that file. Rename or remove the physical file (keep a copy) before you expect /robots.txt to show the published policy.

4. Bot registry

Website → RobotsLLM → Bot registry.

• Seed of 31 bots on install.
• Columns: name, provider, purpose badge, recommendation, merchant decision (allow / block / default) per storefront.
• Sync from ai-robots-txt: diff → pending_review; apply only via admin button. Sync failure does not break the store.

On screen (registry): storefront selector and a table (name, provider, purpose, recommendation, your decision, source, status).

>

On screen (sync): status idle / pending_review / error, last-check time, added / changed / removed counts, and Check now / Apply buttons.

5. Curating llms.txt (AI Discovery)

Website → RobotsLLM → AI discovery.

Generated files:

• /llms.txt - root language index;
• /{lang}/llms.txt - one file per active storefront language.

Structure: title → description → meta (language, market, currency, updated, price disclaimer) → sections (categories / help / policies / Optional).

UI curation:

• pin / unpin / exclude / order (up/down);
• token budget (default 15,000, estimate ≈ length/4);
• max URLs;
• “Generate now” + text preview;
• auto-exclusions: noindex, hidden, disabled; out-of-stock → Optional.

All URLs are absolute canonicals. Languages are not mixed in one file.

Honest statement: llms.txt is a navigation map for AI agents, not a “ChatGPT button”. Primary visibility levers are correct search/user bot allowances + clean data.

On screen (curation): token budget, max URLs, queue counter, and candidate tables with pin / exclude / ↑ / ↓.

>

On screen (preview): last generation status (idle/ok + hash) and the generated llms.txt text after “Generate now”.

6. Markdown mirrors and rewrite

6.1. What is generated

• URL scheme: {seo_path}.md (e.g. /batteries.md, language prefixes per store scheme).
• On disk: var/files/{id}/robots_llm/generated/{storefront_id}/{lang}/... (not a public web directory).
• P0 scope: categories, CMS pages (policies), products (name, brand, SKU, anonymous price + disclaimer, stock, attributes, sanitized description, canonical).

Publisher: tmp → security validation → atomic rename → documents registry → one generation in previous/.

A critical finding (secret / private PII / B2B price) blocks publish; last-known-good remains. A phone/email inside a public product description is a warning, not a block (secrets still block).

6.2. Delivery (PHP serve, not “pure static”)

CS-Cart denies web access to var/ (Require all denied). Rewrites therefore do not serve the file directly from disk.

1. Apache / LiteSpeed (# BEGIN robots_llm section): *.md, /llms.txt, /{lang}/llms.txt → index.php?dispatch=robots_llm.serve&rllm_path=….
2. The robots_llm.serve controller reads the PHP-readable file under var/files/… and emits headers + body.
3. Success headers: Content-Type (text/markdown; charset=utf-8 for .md, text/plain; charset=utf-8 for llms.txt), X-Robots-Tag: noindex, ETag, Last-Modified, Cache-Control: public, max-age=300.
4. A matching If-None-Match / fresh If-Modified-Since → 304 (caching works).
5. Miss: if the document is in the registry, one enqueue; 404 without caching empty “valid” bodies.

The add-on generates an idempotent section between markers:

# BEGIN robots_llm
# RobotsLLM: .md / llms.txt via PHP serve (do not edit by hand)
… RewriteRule → index.php?dispatch=robots_llm.serve …
# END robots_llm

On screen: in the store-root .htaccess, a marked # BEGIN robots_llm … # END robots_llm block with rules to robots_llm.serve (not a path into var/).

Nginx: HtaccessSectionGenerator::nginxSnippet() documents a vhost snippet (not under design/): location for *.md and llms.txt → rewrite to the same dispatch=robots_llm.serve. Do not try_files under var/ - the web server will 403.

After install, verify the section actually landed in the store root .htaccess / vhost (shared hosting may require pasting it by hand).

6.3. Dirty queue

Hooks on product/category/page create/update/delete, price and inventory changes → dirty_queue.

Cron job=queue (every 5 min): batch 100, ~55 s time budget.

The code also has job=reconcile (hash check vs disk) - not part of the recommended 0.1.0 panel schedule; run it only if you need it.

Full rebuild - button on Dashboard / Discovery, or job=rebuild (streaming, checkpoint in job_state).

7. Cron setup (panel commands)

The recommended path is the hoster’s panel cron calling the CS-Cart admin script on the CLI. The exact script name differs per store (often a renamed admin.php). Substitute your PHP 8.1+ binary and your admin script.

Pattern:

/usr/bin/php8.1 /path/to/YOUR_ADMIN.php --dispatch=robots_llm.cron job=JOB

Console mode does not need the cron password. HTTP dispatch still works if you pass Settings → Security → Cron password as cron_password= - CLI is simpler for a panel.

Example crontab (replace PHP and the admin script):

*/5  * * * * /usr/bin/php8.1 /path/to/YOUR_ADMIN.php --dispatch=robots_llm.cron job=queue
*/15 * * * * /usr/bin/php8.1 /path/to/YOUR_ADMIN.php --dispatch=robots_llm.cron job=logparse
30   2 * * * /usr/bin/php8.1 /path/to/YOUR_ADMIN.php --dispatch=robots_llm.cron job=aggregate
45   2 * * * /usr/bin/php8.1 /path/to/YOUR_ADMIN.php --dispatch=robots_llm.cron job=llms
0    3 * * * /usr/bin/php8.1 /path/to/YOUR_ADMIN.php --dispatch=robots_llm.cron job=audit
10   3 * * * /usr/bin/php8.1 /path/to/YOUR_ADMIN.php --dispatch=robots_llm.cron job=registry_sync

In Hosting Ukraine and similar panels, create the same six jobs as a “PHP command”, not as a URL.

On demand (not in the required schedule): job=rebuild - full regeneration; job=reconcile - document hash check.

On screen (add-on settings): a “Cron jobs” info block with interval / job-name hints and the access-log path field.

>

On screen (hosting panel): cron rows with your PHP binary, your admin script, and job=queue / logparse / aggregate / audit / llms / registry_sync.

8. Analytics dashboard

Website → RobotsLLM → Dashboard.

Shows (once cron has data):

• bot requests per day (up to 30 days);
• top bots (verified / unverified);
• purpose breakdown;
• top URLs;
• AI referrals: sessions / orders / revenue by source (utm / referrer: ChatGPT, Perplexity, Claude, Gemini, Copilot…).

Data sources:

• PHP logger (registry UA match on the frontend) - sees requests that hit PHP, including /llms.txt and *.md via serve;
• Access-log parser - hits PHP never sees (other static / logger gaps); requires a log path in settings. If unavailable, only the PHP logger runs (the UI states the limitation honestly).

Verification: IP in provider CIDR ranges (cached) or rDNS for Googlebot/Bingbot; otherwise unverified.

GA4: P0 only stores a property id + a reconciliation help note (full API integration is out of MVP).

On screen (dashboard): cards for 30-day requests, verified %, AI sessions / orders / revenue; an SVG request chart; a top-bots table.

>

On screen (referrals): a source table (ChatGPT, Perplexity, Claude, Gemini, Copilot…) with sessions, orders, and totals.

9. Self-audit and content security

Website → RobotsLLM → Self-audit.

• DOM sanitizer (not regex): strip script/style/iframe/form/hidden → clean text/markdown.
• PII/secret scanner: emails/phones (except official contacts from settings), key patterns (AKIA, sk-, …), injection heuristics.
• Critical → do not publish.
• Self-audit: HTTP (200, MIME, charset, ETag, 304, not login HTML), content structure / absolute URLs, price parity sample.

On screen: a “Run now” button, critical/warning/info filter, a runs table, and a findings list (check code, object, message).

10. Add-on settings (short)

11. FAQ (facts)

Will my robots.txt change as soon as I install the add-on?

No. The add-on publishes nothing without your action. The policy is written to CS-Cart storage only with Publish, after a diff; rollback is one action. If a physical robots.txt exists in the store root, it wins over the database - rename or remove it, or the live storefront will not show the add-on policy.

Do I need a separate VPS?

No. PHP add-on + MySQL queue + panel cron.

Will it break SEO?

It should not, if you use diff and rollback. Policy publish always shows before → after; one-click rollback; first-run backup of merchant robots. .md / llms.txt are noindex so they do not create Google duplicates.

Resource use?

Rough guide: ~5-20 MB disk for ~5,000 products; generation on change (queue), not on every hit. Public /llms.txt and *.md go through PHP serve (cheap file read + ETag/304). Zero-PHP static delivery is not in 0.1.0.

Compatibility

CS-Cart 4.18+, verified on 4.18.3. PHP 8.1+.

How do I roll back?

1. Crawl policy → robots rollback.
2. Restore from var/files/…/robots_llm/previous/… (last-known-good mirrors).
3. Or uninstall the add-on (it tries to restore the first-run backup into native storage).

Does llms.txt guarantee ChatGPT citations?

No. It is a map for agents. We do not promise “+N% visibility” or that llms.txt improves Google rankings.

Free / Pro

v0.1.0 / v1.0 MVP ships as one free package with full P0. Monetization may come later.

12. Seven modules (support map)

1. AI Crawl Policy - presets, RFC 9309 compiler, diff, rollback, simulator.
2. Bot Registry - seed of 31, overrides, ai-robots-txt sync with pending_review.
3. AI Discovery - llms.txt + curation + token budget.
4. Markdown Mirrors - renderers, publisher, queue, rewrite to PHP serve.
5. SEO Robots Tweaks - critical paths, sitemap, mirror noindex.
6. AI Analytics - logger, logparse, verify, referrals, dashboard.
7. Security & Self-Audit - DOM sanitizer, secret blocking, audit runner.

13. Open items (honest)

• Screenshots are not embedded in this file - see the add-on listing page.
• 0.1.0 has no dedicated UI blocker for “physical robots.txt found”: file-wins is CS-Cart behaviour; you remove the file by hand.
• Nginx rewrites are typically pasted into the vhost.
• The settings cron info block may still mention HTTP/cron_password and extra jobs; use the §7 table for the panel schedule.

Aligned with docs/marketing-foundation.md and the actual 0.1.0 code (CS-Cart 4.18.3 stand).