Playbook: AEO foundations
robots, sitemap, llms.txt, and structured data for every site.
Every managed site carries four artifacts that make it visible to search engines and AI answer engines. Generated per site, committed to its repo, deployed automatically.
The artifacts
- robots.txt, two modes:
- Live (site serves production traffic on its real domain): allow all, with explicit Allow blocks for AI crawlers (GPTBot, ClaudeBot, Claude-Web, PerplexityBot, Google-Extended) and a Sitemap line.
- Blocked (
Disallow: /): pre-cutover clones and test sites. A clone must never compete in search with the client’s still-live original. Flip to live mode as part of every DNS cutover.
- sitemap.xml: generated from the repo’s html pages with canonical-domain URLs. Live sites only.
- llms.txt: a concise, human-written summary of who the client is, what they do, and key pages. Written in the client’s voice, no em dashes, nothing invented.
- JSON-LD on the homepage: schema.org type matched to the client (NGO, LocalBusiness, Organization). Only fields we can verify from the site itself; never guess addresses or phone numbers.
Running it
The generator lives at the porter repo level (aeo.py pattern: one atomic commit per site via the git trees API). State per site (live vs blocked) follows the DNS cutover status in the console.
Cutover checklist addition
Flipping a domain now includes: switch robots.txt to live mode, add sitemap.xml, add llms.txt, submit the sitemap in Search Console. See DNS cutover.
Status 2026-07-19: naacp-snoco.org and ariodetail.com fully live (NAACP includes NGO JSON-LD and a 20-page sitemap). paula, ameraid, wafamilyrelief, backpackops-site, unify-20, sandbox are in blocked mode pending cutover decisions.