edgely
Back to Blog
AI & Agents
7 min readUpdated June 1, 2025

Why AI Agents Need a Custom Domain — And How to Set One Up

AI agents scraping or reading your site work best with plain text. Edgely's markdown mode converts your site to clean text for AI scrapers, served on your own branded domain.

TL;DR — Quick Answer

AI agents and LLMs read your site better when it's served as clean markdown instead of raw HTML. Edgely's markdown mode strips all HTML and serves your content as plain text on your own domain — improving AI comprehension and citation accuracy.

How AI Agents Read the Web

LLMs and AI agents access websites in two ways: via web search APIs that scrape and parse pages, or by directly fetching URLs. In both cases, the agent receives HTML — and HTML is noisy. Navigation menus, cookie banners, footers, scripts, and inline styles all inflate the context window and degrade comprehension quality. [1]

When an AI agent cites or reads your site, it performs better with clean, structured text — ideally Markdown. [2]

Why a Custom Domain Matters for AI Discoverability

AI search engines like Perplexity, SearchGPT, and Claude Web Search attribute sources by URL and domain. A branded custom domain (www.yourbrand.com) is cited more cleanly than a yourapp.lovable.app subdomain. Custom domains also signal authority and longevity — factors AI systems weigh when ranking sources. [3]

Edgely Markdown Mode

Edgely includes a Markdown Mode toggle. When enabled, Edgely:

  1. Fetches the HTML from your origin.
  2. Strips all HTML tags, navigation, scripts, and styles.
  3. Converts headings, paragraphs, links, lists, and code blocks to standard Markdown.
  4. Serves the Markdown with Content-Type: text/markdown.

AI scrapers that accept text/markdown receive clean, structured content. Human browsers still get the full site (the toggle can be configured to apply only to certain Content-Type requests).

Setup: AI-Optimised Custom Domain

1. Add your domain in Edgely

Sign up at xedgely.com and add your custom domain with your origin URL as the target.

2. Enable Markdown Mode

In the domain settings toggle Markdown Mode on. Save.

3. Add CNAME

Type:  CNAME
Host:  www
Value: proxy.xedgely.com

4. Test with curl

curl -H "Accept: text/markdown" https://www.yourdomain.com

The response should be clean Markdown — no HTML tags.

Structured Data for AI Engines (GEO)

In addition to Markdown mode, follow these Generative Engine Optimisation (GEO) best practices: [4]

  • Add FAQ schema — direct answers in JSON-LD help AI engines surface your content as a direct response.
  • Add HowTo schema — step-by-step guides are a top citation format for AI responses.
  • Include citations and references — AI systems trust content that cites authoritative sources.
  • Use clear headings — H2/H3 structure helps AI parse your content hierarchy.
  • TL;DR at the top — a concise summary at the article start is exactly what AI engines use for quick answers.

Real-World Impact

Sites that serve clean Markdown from their custom domain have been observed to appear more frequently in AI-generated answers in tools like Perplexity AI and Claude Web Search, because the content is easier to parse, more citation-friendly, and more authoritatively attributed to a real domain. [5]

Edgely is the fastest way to proxy your your AI-optimised site project to a custom domain. It provisions a free SSL certificate, syncs routing to Vercel Edge Config for sub-millisecond lookups, and optionally caches responses at the edge — all for free on the starter plan.

Key Takeaways

  • AI agents read your site better when served clean Markdown instead of raw HTML.
  • Edgely's Markdown Mode converts your site's HTML to structured Markdown automatically — only for AI scrapers.
  • A custom domain is cited more authoritatively by AI search engines than a platform subdomain.
  • Add FAQ, HowTo, and Article JSON-LD schema to maximise AI discoverability (GEO).
  • Include citations and a TL;DR in your content — these are the formats AI engines prefer for direct answers.

Frequently Asked Questions

What is Generative Engine Optimisation (GEO)?

GEO is the practice of optimising content so it appears in AI-generated answers (from tools like Perplexity, ChatGPT Search, Claude, Gemini). It combines structured data (FAQ, HowTo schemas), concise TL;DR summaries, citations, and clean content formatting.

Does enabling Markdown Mode affect human visitors?

Edgely's Markdown Mode serves Markdown to clients that send Accept: text/markdown. Browsers request text/html, so human visitors continue to receive the full HTML site. Only AI scrapers and tools that explicitly accept Markdown receive the converted output.

Will AI systems cite my site more if I have a custom domain?

A branded custom domain signals authority and permanence. AI search engines factor domain reputation and citation patterns into source selection. A custom domain is more likely to be cited consistently compared to a platform subdomain.

What structured data formats do AI engines prefer?

JSON-LD is the preferred format for structured data. Add FAQPage, HowTo, and Article schemas to your pages. Google, Bing, and AI engines all parse JSON-LD from the <head> or <body> of your HTML.

How do I know if an AI agent has visited my site?

AI scrapers identify themselves via User-Agent strings (e.g. GPTBot, ClaudeBot, PerplexityBot). Check your server logs or Edgely access logs for these user agents to see AI traffic to your site.

Sources & Citations

  1. [1]How GPT-4 Processes Web Content— openai.com
  2. [2]Markdown for AI Agents (Anthropic Docs)— docs.anthropic.com
  3. [3]Generative Engine Optimisation — Research Paper— arxiv.org
  4. [4]JSON-LD Structured Data (Google)— developers.google.com
  5. [5]Perplexity AI Source Citation Methodology— perplexity.ai

Ready to add your custom domain?

Set up your reverse proxy in under 5 minutes — free, with SSL and edge caching included.

#ai agent#markdown#llm#web scraping#custom domain