GEO-021·Issue 02·Authority & Off-site·21 / 132
Does Domain Trust Affect Crawling?
Inherited domains often crawl fine and still earn few AI visits. Separate access, crawl priority, and citation trust, then clean up in the right order.
Issue focusGet Seen First: Crawling, Rendering, and AI Bot Accessibility
A US DTC home-goods brand buys a competitor’s domain in an acquisition. The catalog comes with it: thousands of SKU pages scraped from a marketplace years earlier, images on a CDN account nobody can log into, and a returns page naming an LLC the brand dissolved. robots.txt allows everything. The WAF has no bot rules. Search Console shows no manual actions. Six weeks in, the logs say otherwise: retrieval crawlers arrived twice, spent most requests on scraped pages, and never touched the forty SKUs the brand sells.
Domain trust is not a number you can look up; it is three checkable things — penalties, thin history, off-site consistency — and it sets how often crawlers return, not whether they can enter.
Three layers, one misplaced question
Domain trust is a second-layer problem, and teams keep diagnosing it as a first-layer one.
| Layer | Decided by | Effect of thin history | What you can change |
|---|---|---|---|
| Access | robots.txt, WAF, rendering, status codes | Almost none | Allow verified bots, server-render, fix 404s |
| Crawl share | Priority, site structure, freshness | Direct | Drop thin pages, noindex facets, real lastmod |
| Citation trust | Answer units, evidence, entity, third-party corpus | Indirect | Content and off-site work, elsewhere |
Access is close to binary and mostly yours. Crawl share is a triage call the crawler makes without asking. Citation trust is whether a model dares to quote the page.
Most teams rewrite robots.txt, open the WAF, then wait. Clearing old pages changes where requests land, and logs show that faster than ten new articles do. The third layer is separate work — authority and off-site is the lane.
Four signals, two you can check directly
A trust signal that cannot be traced to a record or a log line is a mood, not a diagnosis.
| Signal | Where to look | Pass mark |
|---|---|---|
| Manual actions and security issues | Search Console | Any record stops the project |
| Thin and duplicate share | Sample 200 URLs: unique body, facets, soft 404s | Over half unique; facets under 3x products |
| Off-site consistency | Brand, legal name, address, returns page across site, marketplace, reviews | One story in all four |
| Link anomalies | Fresh-domain spikes, one repeated anchor, reciprocal sites | Investigate before buying |
DA and DR are link-graph estimates, not domain trust, and Google says it keeps no site-wide authority score. The counterexample is routine: DR slips, someone buys links, three months later a link-spam manual action lands. That period’s log would have shown the real movement: product-template requests and 403/404 counts.
Why thin history turns into fewer visits
Every crawler spends a finite number of requests, and each visit is a return-on-crawl calculation: how much unseen or genuinely changed content comes back. The lower the ratio, the weaker the case for a return trip. This is triage, not punishment.
Three leaks matter. Duplication: thousands of scraped pages repeat the catalog, so requests buy nothing. Empty returns: soft 404s, blank facets, loading skeletons answering 200 with no body — no error is raised, the ratio just falls. Misplaced entry: the link graph points at parameter combinations instead of product pages, so volume looks fine while few PDPs get seen. Navigation AI can follow fixes that one.
A manual action is a different path: indexing and ranking fall first, and lower crawling follows. Do not read the symptom as the cause.
Clean up in order: stop, sort, then publish
Cleanup reallocates crawl requests onto pages that earn citations; order decides whether it works.
| Step | Do | Owner | Passes when |
|---|---|---|---|
| 1 Evidence | 90 days of requests by agent and path | Engineering | Top paths, template share |
| 2 Stop the bleed | Pause generators and auto-facet URLs; rate-limit hot paths | Content and engineering | No new URLs of that kind |
| 3 Triage | Rewrite, merge, or 410 each legacy URL | Content | Merges only where a replacement exists |
| 4 Consolidate | Noindex facets, fix soft 404s, prune the sitemap | Engineering | Sitemap and index counts converge |
| 5 Re-test | Same paths at 30 and 60 days | Growth | Template coverage up, legacy down |
Step one is a full AI crawl audit, not a vendor report; the allocation behind step two is how AI crawlers spend their requests.
Two mistakes repeat. Deleting tens of thousands of pages at once destroys value you cannot see: batch by backlinks and orders, and check inbound links first. 301-ing every old URL to the homepage is a soft 404 with extra steps — it tells the crawler those pages never existed.
Example: a 2019 domain with 4,000 legacy pages
Example — a hypothetical US home-storage brand inherits a 2019 domain with about 4,000 auto-generated SKU pages and a returns policy naming the seller it bought from. No real brand, no figures.
Freeze the generator, noindex facet URLs, turn soft 404s into 410s or real content. Sort the backlog by backlinks and order value into merge, rewrite, and delete groups, no more than 500 URLs per batch, two weeks apart. Only then rewrite answer units, and only on templates the crawler already visits.
Acceptance is not revenue. It is three movements: template crawl coverage rises, the unique-body share of requested URLs rises, and the domain starts appearing in citations for the same buyer questions. The first two move in weeks; the third lags. Do not raise the content budget until it moves, and re-measure with server-log analysis.
Where this breaks
A fresh domain has nothing to clean. Rendering, entities, and answer units are the whole job; cleanup wastes a quarter.
Sales concentrate in a marketplace. On Amazon or Shopee, the assistant reads listings, reviews, and fulfillment. Domain history barely matters; fix the feed.
A penalty is under review. Follow the official process and freeze structural changes, or you cannot tell which fix worked.
The domain is being retired. Migration and redirects come first; history travels with them, and cleanup is wasted.
“Domain trust” justifies buying links. Links are not a crawl switch. Consistent entity data and checkable evidence survive a domain move.
This month, export the 500 oldest sitemap URLs and mark three boxes on each: unique body copy, one external link, a crawl hit in 90 days. Any row with three empty boxes gets merged or 410’d before the next content brief.