To get found on Claude as a real estate agent, Anthropic's search crawlers have to be allowed to read your site, your pages have to be findable in the search results Claude draws on, and your pages have to state your facts plainly enough to quote. Claude does not sell placement. This answer explains how Claude's web search works, the three Anthropic crawlers and which ones matter for search, and the steps we take on our own sites.
We run AI-crawler tracking on our group's sites at The Kenna Group at Keller Williams DTC and review which bots read which pages each week. What follows is Anthropic's published guidance plus what we see. Our methods are labeled as ours.
How Claude answers a question about agents
Anthropic added web search to Claude on March 20, 2025, first for paid users in the United States and then more widely. When a question needs current information, such as "who are the real estate agents in Highlands Ranch" or "what are Parker home prices this month", Claude runs a search, reads the pages, writes an answer and cites the sources inline.
In March 2025 Anthropic added Brave Search to its list of subprocessors, with a note pointing to its web search help page. Developer Simon Willison tested Claude's citations against Brave's results for the same query and found they matched. TechCrunch reported the same evidence on March 21, 2025. Anthropic has not published a full description of how Claude picks sources, so treat the Brave link as strong evidence, not a complete explanation.
The three Anthropic crawlers
Anthropic's help center article on its crawlers describes three user agents. Each one is controlled separately in robots.txt:
- ClaudeBot collects public web content that could be used to train Anthropic's models. Blocking it tells Anthropic to leave your future content out of training data.
- Claude-SearchBot navigates the web to improve the quality of Claude's search results. Anthropic says blocking it can reduce your site's visibility in those results.
- Claude-User fetches a page when a person asks Claude a question that needs it. Anthropic says blocking it can reduce your visibility for user-directed searches.
Anthropic says its bots honor robots.txt, support the non-standard Crawl-delay rule, and will not try to bypass CAPTCHAs. Rules have to be set in the top-level robots.txt of each subdomain. Anthropic advises against blocking by IP address, because that can stop its bots from reading your robots.txt at all.
The decision for an agent
Our opinion: an agent who wants to be found by Claude allows Claude-SearchBot and Claude-User. Whether to allow ClaudeBot is a separate question about training, and reasonable people choose differently. The mistake we see most often is a blanket "block all AI bots" setting, added by a security plugin or a hosting company, that also blocks the two search bots. The agent never chose it and does not know it is there.
Open yoursite.com/robots.txt. Look for any "User-agent" line naming Claude-SearchBot, Claude-User or ClaudeBot, and for a "User-agent: *" line followed by "Disallow: /". Then check your firewall or CDN settings. Some services block AI crawlers at the network level, which robots.txt does not show.
Steps to get found on Claude
1. Allow Claude-SearchBot and Claude-User
Fix robots.txt and any firewall rule first. A page Claude's search cannot read is not a candidate for an answer.
2. Check how you show up in Brave Search
Search your name and "[city] real estate agent" on search.brave.com. If your site does not appear for your own name, fix that before anything else: clear title tags, your name in the first sentence, a sitemap, and links to your site from your Google Business Profile, brokerage roster and directory profiles.
3. State your facts in the first 100 words
"Brian Lee Burke is a real estate agent in Highlands Ranch, Colorado, with The Kenna Group at Keller Williams DTC, licensed since 2003." Name, city, role, brokerage and a verifiable fact, in one sentence. Claude quotes sentences like that.
4. Publish dated local numbers and cite where they came from
Claude's answers cite sources inline, so a page that cites its own sources is easy to build on. A line in this format: "Median list price in Littleton: [price], from [your MLS] active listings, updated [date]." On our own sites, every market number rebuilds daily and shows its date.
5. Write one section per question
People ask Claude long, specific questions. "What does a metro district add to a tax bill in Castle Rock?" "How long do homes take to sell in Lone Tree in winter?" Give each a heading that matches the question and a short answer with a number. Mark them up with FAQPage schema that matches the page.
6. Keep your name, phone and brokerage identical everywhere
When Claude finds your site, your Zillow profile and your brokerage roster, they need to agree. Different phone numbers or an old brokerage name give it a reason to leave you out.
7. Watch your server logs
Your hosting logs record each visit's user agent. Count visits from Claude-SearchBot and Claude-User each week and note which pages they read. On our sites, the pages AI crawlers read most are the ones with current numbers and direct answers. That tells us what to build next.
How to check whether Claude mentions you
Ask Claude, with web search on, the ten questions your clients ask about your city. Save each answer and its citations. Repeat on the same day each month. Track three things: whether you are named, whether your pages are cited, and whether Anthropic's search crawlers visited your site that month.
AI Answer Watch does that for you. We track 20 prompts across five assistants, including Claude, every month, report which AI crawlers read your site (GPTBot, ClaudeBot, PerplexityBot and Google-Extended among them), and make two fixes a month. It is $149 a month and in beta. It tracks and improves your chances. It does not promise a mention.
Unlock the full playbook: members only, $99/yrThe rest of this answer, every other playbook and the member prompt library. Included with every Pad Page™.
Member sign-in opens at launch.
AI Answer Watch
Monthly tracking of whether AI assistants mention you, plus who reads your site.
$149/mo
See AI Answer WatchQuestions
Does Claude search the web?
Yes. Anthropic added web search to Claude on March 20, 2025. When a question needs current information, Claude searches, reads pages and cites its sources in the answer.
Which Anthropic bots should a real estate agent allow?
Allow Claude-SearchBot, which indexes pages for Claude's search results, and Claude-User, which fetches pages when a person asks Claude a question. ClaudeBot collects pages for model training; blocking it is your choice and is separate from search.
Does Anthropic follow robots.txt?
Anthropic says its bots honor standard robots.txt directives, support the Crawl-delay extension, and do not try to get around CAPTCHAs. Rules must be set on each subdomain.
What search index does Claude use?
In March 2025 Anthropic listed Brave Search as a subprocessor for Claude's web search, and independent testing matched Claude's citations to Brave results. Anthropic does not publish a full description of how sources are chosen.
Brian Lee Burke is the founder of PadLeaders, a Colorado real estate broker licensed since 2003, team leader of The Kenna Group at Keller Williams DTC (formerly Kenna Real Estate Group) and the author of Mastering Real Estate and The Real Estate Playbook. More about Brian.
- Anthropic: Claude can now search the web, March 20, 2025
- Anthropic Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler? (ClaudeBot, Claude-User, Claude-SearchBot), accessed October 2026
- Simon Willison: Anthropic used Brave Search for Claude's web search, March 21, 2025
- TechCrunch: Anthropic appears to be using Brave to power web search for its Claude chatbot, March 21, 2025
- Schema.org: FAQPage
- PadLeaders AI-crawler log review, kennarealestate.com, October 2026