You assume that if buyers are asking AI tools who the best agents nearby are, your years in the business and your reviews are at least in the running. A setting left behind by whoever last built your site may be telling every crawler to stay out, and nothing about that looks broken from where you are sitting.
On a podcast, an agent who runs a team and coaches on AI content gave the one-step test: type your own website address, add /robots.txt to the end, and look at what comes up. "What you see in the top left corner tells you whether AI is allowed to crawl your site at all."
Why it happens: "A few years ago, blocking bots was standard practice." Plenty of agent sites still carry that setting from a redesign years ago, telling AI systems to stay out, and they do.
His framing of what these systems actually are: "So we think that AI is this magical brain that has its own thought process. That's not the case. What AI is doing is pulling from what it finds online. And if it can't go to your website online, that's a massive problem."
The second piece is schema code, which is what AI reads to understand who you are; without it, a crawler that does get in still has nothing to work with. The third is format: "Most of us that I've seen with real estate websites is they have widgets installed on their page or they have photos with the reviews baked into the photo. AI can't read the photos. So all of this is plain text." An agent with five reviews in plain text can rank ahead of one sitting on hundreds inside a widget.
And the move nobody thinks of, which is to ask the system itself: "I asked AI, why is this person ranking, and why am I not ranking? You're talking to the thing that's ranking you. So ask the questions."
“Type your website’s URL followed by **/robots.txt** and see what comes up.”
What this costs you is invisible by design. A blocked site loads perfectly for you, ranks in Google, converts your own traffic, and simply does not exist for the tool a buyer opens before calling anybody. There is no error and no report. The case in the episode is an agent under four years in with 30 Google reviews and a site built less than a month earlier, ranked fourth among top agents on the largest real estate board in North America — ahead of a ten-year veteran with 195 reviews and a website he had not touched in seven years.
What changes is a ten-minute audit you can run today. Check the file, then read your own homepage the way a stranger would: does it say who you help and where within the first few seconds, are your phone, text and email visible without digging, and are your reviews and credentials in plain text rather than trapped in a widget or an image. The format decides whether any of it counts.