Over the past year, plenty of teams made their content easier for AI assistants to reach. Friendlier robots.txt and fewer bot blocks for the crawlers that feed AI citations. A while back I loosened my own rules for the same reason, to let those crawlers index my work as part of my GEO strategy.
Reading my own access logs this week, I watched someone use that opening to probe. One host worked through a wordlist of config and credential paths, things like /.env and /secrets.json, while rotating its identity across five crawler names: PerplexityBot, ClaudeBot, GPTBot, Perplexity-User, and ChatGPT-User. Same source address the entire time. Real AI crawlers have no reason to ask for your .env file.
Seeing it is a good reminder that the user agent string is just another variable in a script. Anyone can wear an AI crawler's name.
The probes found nothing. No secrets baked into my container images, a WAF in front, and rules on the server that block known probe paths. Cloudflare sits in front of those and filters a baseline of junk before it reaches me.
My next step is changing the gateway to only accept traffic from Cloudflare and drop everything else. Then I can trust the client address Cloudflare passes instead of a name in a header, and client identity rests on something harder to forge.
I am posting this because I keep talking with people on LinkedIn standing up agents and side projects on cloud boxes. More public endpoints worth probing, and everyone is at a different point in locking them down.
AI lowered the barrier to build and deploy, and raised the value of getting security right.
Happy to trade notes with anyone working through the same setup.
Written by Duane Grey
AI Strategy & Implementation
Independent AI consultant helping companies cut through hype and deploy systems that produce real results.