TL;DR: Open yoursite.com/robots.txt in a browser and look for AI crawler names, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot above a “Disallow: /” line. If they’re there, your WordPress site is invisible to AI search, almost certainly because a security or SEO plugin added the blocks without asking you. The 2-minute check, the three places blocks hide on WordPress, and the fix, below.
The 2-minute check
No tools, no developer. Open a browser tab and type your domain followed by /robots.txt for example, yoursite.com/robots.txt. You’ll see plain text rules. Scan for these names:
txt
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Google-ExtendedNow check what follows each. “Disallow: /” under a name means that bot is banned from your entire site. “Allow: /” or no mention means it’s welcome. Blanket rules matter too: a User-agent: * followed by Disallow: / bans everything, AI and Google alike (rare, but we’ve seen it live on real Pakistani business sites after a botched migration).
Two minutes, and you know. If you found blocks, keep reading, because on WordPress, deleting them is not as simple as editing a file.
Why “just edit the file” doesn’t work on WordPress
Here’s the trap that catches nearly everyone: on most WordPress sites, robots.txt isn’t a file at all. WordPress generates it on request, and your plugins inject rules into it. Edit or upload a physical file and a plugin may override or regenerate it on the next update, the blocks quietly return, and you never think to look again.
So the fix means finding which plugin wrote the rules. Three places, in order of likelihood:
1. Your security plugin. Wordfence, All-In-One Security, Solid Security and similar tools increasingly ship “block AI crawlers” or “prevent AI scraping” toggles, some enabled by default after updates. Open the plugin’s settings, search for “AI,” “bots” or “crawlers,” and switch blocking off (or configure it per the next section).
2. Your SEO plugin. Rank Math, Yoast and AIOSEO include robots.txt editors, and some added one-click AI-blocking features. Check the robots.txt editor screen; whatever it shows is close to what your site serves. Make your changes here so they persist.
3. Cloudflare (if you use it). This one hides outside WordPress entirely: Cloudflare’s dashboard has a “Block AI bots” toggle and Bot Fight Mode, either of which can reject AI crawlers with a perfect robots.txt, the block happens before your site is even asked. Check Security settings, and glance at the firewall events log for the bot names above.
After any change, re-open yoursite.com/robots.txt in a fresh tab (or incognito, to dodge caching) and confirm the served version actually changed.
What should the rules say?
For a business that wants customers, our recommendation is the simple one: allow everything. The bots named “SearchBot” and their kin are what place you inside live ChatGPT, Perplexity and Copilot answers; blocking them removes you from the fastest-growing recommendation channel in existence while protecting almost nothing, since your public site is already public.
If you have a considered reason to opt out of AI training while staying visible in AI answers, block only GPTBot and Google-Extended and explicitly allow OAI-SearchBot, ChatGPT-User and PerplexityBot, the full reasoning and copy-paste configurations are in our companion piece, should you block AI crawlers?.
Put it on the calendar
One habit turns this from a fix into insurance: re-run the 2-minute check after every plugin update, or monthly if that’s easier to remember. The most common way Pakistani businesses lose AI visibility isn’t a decision it’s a security plugin update silently re-enabling blocks months after someone fixed them. Thirty seconds of glancing at robots.txt is the cheapest monitoring you’ll ever do; catching a regression in week one instead of quarter three is worth a season of customers.
Frequently Asked Questions
How do I check if my WordPress site blocks AI bots?
Visit yoursite.com/robots.txt in a browser and look for AI crawler names, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot above “Disallow: /” lines. Then check your security plugin, SEO plugin and Cloudflare settings, since on WordPress those tools generate the rules.
Which plugin adds AI blocking to WordPress?
Most commonly security plugins (Wordfence, All-In-One Security, Solid Security) via “block AI crawlers” features, and some SEO plugins through their robots.txt editors. Cloudflare’s dashboard toggle can also block AI bots independently of any plugin.
Should I unblock AI bots on my business website?
Yes, if being found by customers is the goal, AI search bots power the live answers where people now ask for recommendations. At most, block only the training crawlers (GPTBot, Google-Extended) and keep the search bots allowed.
The 2-minute check is question one of twelve
Crawler access is the first gate; eleven more factors decide whether AI can read, trust and recommend you. DigiMSM, Pakistan’s 1st AI-powered digital marketing agency, checks all of them free in an AI Visibility Audit. WhatsApp +92 335 4155677 or digimsm.com/contact-us/.
Published by the DigiMSM editorial team, Islamabad. Last updated July 2026.