Should you block AI web crawlers from your Squarespace website?
Did you know that by default all web pages are are accessible to website crawlers, including AI (artificial intelligence) ones. Certain third-party web crawlers automatically scan the internet and publicly available websites.
I believe we should all have a choice, but you also need to know what you are choosing. By opting out of AI crawlers you could also be opting out of your business being found in AI results and therefore reducing visibility and website traffic. By opting in, your website content may be used to train AI models, which I know poses an ethical and legal issue for a lot of people, especially those in the creative sectors.
The use of AI and the ethical implications of this technology, is something that we will need to grabble with as this technology develops at a breathtakingly rapid pace. Lawsuits have already been filed, including the New York Times suing OpenAI and Microsoft, and a growing number of artists, authors and other creatives are also voicing their concerns of their work being used in this way.
If you are concerned about the ethical and legal implications of AI web crawlers, then blocking AI crawlers from scanning your website in the future is a step you can take and I’m happy to show you how. But it should be an informed decision as it can impact the visibility of your business.
Why I stopped blocking AI crawlers
Back in April 2024, when I first wrote a version of this blog post I blocked AI crawlers from my website too. But this landscape is changing quickly, and during getting my SEO accreditation with SEOSpace I realised that if I block AI crawlers from my website it also means my website and my business won’t be surfaced in AI search results or recommendations.
And while Google is still the main search platform this might not seem a problem, but it is shifting and more and more people are turning to AI platforms such as Chat GPT and Perplexity for search queries and information. On top of that Google’s response has been to release it’s AI Overview feature (now live in the UK), so this is what now appears at the top of a lot of Google’s search results.
All said and done I am still grappling with the bigger ethical and environmental implications of AI, but making my business harder to discover is counterintuitive to my mission of helping good businesses doing good things become THE destination for what they do so well. My viewpoint may change again in the future, but importantly I wanted you to make an informed decision on what is best for your business.
How to turn off the AI Crawler setting on your website
If you’re aware of the implications and you still want to block AI crawlers from your Squarespace website then here I walk you through how to exclude your Squarespace website from any future AI crawler scans, in a few easy clicks.
Log into Squarespace and go to your website
Select ‘Settings’ towards the bottom of the left-hand menu
Scroll down and click ‘Crawlers’
Next you’ll see two options ‘Search Engine Crawlers’ and ‘Artificial Intelligence Crawlers’
Make sure that the ‘Search Engine Crawlers’ option is toggled on (showing as green) if you want your website to come up in search, such as on Google
Then click the ‘Artificial Intelligence Crawlers’ toggle changing it from green (on) to black (off)
Hit ‘SAVE’ in the top right corner
Taking this step now won’t remove any content previously collected, but it will prevent any future AI crawls.
