How to block AI web crawlers on Squarespace
Did you know that AI (artificial intelligence) crawlers can scan your website? It didn’t occur to me either until recently. Certain third-party web crawlers automatically scan the internet and publicly available websites.
Sometimes this is really important, such as with search engine crawlers. If you want your website pages to be indexed and therefore able to appear in search, search engine crawlers need to be able to scan your website. But, what about AI companies who can collect and use the content of your website to train their models? That’s your call too, but as this setting is switched on by default you may not even realise it’s your decision to make.
The use of AI and the ethical implications of this technology, is something that we will need to grabble with as this technology develops at a breathtakingly rapid pace. Lawsuits have already been filed, including the New York Times suing OpenAI and Microsoft, and a growing number of artists, authors and other creatives are also voicing their concerns of their work being used in this way.
If you are concerned about the ethical and legal implications of AI web crawlers, as I am, then blocking AI crawlers from scanning your website in the future is a step you can take.
Here I walk you through how to exclude your Squarespace website from any future AI crawler scans, in a few easy clicks.
Turn off the AI Crawler setting on your website
Log into Squarespace and go to your website
Select ‘Settings’ towards the bottom of the left-hand menu
Scroll down and click ‘Crawlers’
Next you’ll see two options ‘Search Engine Crawlers’ and ‘Artificial Intelligence Crawlers’
Make sure that the ‘Search Engine Crawlers’ option is toggled on (showing as green) if you want your website to come up in search, such as on Google
Then click the ‘Artificial Intelligence Crawlers’ toggle changing it from green (on) to black (off)
Hit ‘SAVE’ in the top right corner
Taking this step now won’t remove any content previously collected, but it will prevent any future AI crawls.
