A robots.txt file is a text document that contains instructions for search engine crawlers. It shows search engines which pages of your website they are allowed to access and which they are not.
How to add a robots.txt file to your website
- Go for Settings and select Settings
- Select the Files tab
- Check the Enable robots.txt checkbox
- Insert your script and confirm with Apply