docs
  1. Scayle Resource Center
  2. Sitemap
  3. Sitemap Add-On
  4. Manage Robots.txt

Manage Robots.txt

A robots.txt file is a text file that is placed on websites to provide instructions to crawlers. These instructions tell search engines which parts of the website should be crawled and indexed and which parts should be excluded from indexing.

Create robots.txt

  1. Go to Add-Ons > Sitemap Add-On > Robots.txt.
  2. Click on Add Robots.
  3. Select shop country for which the robots.txt should be generated.
  4. Enter the robots.txt content.
  5. Click SAVE.

Create Robots.txt file

Edit robots.txt

Click on the edit icon.

Delete robots.txt

Click on the trash can icon.

Edit/delete robots.txt file