Blog of King Software

Learn online with us in simple steps.

Robots.txt file In magento


Created by Bhavesh Gohil & Dhruti Patel on Date: 05/07/2021

    Your Magento 2 store has a variety of URL patterns that you don’t want search engines to touch.  : 

  • Here are the pages we recommend barring off from search engines:
  • Internal search result pages: /catalogsearch/ and /search/. While we recommend adding the robots noindex directive (either the meta tag or HTTP header) to these pages as a second line of defense, the first line of defense is the robots.txt file.
  • Login pages /customer/account/login/ – login pages.
  • URLs containing SID and PHPSESSID parameters.
  • Reference the XML sitemap in the robots.txt file.
  • Filtering criteria such as price, color, material and size.
  • This boils down to the following robots.txt:
  • User-agent: * Disallow: /catalogsearch/
    Disallow: /search/
    Disallow: /customer/account/login/
    Disallow: /*?SID=
    Disallow: /*?PHPSESSID=
    Disallow: /*?price=
    Disallow: /*&price=
    Disallow: /*?color=
    Disallow: /*&color=
    Disallow: /*?material=
    Disallow: /*&material=
    Disallow: /*?size=
    Disallow: /*&size=
  • Editing your robots.txt file in Magento 2

  • In order to edit your robots.txt file in Magento 2, navigate to:
  • STEP 1: Content > Configuration (Under Design).

    STEP 2: There, edit the settings for Main Website instead of the Default Store View, because the latter won’t allow you to change the robots.txt file.


    SEO & DIGITAL MARKETING IN BARKING, Essex , London.-Kingsoftware

  • On the screen that follows, scroll down to Search Engine Robots. There you can define your robots.txt content:

  • SEO & DIGITAL MARKETING IN BARKING, Essex , London.-Kingsoftware

  • XML sitemap: limits and robots.txt
  • Things get interesting down below; under Sitemap File Limits, you’ll find the fields Maximum No of URLs Per File and Maximum File Size.

  • SEO & DIGITAL MARKETING IN BARKING, Essex , London.-Kingsoftware

    XML sitemap: Maximum No of URLs Per File

  • For most Magento stores, the default Maximum No of URLs Per File limit is correct as it aligns with SEO best practices.
  • XML sitemap: Maximum File Size

  • You can raise the maximum file size to 52428800, as this equals 50 MB.
  • XML sitemap: Enable Submission to Robots.txt

  • Once this is all set up, it’s time to create the XML sitemap and configure its (re)generation settings. To do this, go to Marketing > Site Map (under SEO & Search) and click the orange Add Sitemap button.

  • SEO & DIGITAL MARKETING IN BARKING, Essex , London.-Kingsoftware

    In the screen that follows, you define the XML sitemap’s name and location: : 

    STEP 1: Fill in the XML Sitemap’s name in the field Filename, example: sitemap.xml.

    STEP 2: Then fill in the XML sitemap’s path in the Path field – for example /pub/ in our case because of the nature of the Magento installation in our test setup.

    STEP 3: Next, click the Save & Generate button, and you’re all done!


    SEO & DIGITAL MARKETING IN BARKING, Essex , London.-Kingsoftware