Skip to main content

What is a robots.txt file, and what best practices should I consider when updating it?

Updated

This article explains what a robots.txt file is, how it is used by search engines, and how Pulse Commerce merchants can request updates to their site’s robots.txt file in a secure and supported way.

Purpose: The robots.txt file provides instructions to search engine crawlers about which areas of your site should or should not be indexed. It is a critical tool for SEO governance, privacy control, and crawl efficiency.

The following excerpt from Wikipedia provides a widely accepted definition of the robots.txt file:

A robots.txt file on a website functions as a request that specified robots ignore specified files or directories in their search. This might be for privacy from search engine results, to prevent misleading or irrelevant content from being indexed, or to restrict automated applications to certain data.

For websites with multiple sub-domains, each sub-domain must have its own robots.txt file. If example.com has a robots.txt file but a.example.com does not, the rules that apply to example.com will not apply to a.example.com.

The robots.txt file must be placed in the root directory of your website and is accessed by crawlers at:

https://<yourdomain>/robots.txt

Important: For security reasons, Pulse Commerce merchants do not have direct access to the site root directory. All robots.txt changes must be coordinated through Pulse Commerce Support.

To update your site’s robots.txt file, follow the steps below:

  1. Create or modify a robots.txt file based on your needs. Refer to https://en.wikipedia.org/wiki/Robots.txt for syntax examples and directives.
  2. Use FTPS to upload the robots.txt file to your Pulse Commerce site’s FTP directory.
  3. Create a Support Ticket requesting that the uploaded robots.txt file be placed in the site root. Be sure to specify the exact FTP path and filename.
  4. After the change is applied, verify it by visiting: https://<yourdomain>/robots.txt

Once your robots.txt file is live, you should:

  • Confirm the file loads without a 404 or permission error
  • Validate syntax using Google Search Console or third-party robots.txt testers
  • Allow time for search engines to re-crawl and apply updated rules
Tip: An invalid robots.txt file can cause crawlers to ignore all rules, potentially exposing content you intended to restrict.
  • Do not use robots.txt for security: Sensitive content should be protected by authentication, not crawl directives.
  • Avoid blocking critical assets: Do not disallow CSS, JavaScript, or image directories required for proper page rendering unless explicitly necessary.
  • Use noindex sparingly: Prefer meta tags for page-level indexing control when possible.
  • Document changes: Keep a versioned copy of your robots.txt file so changes can be audited or rolled back if SEO issues arise.
  • Coordinate with SEO stakeholders: Always review robots.txt changes with your SEO or marketing team before deployment.
Summary: The robots.txt file helps guide search engine crawlers and manage how your Pulse Commerce site is indexed. Because the file must reside in the site root, all updates require FTPS upload followed by a support request. Careful planning and validation are essential to avoid unintended SEO or visibility issues.

Related articles