What is a robots.txt file and how do I change it for my site?

The following excerpt from Wikipedia.com provides a good definition of the robots.txt file:

A robots.txt file on a website will function as a request that specified robots ignore specified files or directories in their search. This might be, for example, out of a preference for privacy from search engine results, or the belief that the content of the selected directories might be misleading or irrelevant to the categorization of the site as a whole, or out of a desire that an application only operate on certain data.

For websites with multiple sub-domains, each sub-domain must have its own robots.txt file. If example.com had a robots.txt file but a.example.com did not, the rules that would apply for example.com will not apply to a.example.com.

The robots.txt file must be placed in the root of your store. Since merchants do not have access to the site root for security reasons.

To update the robots.txt file: 

1. Create the robots.txt file as needed for the site. See http://en.wikipedia.org/wiki/Robots.txt for examples.
   
   
2. Use FTP to upload the file to your Pulse Commerce site's FTP directory.
   
   
3. Reach out to your Account Manager to request that the file be added to your site root. Specify the location of the robots.txt file uploaded via FTP.
   
   
4. You can check your site's current robots.txt file via http://<mydomain>.com/robots.txt
   Replace <mydomain> with your site's domain name.