Care About Search Rankings? You Must Know These Things About Your Robots File.
Did you know every website should have its own robots.txt file? It's true.
A robots file is the very first thing search engine spiders look for when they visit a website. The very first thing.
What does a robots.txt file do?
A robots.txt file tells Google and the other search engines exactly what to look at and what not to look at in your site. It makes the search engines' jobs easier and more efficient.
What does a robots.txt file not do?
Contrary to what you may have heard, a robots.txt file does not directly boost your search engine rankings.
However, search engines are on the hunt for your robots.txt file when they first arrive at your site, and if they find it missing, they may not pay attention to important parts of your site (or, even worse, they may pay attention to parts of your site you don't want them to see).
There is a specific format that a robots.txt file should follow and unfortunately, they can sometimes be tricky to set up correctly, depending on your website's content and structure. If there are issues in the robots.txt file, important sections of your website can be accidentally blocked from showing up in the search engines.
In a few unfortunate cases, we've even seen entire websites that were completely blocked from showing up in the search engines—all because their robots.txt file had been set up incorrectly.
Does your site have a robots.txt file?
Here's how to check and see if your website has a robots.txt file:
- Visit your website. (For example, we would go to http://www.timeforcake.com)
- After the last portion of your website address (for example: ".com" or ".net"), type in /robots.txt
- Hit Enter on your keyboard
If a white page with some text in the upper left corner is displayed, congratulations! You have a robots.txt file. (Note: you may wish to have your webmaster review your robots.txt file to ensure it's configured properly.)
If your browser returns a 404 Error page or tells you the requested URL cannot be found, you're in trouble: your website doesn't have have a robots.txt file.
My site has no robots.txt file! What do I do?
Don't panic. A robots.txt files can be added very quickly to your site.