Having a good Robots.txt file is an important part of optimizing your website with search engines. The problem is that it can take two weeks for the search engines to reflect your changes to your Robots.txt file in their search results, so it is important to know if you’ve made correct changes as soon as possible.

Google has a service called Google’s Webmaster Central that allows you to check and see if your Robots.txt file is valid. If you haven’t used this service before, you can sign up for a free account and submit your website to it. Once you’ve submitted your URL, you can instruct it to go grab your Robots.txt file and validate it by going to the Tools section on the left and selecting Robots.txt Analysis. Click Check and it will give you an Allowed or Blocked message. Allowed means it was able to crawl, Blocked means it couldn’t.

Here is how my Robots.txt file currently looks:

User-agent: *
Disallow: /wp-content/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /cgi-bin/
Disallow: /adlogger/
Disallow: /banner/
Disallow: /dreamdiary/
Disallow: /projects/
Disallow: /statistics/
Disallow: /theme_backup/
Allow: /wp-skins/
User-agent: Googlebot
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /*.php*
Disallow: */trackback*
Disallow: /*?*
Disallow: /z/
Disallow: /wp-*
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.txt$
Allow: /wp-content/uploads/
User-agent: Googlebot-Image
Allow: /*

This one shows as Allowed. :)