Appendix: References
RFCs and existing standards
Crocker, D., "Standard for the Format of ARPA Internet Text Messages", STD 11, RFC 822, UDEL, August 1982, http://www.ietf.org/rfc/rfc822.txt
Berners-Lee, T., others, "Uniform Resource Locators (URL)," RFC 1738, December 1994, http://www.ietf.org/rfc/rfc1738.txt
Fielding, R., "Relative Uniform Resource Locators", RFC 1808, UC Irvine, June 1995, http://www.ietf.org/rfc/rfc1808.txt
Berners-Lee, T., Fielding, R., and Frystyk, H., "Hypertext Transfer Protocol -- HTTP/1.0." RFC 1945, MIT/LCS, May 1996, http://www.ietf.org/rfc/rfc1945.txt
Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels," BCP 14, RFC 2119, March 1997, http://www.ietf.org/rfc/rfc2119.txt
Costello, A., "Punycode: A Bootstring encoding of Unicode for Internationalized Domain Names in Applications (IDNA)", March 2003 , http://www.ietf.org/rfc/rfc3492.txt
Berners-Lee, T., Fielding, R., et al, "Uniform Resource Identifier (URI): Generic Syntax", January 2005, http://www.ietf.org/rfc/rfc3986.txt
IANA, "Well Known Ports, the Registered Ports, and the Dynamic and/or Private Ports", last updated March 2010, http://www.iana.org/assignments/port-numbers
Unicode Inc., "Unicode FAQ: UTF-8, UTF-16, UTF-32 & BOM", last update February 2010, http://www.unicode.org/faq/utf_bom.html
Previous robots.txt documents
Koster, M., "About /robots.txt", http://www.robotstxt.org/robotstxt.html
Koster, M., "Internet draft: A Method for Web Robots Control", November 1996, http://www.robotstxt.org/norobots-rfc.txt
Koster, M., "A Standard for Robot Exclusion", June 1994, http://www.robotstxt.org/orig.html
Conner, S., "An Extended Standard for Robot Exclusion", November 2002, http://www.conman.org/people/spc/robots2.html
Shared robots.txt extensions
Tools
Other implementations
Yahoo, Inc. "How to Prevent Your Site or Certain Subdirectories From Being Crawled", last updated February 2010, http://help.yahoo.com/l/us/yahoo/search/indexing/slurp-02.html
Yahoo, Inc. "How to Reduce the Number of Requests the Yahoo! Search Web Crawler Makes on Your Site", last updated February 2010, http://help.yahoo.com/l/us/yahoo/search/indexing/slurp-03.html
Bing, "robots speaking many languages", November 2009, http://www.bing.com/toolbox/blogs/webmaster/archive/2009/11/05/robots-speaking-many-languages.aspx
Bing, "prevent a bot from getting "lost in space" (sem 101)", August 2009, http://www.bing.com/community/blogs/webmaster/archive/2009/08/21/prevent-a-bot-from-getting-lost-in-space-sem-101.aspx
Bing, "robots exclusion protocol: joining together to provide better documentation", June 2008, http://www.bing.com/community/blogs/webmaster/archive/2008/06/03/robots-exclusion-protocol-joining-together-to-provide-better-documentation.aspx
Bing, "more crawling improvements from msnbot", April 2008, http://www.bing.com/community/blogs/webmaster/archive/2008/04/18/ramping-up-msnbot.aspx
Bing, "crawl delay and the bing crawler, msnbot", August 2009, http://www.bing.com/toolbox/blogs/webmaster/archive/2009/08/10/crawl-delay-and-the-bing-crawler-msnbot.aspx
Ask, Inc, "The Ask Website Crawler FAQ", (unknown date), http://about.ask.com/en/docs/about/webmasters.shtml
Other references
Apache Software Foundation, "Apache Core Features", 2009, http://httpd.apache.org/docs/2.0/mod/core.html
Crow, D., "Robots Exclusion Protocol: now with even more flexibility", July 2007, http://googleblog.blogspot.com/2007/07/robots-exclusion-protocol-now-with-even.html
W3C, Ragett, D., Le Hors, A., Jacobs, I., "HTML 4.01 Specification", December 1999, http://www.w3.org/TR/html401/cover.html
Wikipedia et al., "Robots exclusion standard", last updated February 2010, http://en.wikipedia.org/wiki/Robots_exclusion_standard
Wikipedia et al., "Robots meta tag", last updated August 2010, http://en.wikipedia.org/wiki/Robots_meta_tag#The_robots_attribute
