get_robotstxt           downloading robots.txt file
guess_domain            function guessing domain from path
parse_robotstxt         function parsing robots.txt
paths_allowed           check if a bot has permissions to access
                        page(s)
print.robotstxt         printing robotstxt
print.robotstxt_text    printing robotstxt_text
remove_domain           function to remove domain from path
robotstxt               Generate a representations of a robots.txt file
rt_cache                get_robotstxt() cache
