%>%                     re-export magrittr pipe operator
get_robotstxt           downloading robots.txt file
get_robotstxt_http_get
                        get_robotstxt() worker function to execute HTTP
                        request
get_robotstxts          function to get multiple robotstxt files
guess_domain            function guessing domain from path
is_valid_robotstxt      function that checks if file is valid /
                        parsable robots.txt file
parse_robotstxt         function parsing robots.txt
paths_allowed           check if a bot has permissions to access
                        page(s)
paths_allowed_worker_spiderbar
                        paths_allowed_worker spiderbar flavor
print.robotstxt         printing robotstxt
print.robotstxt_text    printing robotstxt_text
remove_domain           function to remove domain from path
robotstxt               Generate a representations of a robots.txt file
rt_cache                get_robotstxt() cache
