2023-05-10 23:59:57 by Mark Davies | Files touched by this commit (4) |
Log message: py-robot-detection: add version 0.4 robot_detection is a python module to detect if a given HTTP User Agent is a web crawler. It uses the list of registered robots from http://www.robotstxt.org: (Robots Database) |