You must log in or # to comment.
In my experience, the single biggest bully on the internet are the servers controlled by Meta which in my experience literally perform DDoS attacks whilst crawling, hitting sites several orders of magnitude more than all the others combined.
Actively blocking them was the only option left.
Are there any good log monitoring programs that will automatically blacklist the IP of any crawler that ignores robots.txt?
Yeah, I’ve been curious if you could explicitly block a page in robots.txt, hide an invisible link to the same page in your footer, then kinda have it act like an immediate IP block when someone requests it.
deleted by creator
