Tea@programming.dev to Technology@lemmy.worldEnglish · 14 days agoCloudflare announces AI Labyrinth, which uses AI-generated content to confuse and waste the resources of AI Crawlers and bots that ignore “no crawl” directives.blog.cloudflare.comexternal-linkmessage-square128fedilinkarrow-up11.34Karrow-down110cross-posted to: technology@lemmy.ml
arrow-up11.33Karrow-down1external-linkCloudflare announces AI Labyrinth, which uses AI-generated content to confuse and waste the resources of AI Crawlers and bots that ignore “no crawl” directives.blog.cloudflare.comTea@programming.dev to Technology@lemmy.worldEnglish · 14 days agomessage-square128fedilinkcross-posted to: technology@lemmy.ml
minus-squarelily33@lemm.eelinkfedilinkEnglisharrow-up10·14 days ago while allowing legitimate users and verified crawlers to browse normally. What is a “verified crawler” though? What I worry about is, is it only big companies like Google that are allowed to have them now?
minus-squarewingiee@lemm.eelinkfedilinkEnglisharrow-up17·14 days agoI assume a crawler which adheres to robots.txt
minus-squarelily33@lemm.eelinkfedilinkEnglisharrow-up5·14 days agoI would love to think so. But the word “verified” suggests more.
minus-squarekilleronthecorner@lemmy.worldlinkfedilinkEnglisharrow-up2·14 days agoIP verification is a not uncommon method for commercial crawlers
What is a “verified crawler” though? What I worry about is, is it only big companies like Google that are allowed to have them now?
I assume a crawler which adheres to robots.txt
I would love to think so. But the word “verified” suggests more.
IP verification is a not uncommon method for commercial crawlers