Tea to Technology@lemmy.worldEnglish • 29 days agoCloudflare announces AI Labyrinth, which uses AI-generated content to confuse and waste the resources of AI Crawlers and bots that ignore “no crawl” directives.blog.cloudflare.comexternal-linkmessage-square178fedilinkarrow-up11arrow-down10cross-posted to: technology@lemmy.ml
arrow-up11arrow-down1external-linkCloudflare announces AI Labyrinth, which uses AI-generated content to confuse and waste the resources of AI Crawlers and bots that ignore “no crawl” directives.blog.cloudflare.comTea to Technology@lemmy.worldEnglish • 29 days agomessage-square178fedilinkcross-posted to: technology@lemmy.ml
minus-square@wingiee@lemm.eelinkfedilinkEnglish0•29 days agoI assume a crawler which adheres to robots.txt
minus-square@lily33@lemm.eelinkfedilinkEnglish0•29 days agoI would love to think so. But the word “verified” suggests more.
minus-square@killeronthecorner@lemmy.worldlinkfedilinkEnglish0•28 days agoIP verification is a not uncommon method for commercial crawlers
I assume a crawler which adheres to robots.txt
I would love to think so. But the word “verified” suggests more.
IP verification is a not uncommon method for commercial crawlers