Except that's not what those articles say. Or they are at least quick to clarify, "adherence to a robots.txt is entirely voluntary." You're not "blocking" anything.According to many of the articles found via https://www.google.com/search?q=block+ai+bots you can indeed block AI bots using robots.txt.
It's the equivalent of putting your valuable data out on a public sidewalk with a robots.txt Post-It note on it which says "Please do not steal this data."
You'll get people who don't steal it, people who tell you they didn't steal it but they actually did, and people who never even looked at the note while they stole the data.
Statistics: Posted by EA117 — Fri Jun 07, 2024 3:13 am