Cloudflare Debuts Bot Blocker to Help ‘Internet Survive Age of AI’

Software firm Cloudflare has introduced a tool to block bot crawlers from accessing web content without permission.

The new offering, announced Tuesday (July 1), lets website owners decide if they want artificial intelligence (AI) crawlers to access their content, and determine how AI firms can use it. It also lets site owners set a price for access via a “pay per crawl” model.

“For decades, the Internet has operated on a simple exchange: search engines index content and direct users back to original websites, generating traffic and ad revenue for websites of all sizes,” the company said in a news release. “This cycle rewards creators that produce quality content with money and a following, while helping users discover new and relevant information.”

But that model, Cloudflare contended, is broken, with AI crawlers collecting things like words and images to generate answers without sending visitors to the initial source, robbing creators of revenue and the satisfaction of knowing someone is viewing their work.

“If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone — creators, consumers, tomorrow’s AI founders, and the future of the web itself,” said Matthew Prince, co-founder and CEO of Cloudflare.

“Original content is what makes the Internet one of the greatest inventions in the last century, and it’s essential that creators continue making it,” Prince added. “AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate.”

Writing about this issue last year, PYMNTS noted the significant financial implications of content scraping, as each company invests heavily in researching, writing and publishing website content. Experts argued that allowing bots to scrape this material freely undermines this work while leading to derivative content that potentially outranks the original on search engines.

“Beyond content theft, scraping can have detrimental effects on website performance,” that report said. “Unchecked bot activity may overload servers, slow down websites and skew analytics data, potentially increasing operational costs. These consequences underscore the urgency of many content providers implementing robust protective measures.”

All the same, that report said, experts have been divided on effectiveness of new anti-scraping tools, with some cautioning that their track record is still unproven, and others more optimistic about their potential.

At the time, Cloudflare had just introduced another tool to fight AI-data harvesting, which Pankaj Kumar, CEO of Naxisweb, acknowledged in an interview with PYMNTS.

“Its purposeful blockage focuses exclusively on AI bots so that people can still visit the site or search engine robots can continue to crawl it. Search engine optimization (SEO) performance is not compromised, while unauthorized scraping is prevented by selective blocking,” Kumar said.

The post Cloudflare Debuts Bot Blocker to Help ‘Internet Survive Age of AI’ appeared first on PYMNTS.com.