Cloudflare has accused artificial intelligence company Perplexity of using hidden tactics to bypass restrictions designed to stop automated bots from collecting website data.
In a statement published Monday, Cloudflare said it had received multiple complaints from its customers claiming that Perplexity was still able to view and collect information from their sites, even though they had taken steps to block its activity. These blocks were implemented through a robots.txt file, a common tool that tells search engine bots which parts of a website they can or cannot access.
According to Cloudflare’s engineers, testing confirmed that Perplexity’s official crawler — the automated system responsible for scanning and indexing web content was being blocked as expected. However, the company claims Perplexity was also using other, less obvious methods to gain access to pages where it was not permitted.
As a result, Cloudflare said it has removed Perplexity from its list of verified bots and updated its own security rules to detect and block what it called “stealth crawling.” The company stressed that trustworthy crawlers should operate transparently, follow site owner instructions, and clearly state their purpose.
This dispute comes shortly after Cloudflare introduced new tools allowing website operators to either block AI crawlers completely or charge them for access. The move is part of a broad
[…]
Content was cut in order to protect the source.Please visit the source for the rest of the article.
Read the original article: