I posted to HN, and then my website was flooded with AI bot traffic
A HN post on a feed reader drew heavy AI scraper traffic, with one service accounting for nearly half of requests; the author responded with blocking and rate limits. The sample is small and cannot show broader patterns.
Sammy Straus recounts posting his customizable feed reader, sprinklz.io, to Hacker News, where it received about a hundred upvotes. Days later the site slowed under heavy load; logs showed most requests never fetched articles or ran JavaScript, and nearly half of traffic came from one AI newsletter scraper polling many times per minute.
After blocking it and known crawlers, adding robots.txt and rate limits, he plans infrastructure improvements to handle real traffic better.
The bits that matter
- The site is a single-page application; most bot requests did not execute JavaScript or fetch news articles.
- Nearly half of traffic came from one unnamed service that monitors site changes and summarizes them into an AI newsletter.
- Other traffic included known crawlers (Claude, OpenAI, Google) and vulnerability scanners probing paths like /wp-content/phpinfo.php.
- The author added robots.txt allowing search crawlers, rate limits, and used Claude to test vulnerability paths.
- The author is improving infrastructure for higher QPS and better scaling.
Discussion analysis
Analysis details
10 comments analyzed. Analyzed .
The sample selects active discussion branches and includes available parent comments. It may omit parts of the full thread. The analysis included 10 of 20 usable stored comments because of the input limit. Selected themes do not measure community opinion or how common a view is.
Cloud provider abuse and responsibility
A commenter reports blocking major cloud ASNs (AWS, Google Cloud, Azure, Alibaba, Oracle) due to spam, vulnerability scanning, and aggressive scraping, and argues providers must control abuse under their networks. A reply compares the incentive to ad click fraud/scam ads, asking why a paid cloud provider would stop it; another asks whether cloud users therefore avoid automatic IP bans.
Network-level versus application-level blocking
One reply argues application-level banning is inefficient and recommends network/IP-level blocking that silently drops packets. The original commenter replies that such bans are already enforced at Cloudflare's edge and never reach the application.
Collateral impact on legitimate users and detection difficulty
The discussion notes that IP/ASN bans can affect legitimate users who route through cloud services or work VPNs, while residential botnets make abusive scrapers harder to block without collateral damage.
Contribution requirements as an anti-spam alternative
A reply asks what alternative exists and suggests requiring contributions before posting links, but notes this could incentivize bots to post comments just to meet the requirement.
Single-page blog layout and navigation
A commenter is confused that the site appears to be a single page containing all posts, with in-page table-of-contents links and no sitemap, though RSS confirms it. A reply attributes this to org-html-export-to-html.
Withholding the identity of a specific user
One commenter asks for the identity of an unnamed specific user; the other declines to name names.