10 ms·
What happened to TheNumbers.com
- draw_down 3mo agoSorry! Just yesterday many of us decided that AI scraping isn't a real problem, and anytime it is blamed, it's a cover for something else. https://news.ycombinator.com/item?id=49005747 https://news.ycombinator.com/item?id=49005747
- dumberquestions 3mo agoIronic that polymarkets were being advertised as helping society make better predictions.
- jambalaya8 3mo agoJohn Brunner and Alvin Toffler both made stark warnings wrapped in futuristic giddiness about things like it. People like Fuller no doubt thought polling on a large scale was terrific. There were old usenet groups and BBS subs (minus the money aspect) experimenting with the model. I do not believe they ever are or were good in a largescale model (money or not).
- mrandish 3mo agoI mean wisdom of crowds, super-forecasters, calibration and pre-registration are useful tools that can result in better predictions, turning it into online gambling is where it went sideways.
- Terr_ 3mo agoMini-rant: Promoters claim that the system serves a public good by helping society discover/converge on useful truths sooner. Even where the question/event is of public interest, the opposite happens instead. People with expertise or non-public information are incentivized to misdirect and delay as long as possible, as that maximizes what they can make from betting.
- ethagnawl 3mo agoAt the risk of oversimplifying things from a distance, this site -- especially the free, public-facing part of it -- seems like it would be an ideal candidate for a rewrite using static site generator/framework. That, coupled with a bot-aware CDN should keep them online at a reasonable cost for many years to come. Otherwise, I'm very curious to know more about their old and new architecture and what sorts of mitigation/scaling strategies they've started using to keep the site online.
- tengada1 3mo agoYeah this seems like one of the easiest access models to create an excellent security model for -- basically just static publishing. Sounds like they were in need of a rewrite anyway! It probably seems daunting but to be honest this feels like a weekend's work at this point with LLM assistance. Not to be glib!
- jaredwiener 3mo agoBut it also destroys the business model behind the site. Sure, you could technically redesign to handle the bot traffic, but if the bot traffic is just taking the data and reducing any need for humans to visit the site, why is he putting in the effort to maintain the site?
- hyperhello 3mo agoWell, why is he? If it's for ad revenue, this problem is surely global to the web. If it's because he likes to do it, it shouldn't matter if one AI or a trillion scrape the site. I feel like we badly need a rethink of the web architecture after DoubleClick anyway. Maybe the name of the game should be to cut down a site's assets very hard and use static hosting for them. This interferes with crummy sites that show a mess of inlined ads every refresh, but now that there are a billion 'poor users' this is no longer feasible.
- jaredwiener 3mo agoMaybe more an inherent problem with these chatbots. LLMs are great, but they aren't producing new information. You still need people for that. But if you cut down any incentive for the people to do that, the LLMs will starve.
- brcmthrowaway 3mo agoIts just too easy for a technically minded bored person to produce slop that hammers websites. There needs to be a penalty for this.
- hyperhello 3mo agoIf the scrapers are going to get it anyway, put the data up as a zip somewhere.
- jjgreen 3mo agoIt would not make any difference.
- codemonkey-zeta 3mo agoIndeed, the article mentions Wikipedia experiencing similar scraping pains, even though they already DO have bulk data available.
- HeatrayEnjoyer 3mo agoWho are running these bots? I presume developers at all of the frontier labs know (or at least would know to look for) Wikipedia has bulk APIs for automated access. Unnecessary scraping increases their workload/costs too, so why in 2026 is this still a problem?
- esseph 3mo agoBlack market and gray market data. All the firms want data. All the other firms want data. The banks want data. The other criminals also want data for their crimes and schemes. Oh insurance companies, and the ATS systems. Everybody wants as much data as they can get and they don't care how they get it.
- esseph 3mo agoThis data selling also happens with leaks of all kinds like medical data, often to current or future employers, health insurance companies, etc.
- antisthenes 3mo ago> Read the Docs, a non-profit that hosts documentation for open-source software, who watched a single crawler download 73 terabytes of zipped HTML in one month, costing it over $5,000 in bandwidth From the article. Not the same site, but an example of the same issue.
- abetusk 3mo agoI think people are missing one of the points of the article. It's not just that agents are hammering the site, it's that there might be lurking vulnerabilities that allow malicious usage, which is why it went down, then came back up with a fraction of the data and a reduced design. The article says (speculates?) that malicious users are trying to get privileged access for an edge in prediction market betting. From the article: > If you could see The Numbers data before everyone else, every single week, you would have a significant edge over all the other traders - learning the answers slightly ahead of publication would allow you to front-run the trades.
- squidproquo 3mo agoIf it's the case that prediction market traders were trying to front-run the market outcomes, the prediction markets should put the fix on their end by locking down trading before a market closes with some time buffer to prevent this.
- blurgo 3mo ago[dead]
- eru 3mo agoWhy should that be necessary? You can unilaterally stop trading 'before a market closes with some time buffer to prevent this.' No need for centralised action.
- brookst 3mo agoBecause markets succeed or fail based on perceived fairness. It is in a prediction market’s best interest to not become the place where you go to get fleeced. Today they’re the Wild West and run like the early days of darknet markets, but if the
- fhdkweig 3mo agoCould you please rephrase your last sentence? I appears to be incomplete.
- jambalaya8 3mo agoI always liked this site, but reading this and seeing the anger about expecting the site maintainer to do things for you is repulsive. Frankly, if he wanted to pull his site down with no notice that is perfectly within his right. It was/is his site. He doesn't owe anyone a .tar.gz either. His work.
- BraveOPotato 3mo agoI agree. I've seen it happen before on a project I use. I decided to take a look at the repo for one of the plugins, and I saw a heinous issue that basically was TELLING (not even asking) the maintainer to fix it. Deplorable behavior indeed
- jambalaya8 3mo agoLike dominoes, as soon as it is accepted in a few places, people think it is acceptable to push to 'share'. It's almost terroristic sometimes, the pressure some maintainers are under.
- deleted 3mo ago[deleted]
- nater5000 3mo agoThis is an odd example to present this argument through. I'm not defending AI bots overwhelming websites or hackers motivated by Polymarket, etc., but I don't really believe a 30 year old website with "approximately 160,000 source files serving around 2 million pages" is a good litmus test for the state of the online world. What's worse is the absurdity that this basic site offering niche data would be targeted because Polymarket depends on it for some of their bets, something that most websites don't have to deal with. Frankly, that seems like a much more interesting angle to explore than "this old website that should have been re-written multiple times over the last three decades now doesn't have a choice but to be re-written." Oddly, it seems like the solution, at least in this specific case, is relatively simple (and ironic): pay for an LLM to re-write this site in a modern, scalable, secure way and have that LLM monitor the site to ensure it is behaving correctly. Put this thing in a modern platform behind a proper cache (i.e., throw the whole thing up in Cloudflare) and many of these problems just don't exist anymore. If this site is as basic as it seems, this could be a weekend project. It's certainly an interesting story, and I don't blame the owner of The Numbers for handling his site the way he has, but framing this as "AI is destroying our beloved internet" just seems obtuse, at least through this lens.
- jaredwiener 3mo agoIsn't this the NRA's argument? The only way to stop a bad guy with a gun is a good guy with a gun. Or at least somewhere between that and full on protection racket. LLMs come on the scene, hammer the site until it goes offline or racks up bills that threaten bankruptcy, and the solution then is to pay for an LLM to fix it and monitor it. That's a real nice site ya got there, it would be a shame if massive datacenters going up around the world were to start hammering it from tens of thousands of IP addresses....
- jambalaya8 3mo agoThe actual solution is damn simple and painfully obvious: make things like so-called "prediction markets" illegal. The intelligence community came to the conclusion (after research and experiments) that things like "prediction markets" were a bad idea in 1996. Far worse now than then, with the net and AI of 2026 and the insane number of people now online that want to make an easy buck. Don't get me wrong; I am guessing some of us on here would do well on those places. But they should not exist.
- gajus 3mo agoWhat a throwback. Back in 2015 I have started Applaudience, which at the time was the only provider of real-time cinema ticket sales data. I still remember comparing our numbers against TheNumbers.com as part of calibration. I have since moved on to other businesses, but this remains one of my favorite pieces of technology that I have developed. Would love to bring it back one day.
- datadrivenangel 3mo agoIf the AI companies destroy the open web, eventually they'll need to start curating knowledge sources just like netflix makes movies and amazon has physical stores...
- nradov 3mo agoAlready happening. Frontier LLM vendors have been hiring human domain experts specifically to create their own proprietary training data in targeted verticals.
- hebleb 3mo agoGood read, I was so curious how this happened a few months ago
- tehjoker 3mo agoThe real story here is that prediction markets were banned for a reason and loosening the rules is causing chaos just as was expected. AI plays little role in this story, hacking by humans would also be motivated by financial returns, unless the element is that AI hacking is cheaper and the returns are not so big.
- primitivesuave 3mo agoA couple years ago, I was running a website which allowed the public to view all the US government handouts to small businesses during the COVID-19 pandemic. It also tracked all the fraudulent loans being prosecuted by the DOJ, and allowed anyone to run structured queries over the public dataset. There was a "donate" button which took in ~$2k in donations over the lifetime of the site, and you could download the entire underlying dataset (around 10 GB uncompressed) for free directly on the site. Despite the "download all data" link being prominently placed on the front page, the AI scrapers decided it would be more efficient to download terabytes upon terabytes of raw HTML by paginating through every possible facet on the search endpoint. Even with CloudFront caching results and a fairly efficient backend setup, the monthly bill ended up with around $1k just going toward network ingress/egress, so I shut down the site the following month.
- tailscaler2026 3mo agoBuilding on AWS is a financial time bomb.
- deleted 3mo ago[deleted]
- ninalanyon 3mo agoAre you not able to put limits on how much the site can spend?
- shermantanktop 3mo agoSure, but many a hobbyist has discovered the need for that the hard way.
- mystifyingpoi 3mo agoAs of 2026, still not, and probably never.
- primitivesuave 3mo agoCompletely agree, I've worked at several companies where I saw the AWS bill balloon from five to six figures, usually because of pointless over-provisioning. However, it all became worth it when I saw Jeff Bezos go to space. [1] 1. https://www.youtube.com/watch?v=IOmX793-5t4 https://www.youtube.com/watch?v=IOmX793-5t4
- podgietaru 3mo ago"One Reddit theory even suggested it was a deliberate rug pull designed to cripple the free site to push people towards paid products." I know this isn't really the point of the article, but I've been thinking about this a lot. I wonder if we're going to see more resources go this way. I used to publish little doo-dads as opensource software. Not because it was something that was legitimately ground breaking or anything (it absolutely wasn't) but because I had a problem, and I thought "heh wouldn't it be cool if someone else had a similar problem and could use my resource for it." But now I'm really reluctant to give more stuff to the free web. Because the fact that it gets scraped and added to a pile of training data to later be monetized really rubs me the wrong way. And I can see why maintainers of sites like this, or other free but incredibly useful resources might start to get irate at that.
- CamperBob2 3mo agoThis attitude makes zero sense to me. You benefit from the "training data" just like everyone else does. If you don't, that's a problem with you, not a problem with AI models. AI solves exactly the meta-problem you describe: "I had a problem and needed to write a one-off doo-dad utility program to solve it." Now you can do something with your time besides writing pointless one-off doo-dads. As for monetizing the training data, (a) it cost hundreds of millions of dollars to generate the weights, so why begrudge the companies that made the investment and did the research necessary to make it happen?; and (b) rest assured, whatever your doo-dad does, an open-weight model like GLM 5.2 can generate it for free using your own hardware. So you don't have to pay anyone in that case. Well, except nVidia, I guess. Point granted there.
- paxys 3mo agoI know people have opinions about Cloudflare but why not use it here, at least as a stop gap? Stopping bot traffic is one thing it does very well.
- NetMageSCW 3mo agoCost?
- esseph 3mo agoIt's free for this
- spiderfarmer 3mo agoIt tried it. Hundreds of “genuine” visitors per day on a new website with no search engine presence and no links. That’s a very leaky fence..
- paxys 3mo agoHundreds is better than billions.
- philipkglass 3mo agoI set Cloudflare up a couple of months ago specifically to block bot traffic. It didn't do anything for me. Dumb bots were still hitting every special link on my wiki fast enough that the server was continually swamped running Lua scripts. 65% of the traffic for my English-language site was coming from Vietnam. But I didn't want to block Vietnam altogether, because my hobby site has genuine users from there too. I eventually just shut down my mediawiki instance. I couldn't find a way to keep it online and still run on an affordable VPS.
- NetMageSCW 3mo agoWhat does a cyber attack have to do with AI scraping?
- john_strinlai 3mo agothey both have a risk of harm which the site operator was no longer comfortable with.
- Good4boothee 3mo ago[dead]
- vachina 3mo ago> The world we have built thus far is so incredibly ill-prepared for the power and scale of the AI models we all have access to. Exactly, so use the AI to secure your servers. Ask the AI to audit your site for any security holes. If unable to rewrite, at least harden the existing code. AI’s are really really cheap (and fast) security consultants now.
- advisedwang 3mo agoThis article, and possibly the owners of the site, seem to muddle together several problems: 1. bot traffic causing infrastructure cost 2. scraping circumventing paying for licenses 3. the risk of hacking
- shermantanktop 3mo agoYou missed the speculated motivation: unrestricted prediction markets, which are an open and broad incentive to do whatever actions might provide a slight edge in betting. Webscraping is among the more benign things that gambling-on-anything can drive. And even that has a negative impact, as seen here. I have no idea why those sites are legal.
- aaron695 3mo ago[dead]
- VulgarExigency 3mo agoThey should sue Anthropic for this distillation attack
- ajkjk 3mo agoIt feels like there is fundamentally missing infrastructure here that is needed to make these problems go away. Basically bots need to be (somehow) paying for the traffic they create, or prevented from creating it, or told to go away and then fined if they violate the request. No idea how to do these or even at what level in the stack they should happen, but they need to happen eventually somehow.
- polio 3mo agoEvery website I visit could get a fraction of a cent in my Cloudflare Wallet. A human browsing incurs a few dollars a month. Plus, any website I access in this way decides not to show me ads either and just charges me the cost of serving the data plus a nominal profit margin. Disclosure: I am a shareholder and would love for them to solve the AI bot problem and the ad problem like this.
- wredcoll 3mo agoMaybe we'll all eat our words and cryptocurrencies will actually become useful for something.
- weird-eye-issue 3mo agoThis does not require cryptocurrency
- john_strinlai 3mo agowe've been working on basically the same problem in the email space for a few decades (legit email vs. mass spam). its a very hard (i think impossible) problem.
- cogogo 3mo agoI think I probably use AI like a lot of consumers out there. Search engines have gotten bad and AI really good at answering fairly specific questions. Often pointing at sites like wikipedia. Pretty clear changing my behavior will have zero impact but it is certainly part of the problem. Feels a lot like my CO2 consumption. *Edit - CO2 creation
- spiderfarmer 3mo agoGPTBot crawled millions of pages on my website. I get a handful of visitors from them. Meanwhile Google sends me 10k visitors each day. So GPTBot is now blocked.
- djoldman 3mo agoJust FYI, the bigger companies all allow you to block crawlers via robots.txt: # Block Anthropic (Claude) User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Claude-User Disallow: / # Block OpenAI (ChatGPT) User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / # Block Perplexity User-agent: PerplexityBot Disallow: / # Block Google's AI Training User-agent: Google-Extended Disallow: / User-agent: Google-Extended-Factual Disallow: / # Block Microsoft's Search & AI Crawler User-agent: Bingbot Disallow: /
- frereubu 3mo agoThis is good to know, but a bit of all-or-nothing. It's a shame that, for example, Google doesn't support the crawl-delay field so you can tailor their crawling to your setup: https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec#syntax https://developers.google.com/crawling/docs/robots-txt/robot... I presume it would also cut you off even more from referral traffic.
- FinnKuhn 3mo agoThey don't support it yet. It seems as if Cloudflare is trying to use it's power to force Google to change this though. https://techcrunch.com/2026/07/01/cloudflares-new-policy-pushes-ai-companies-to-pay-for-publishers-content/ https://techcrunch.com/2026/07/01/cloudflares-new-policy-pus...
- spiderfarmer 3mo ago90% of bot traffic on my network of websites is through headless Chrome, via residential bots nowadays. Impossible to block. Not even for Google, as they inflate my Adsense numbers as well.
- 6510 3mo agoI once had the dumb idea to make websites in pdf but the idea is growing on me. Perhaps it should even be in animated gif with <img> <map>'s
- tailrecursion 3mo agoDo the AI labs in the U.S. publish the AWS IP address sets that their crawlers use? Is that not an effective way to block those crawlers anymore?
- theragra 3mo agoThey are often from residential IP.there are even businesses that allow you renting such IPs
- weird-eye-issue 3mo agoNobody serious would ever use an AWS IP lol
- freediddy 3mo agoAll new content will be behind paywalls. This will be the new way the Internet works and I've said this for the last several years. There's no value in writing unique content just to have AI steal it and distribute it without you getting any clicks. The only way it works if it there's a licensing agreement and if they don't want to pay, then who cares, you weren't going to make any money from them anyway.
- npilk 3mo agoWhat if you aren't motivated solely by making money? There will still be new free content created by hobbyists, passionate creators, altruists, etc. (Agree with you more generally.)
- freediddy 3mo agoWhy would you produce content and have no one read it or visits your site, but OpenAI and Anthropic make millions from it? At some point it becomes stupid to just give free money to these companies when they steal literally all your traffic and content. As per the article, Anthropic sends 1 page view for every 38,000 views they get.
- ssl-3 3mo agoVery few people showed up to read stuff on the old web, and yet: It existed.
- freediddy 3mo agoThe "old web" hasn't existed in 30 years now. I was a part of the old web and if people in the early 90s knew that someone was profiting off their work it would have killed it immediately.
- ssl-3 3mo agoI was a part of the old web, too. Everything that I published in this way was placed online with the distinct expectation that everyone (and everything) would do whatever they wanted with it.
- deleted 3mo ago[deleted]
- xeyownt 3mo agoPrediction market as well as anything related to gambling must disappear from earth surface. Full stop.
- theragra 3mo agoI have small website with archive of older radio broadcasts mostly in Russian. Lately, 90% of traffic comes from USA ;) Previously it was like 5%. URL is http://radar.lv http://radar.lv btw Thanks God i can serve up to a terabyte per month of traffic easily, otherwise it would be a catastrophe. I am not against bot scraping, but I worry about stability for meat visitors. So, I think of enabling payments for website visits cloudflare recently developed. I also have the problem of old technology like the mentioned site. While my personal blog uses static generator, archive website uses ancient Drupal version, which has no security patches for many years already.
- ropable 3mo agoThis is another reason we can't have nice things. The level of entitlement required to send an angry message to someone complaining about their free resource being offline is mind-boggling, though.
- cawksuwcka 3mo ago[dead]
- puskavi 3mo agoEh, wouldnt just adding some POW challenge, like anubis solve the scraping by making it hurt crawlers wallets?
- rodarmor 3mo agoThere’s an easy solution: Bruce should place large bets on the prediction markets right before publishing the relevant data. It is legal, ethical, and, for him, risk free.
- Centigonal 3mo agoOur internet ecosystem is becoming more and more hostile to open information commons. I like open information commons -- what do we do about this?
- BrenBarn 3mo agoEvery example of this confirms my view that we should treat the spread of these AI scrapers like we treat the proliferation of drugs. We should be seeking to bust AI rings like we seek to bust drug cartels.
- Drdiamond 3mo agoThis is such a sad worldview. Life without any challenges and solved using AI. it will vastly decrease our learning
- breve 3mo agoAI companies really do socialize the costs and privatize the profits. Sites like The Numbers have to take on the cost of surviving the AI onslaught and the AI companies return nothing back to them.
- jambalaya8 3mo agoworse; it isn't just no return, it is also cost.
- daniela-scott 3mo ago[flagged]
- Cthulhu_ 3mo agoTo a point; as the post mentions, they added instructions for LLMs to their website and increased their licensing inquiries tenfold (the article didn't state anything about actual licensing payments being made though). I don't think there would be any issues per se if the scrapers just paid licensing fees to get the good / complete data. But the issue was that they started to try and find exploits to get to data earlier.
- baud9600 3mo agoOne thing I’m wondering is whether we need an open source set of technical patterns and libraries for dealing with this changing traffic mix. Millions of small sites and creators don’t have the ability to design their own protections against large scale automated access. If useful content now attracts aggressive crawler traffic, many sites will be too expensive or unreliable to run. Is part of the answer a community response? Perhaps a community-maintained toolkit, based on traffic data, that host sites can apply? It could include standard agent identification, rate limiting, traffic classification, access policies, caching, challenge mechanisms, logging, attribution and usage control etc etc. In effect, we need much stronger road rules for today’s automated traffic, available as open technical patterns and libraries rather than every site owner having to invent this alone (they won’t).
- Cthulhu_ 3mo agoThe community response was respecting robots.txt, but since that was more of a gentleman's agreement than a legal requirement, hungry AI scrapers disregarded it. Which brings us to the old fashioned flood control mechanisms. That is, the toolkit you propose already exists and has been used for decades in various iterations to protect against various forms of attack (slashdotting, DDOS attacks, overzealous search engines, and now AI scrapers). Have you looked into those before? Companies like Cloudflare have been at the forefront of this field for a long time now.
- m-i-l 3mo agoYes, a community effort against the botnets would be great. The "good bots" which identify themselves and follow the rules are easy enough to block, so not a problem. It is the "bad bots" which pretend to be real users and hide behind residential proxies, and so are almost impossible to block at the moment, which are the problem. Given that there are big companies openly (i.e. on the clearweb, not even darkweb) selling access to these residential proxy botnets of compromised smart TVs[0] and mobile phones and other devices, can we not simply get a database of the these IPs and block access from them? I would venture that almost every single one of those residential users are unaware that they have devices in their homes which have been compromised and are being abused in this way, so if they were to start seeing messages from more and more sites along the lines of "Access to this site has been blocked because unusual traffic has been detected from your computer network. Please check all devices on your network and remove any malware which may be routing this traffic." then maybe we could start addressing the problem at the source. [0] https://news.ycombinator.com/item?id=49000864 https://news.ycombinator.com/item?id=49000864
- jagermo 3mo agoprediction markets were a huge mistake.
- PowerElectronix 3mo agoOnly solution I see is to ban user that abuse the system. Too much requests, too much bandwidth and you get into the blacklist and are cutoff from the human side of the internet. With the amount of money AI labs are burning, someone should just set up some infra to host the data they want and charge for access to it, instead of abusing the goodwill of legitimate pages that offer it for free.
- drumhead 3mo agoThe Numbers is such a brilliant site, this explains why it came back in such a stripped down form. AI and prediction markets giving me more reason to hate them.
- MikeTheGreat 3mo agoGenuine question: do prediction markets open up a new revenue stream for TheNumbers.com? Specifically: they suspect that the motivation for trying to hack their site (for at least some people) is that they wanted early access to numbers that folks were betting on in prediction markets. Since they've got those numbers they can just bet on them, then benefit from their perfect knowledge. On the one hand this does seem incredibly unethical (it's clearly insider trading). On the other hand the CEO of PolyMarket has said that insider trading is part of the point of PolyMarket: https://youtu.be/ZN4njIQcSR4?si=ztyTtgjeHSJbNjSZ&t=1566 https://youtu.be/ZN4njIQcSR4?si=ztyTtgjeHSJbNjSZ&t=1566