I run a web to text usenet gateway and it has been an interesting challenge to scale it to deal with this. Article fetch and retrieval is extremely efficient b

I run a web to text usenet gateway and it has been an interesting challenge to scale it to deal with this. Article fetch and retrieval is extremely efficient but I apply JWZ threading and that can cause a single article to make a number of overview requests which are more expensive depending on the depth. I solve it currently with caching but will eventually implement a thread backend on the NNTP side to keep thread roots updated at insertion time and it will be a very cheap read request. One persistent thought is what are people doing with this data? I get that people want to train models, but I also have a hard time believing there are more than a hundred companies with the resources to spider the web like this and actually do anything meaningful with all that data. Academics, researchers, and people working on lower level innovation are probably well off with CommonCrawl.. only people trying to make frontier models really need fresh and endless data right?