This is one of those obvious things that people have a blockheaded aversion towards. In every job I’ve consulted on, there has been a component where pagination

This is one of those obvious things that people have a blockheaded aversion towards. In every job I’ve consulted on, there has been a component where pagination is the bottleneck. And it’s rarely where pagination could be useful (like a static list of products). I typically see it when someone builds an aggregating (and aggravating) microservice that pulls all of the data from another service; that blocks the entire database. Perhaps it’s the world of ‘Django developers’ I find myself in. But the fact this simple design pattern reaches the front page of Hackernews worries me. I’ve personally had backend developers give confused looks and diatribes about simplicity when I’ve suggested this approach. Frontend developers frequently give the greatest resistance, which I think is because they most often act in concert with product managers who are welded to certain UX idioms that they chose without fully imagining the engineering consequences of. One approach which I try to push people towards is fetching a big list (N≈5000) and paginating it however the UX designer wants. The response of the big list should be sparse, that is without including additional fields that aren’t displayed by the front end (and fetching more data when the user navigates). You’ll frequently get puzzled looks suggesting this, but benchmarks will usually show that fetching 5000 rows and a small number of columns takes a few milliseconds, and can be serialised into 150kb. On balance it ends up being faster: as you’ll get fewer network requests; fewer round trips to the database; fewer fetches; and less time spend serialising (which is a major bottleneck itself that Python developers ignore).