4 ms·
When someone says, hey forget postgresql and just use SQLite, and someone else says good luck if you hit n rows, and someone says it can easily handle n+m rows
by onion2k 4y ago
When someone says, hey forget postgresql and just use SQLite, and someone else says good luck if you hit n rows, and someone says it can easily handle n+m rows with x memory …
This is a good example of how devs get things wrong. The argument between which database to use because you might hit the row limit has zero benefit to the user. It's devs wanting to avoid future work to migrate from one database to another; a problem most apps will never actually see.
There's even an aphorism about this exact thing in dev: YAGNI https://en.m.wikipedia.org/wiki/You_aren%27t_gonna_need_it https://en.m.wikipedia.org/wiki/You_aren%27t_gonna_need_it
Solve the problem you have, not thr problem you want.
- adrianN 4y agoMaybe most apps never see the problem of hitting the row limit because the devs spend time beforehand thinking about which limitations they're likely to hit.
- onion2k 4y agoIt's definitely possible, but the limit is 18,446,744,073,709,551,616 rows, or 281 terabytes, whichever the database hits first, so you probably could put the discussion off until year 2 or 3 if you really had to. Kind of like maybe the 'Send your friends money, divide the restaurant bill easily!' app founders first think will change the world could get really popular if it gets on the frontpage of HN..
- christkv 4y agoI have a feeling you will long before experience other problems that makes likely you have to migrate to a different storage system before you hit that limit.
- noir_lord 4y agoGiven it's ~2 billion rows for every person the planet I reckon so. Other than Facebook who has 2 billion rows for every person the planet (tongue in cheek).
- burntoutfire 4y agoTelemetrics and sensors data could (at least theoretically) produce billions of row per hour per single sensor.
- bluGill 4y agoI work on such a system and one of the things we do is figure out which of that sensor data to throw away. In most cases the change between measures is too small to be worth saving, and because we have less data saved we can analyse it faster and thus do something useful with it. The purposes of all that data isn't to fill storage, it is to allow making useful decisions based on it. If you can't make a better decision based on sensor input then don't save it.
- bluGill 4y agoLong before you hit any row limit you will have performance problems because you have far too many rows to process in a reasonable amount of time. If you think about this problem and ensure you don't have too much data to process the row limit is so absurdly high you will never hit it.
- zhte415 4y agoChoosing to do nothing is still choosing to do something. In choosing to do something, one makes an informed choice. How informed that is is up to you. Row limit may be one, or performance or feature set may be another - say, choosing Postgres for some data types and easier indexing. Making informed choices are of benefit to the user. How much benefit depends on one's opportunity cost doing X instead of Y. Choosing not to do X is still a choice, an informed one or an uninformed one. Or to step away from choice of database, take privacy. I see an immeasurable number of things on Show HN that take user data but don't have a privacy policy, if they have one it's often copied and pasted for somewhere to placate Mailchimp or another 3rd party service, and very rarely is it actionable. Not launching a product is of course problematic for a user. Launching one that doesn't confirm to norms and rights a user expects will also be problematic. Should we not launch? How informed should your choice be? California State law, GDPR? How about Malaysia, or South Korea? There are rabbit holes one can dive in to. There are reasonable lengths to dive, of which zero should not be a default case. At least have a look inside. Solving a problem isn't rushing something out full of uninformed choices.
- onion2k 4y agoThere are reasonable lengths to dive, of which zero should not be a default case. The fact you're even aware that there's an alternative to choose means you're already slightly informed[1]. Being a bit more informed so that you know a solution will actually work is fine. I am not suggesting you shouldn't look at alternatives. I'm saying you should look at alternatives from the perspective of the solving the problem you have rather than arguing about pointless trivia that won't ever impact you or the app you're building (like row limits). [1] Every web dev who used nothing but MySQL for 15 years should be crying now.
- zhte415 4y ago> from the perspective of the solving the problem you have Completely. A small planning goes a long way, and helps define what problem is being solved. This is a particular peeve I have with lauding of pivoting. The plan didn't work out and you're now doing something else? That's great. At least something was gained and you're looking forward. But don't design for it. In other industries this goes by the perhaps less jargony terms salvage or wasteful production. But putting salvageable and waste in the description of a product solution doesn't sound as nice, and nor should it.
- adamauckland 4y agoLong time ago we ran several large insurance websites, fairly standard SQL Server backend. Once day I got into work and a slightly panicky dev said "We need to change the primary key on all the tables" It was using an int and at the current rate of data entry, was going to blow the maximum record number in a couple of days.
- hungryforcodes 4y agoMust have been in the Jerassic period of MS SQL Server primary keys...
- foobarbaz33 4y agoIf you choose a 32-bit integer for your PK it can only address about ~4 billion records. That was true 1000 years ago, and will be true forever until a black hole swallows everything and changes the laws of reality.
- bluGill 4y agoForget how many angels can dance on the head of a pin, the real question is can God change the value of pi, or can God make 32 bit integers larger than our current max? In both cases what it means to math is more Interesting than any yes/no.
- jerrre 4y agoyou could take this anecdote as a proof for how rarely these things happen
- adamauckland 4y agoYeah, that's true, the one time in 25 years.