5 ms·
Saying that something has "Zero latency" because it's "on disk" is off-putting from a technical honesty perspective.
by xnorswap 3y ago
Saying that something has "Zero latency" because it's "on disk" is off-putting from a technical honesty perspective.
- JohnBooty 3y agoIt's frustratingly close to correct. Since it's on local disk, it winds up being cached in in-process RAM, which is..... still not zero latency... but has so much less latency compared to talking the to database over a network stack... that I'm alllllllllmost sorta kinda okay with it. Almost. lol
- deleted 3y ago[deleted]
- rmbyrro 3y agoHe's comparing to a networked database. In that context, a local disk has virtually zero latency for most use cases. Also, SQLite can run in-memory. If your process is long running and keep the DB in RAM, the latency is even closer to zero.
- thaumasiotes 3y agoWait until you see my all-storage-in-the-registers database. ;D
- isoprophlex 3y agoEspecially if you tout it as the solution for the n+1 problem...
- Ensorceled 3y agoWhen really what you are really doing is putting off your n+1 problem until 6 months from now when you have actual active users with real data ...
- fodkodrasz 3y agoIsn't that a rational business decision?
- Ensorceled 3y agoFinding out your entire app can't scale with real data is not really something you want to find out in production. I'm not saying "don't use SQLite", I'm saying "reduced n+1" problems ISN'T a good reason.
- fodkodrasz 3y agoAFAIK SQLite can scale quite a bit with WAL2, and then probably it is easy to upgrade it to Postgres. If your data model was bad from the start up, it doesn't matter if you used Postgres initially. If you find out your data model was bad, and is a constraint to scaling, you can find it out without spending a lot on insanely scaled RDB instances. I'm for the build one to throw away approach, and SQLite is often a good tool in the toolbelt for the initial attempt of this.
- Ensorceled 3y agoNow we're getting silly. If you have n+1 issues you already have a bad data model which is why what you are really saying is "SQLite lets you have a bad data model which gives you velocity!!!!!" which is obviously (I hope it's obvious) not a good thing.
- yawaramin 3y agoBut the thing is that in SQLite you effectively don't have N+1 issues.
- fodkodrasz 3y agoFrom a business perspective: You might not have to solve that problem at all in 6 months, because you might not get so many users for the product, that scalability is a problem. You often need velocity more than clean code, because you are not sure if the idea will get traction at all. In that case, from a business perspective, to be able to experiment you need velocity more than a good data model. I did this kind of exploratory development both with SQLite and Postgres, both had different strengths and weaknesses. Had the purist clean data model with Postgres cost more than it should have when the product had to pivot the product. Also have seen the scalability limits of SQLite. Just shutting down a POC project backed by SQLite, which failed to get traction mostly because of business execution problems, and SQLite saved a lot of effort for the implementation, and using a "cleaner" and "more scalable" solution would have only reduced velocity, but would not have got us more users, the system worked fine this way (even has N+1 queries in some places). Should we got to the point that SQLite starts to become a bottleneck we could have afforded to clean the data layer (we had to rework countless times already as requirements were in a flux) up a bit and move to more scalable solution. Others might stick Firebase or MongoDB there, and it also does work for a while, and that is also a valid decision IMHO from a business perspective. I think this is a kind of topic where the premature optimization is the root of all evil meme can apply, depending on the product you are making and the available resource pool. Not for the hello world examples of course, and not for the well defined well funded project, but there are lot of exploratory attempts out of the bigco/vc funded unicorn world.
- refset 3y agoExactly, the n+1 problem is really about query optimization, latency just makes it more pronounced.
- acomjean 3y agoI see the idea of sql lite, but a mysql / Postgres store a lot in memory so no disk access required (if not the data the indexes stay memory resident) We have an old web based tool that uses Java and luncene indexed file system based database. It’s a pain to work with frankly. https://lucene.apache.org/ https://lucene.apache.org/
- dham 3y agoTrue but the new hotness is edge and your database can be really far from where you actually deploy. This is what this article is trying to push back against. Mainly because Kent is all about Remix and Remix is more about just deploying to traditional servers.
- willsmith72 3y agoremix is actually great for the edge. Lots of people run it on cloudflare workers in particular https://remix.run/docs/en/main/guides/performance#this-website--flyio https://remix.run/docs/en/main/guides/performance#this-websi...
- _joel 3y ago> We have an old web based tool that uses Java and luncene indexed file system based database. It’s a pain to work with frankly. Elasticsearch waves
- RedShift1 3y agoNote that your operating system caches file system reads so SQLite doesn't need to put anything extra in RAM. Most reads will be from RAM, so no disk access required.
- hahn-kev 3y agoCompared to something over lan or internet, it is an order of magnitude less latency, so from an internet perspective it is 0
- j-pb 3y agoRam on a different machine via ethernet has lower latency than spinning disks in the same machine.
- Cthulhu_ 3y agoYou're reaching. Modern filesystems cache disk access in memory, and modern hardware doesn't use spinny disks for applications.
- yawaramin 3y agoAlso, SQLite has a configurable local page cache which it maintains directly in your application's process. There's a very likelihood that you're just hitting that in-memory cache directly, let me emphasize, in your own process. In-process memory cache hit is faster than separate-process memory cache hit over ethernet.
- bravetraveler 3y agoWe're actually to a point where storage bandwidth can exceed memory! NVMe arrays are weird, can be better to just rip it from the drives if you need bandwidth over latency Even then, with how easy the memory cache is to invalidate... I'm not sure even that's worthwhile. The caching isn't some Paragon of performance - there's only so much it can do, when looking at it funny invalidates it.
- merpnderp 3y agoWe've spent ridiculous amounts of money on faster disk and network and we're still at around 30ms for our average server ping to the DB.
- 3y ago