4 ms·
Speedy transactions in multicore in-memory databases
- gigatexal 4y agoI would love to see more of these kinds of discussions on HN. Where do folks go to be on the forefront data world (databases,algorithms, distributed systems, etc)?
- alfiedotwtf 4y agoI'm currently going through one of the papers on https://www.nowpublishers.com/DBS https://www.nowpublishers.com/DBS and finding it good.
- deleted 4y ago[deleted]
- gigatexal 4y agoSome of these journals are behind a paywall. :( If they're from publicly funded universities I'd wish the scholarly work would be free. I understand things coming out of Microsoft Research, for example, being pay-walled.
- anonymousDan 4y agoIf you're interested in academic papers, look at the top 2-3 conferences in each field (e.g. VLDB/SIGMOD for databases, SOSP/OSDI/USENIX ATC/EuroSys/NSDI/PODC(more theoretical) for systems/distributed systems. Some of the above might require an ACM digital library subscription to see the papers (which is really good value in my opinion), but others are completely free (i.e. the USENIX conferences). In many/most cases you can often find a copy of the paper online on the author's website.
- lvogel 4y agoall VLDB papers are also completely open access. You can find e.g. last year's papers here: http://vldb.org/pvldb/volumes/15 http://vldb.org/pvldb/volumes/15.
- _448 4y ago> Where do folks go to be on the forefront data world (databases,algorithms, distributed systems, etc)? Search for IRC channels on the topics of interest.
- amelius 4y agoIt supports multicore but it doesn't seem to support true distributed operation, which is what most applications require nowadays.
- gbin 4y agoInteresting comment. Scaling "out" is costly in complexity and only needed... well... If it is needed! 700k transactions per second on a single machine from 2013 is probably millions per second on a modern machine. We need to be careful and always ground the scale to reality, for example: the number of flight reservations per year worldwide is ~3B. Let's say you have 10 attempts / search per résa: 30B, then 3 transaction per, 90B. This is still less than 300 transactions per second in average! I believe the lost art of squeezing out the performance of one machine is very valuable in today's world because a single machine can definitely achieve "most applications" at the human scale with no atomic clock to sync db transactions across the planet with thousands of nodes.
- amelius 4y agoYeah, but flights are relatively expensive and aren't scheduled as often as e.g. social media messages are sent. Also, you always have to be prepared for bursts of requests. And of course some new requirements can always be thrown in by management which may invalidate any assumptions you made about speed.
- jandrewrogers 4y agoAny data source where events are created by intentional human action, including global social media platforms, have relatively trivial event rates even during bursts. It isn't the event rate that gets you, it is the rest of the workload processing and serving content. Just landing and storing the events can be easily handled by a single modern server with capacity to spare. Machine sourced data, by comparison, has event rates that are orders of magnitude higher.
- threeseed 4y agoScaling out isn't just about performance. It's also about high availability. Which is something everyone needs to consider when using cloud infrastructure. AWS provides 99.5% EC2 uptime guarentees which is ~2 days a year of outages. That is not simply acceptable for most use cases and is why a single server just won't cut it.
- WatchDog 4y agoSo this design uses an epoch of 40 ms, does that mean any write operation will take on average 20ms? Quite a lot of latency compared to regular databases.
- singron 4y agoI think so. Figure 7 is the only figure in the paper that shows latency, and it appears the epoch defines a latency floor. The system is aggressively optimized for throughput at the expense of commit latency, which is common for anything with group commit.