8 ms·
At least in database kernels, we noticeably reached this threshold around five years ago with typical server hardware. This is an interesting computer science p
by jandrewrogers 8y ago
At least in database kernels, we noticeably reached this threshold around five years ago with typical server hardware. This is an interesting computer science problem in that virtually all of our database literature is based on the presumption that I/O is much slower than CPU.
If you cleanroom a database kernel design based on the assumption that I/O performance is not the bottleneck, you end up with an architecture that looks very different than the classic model you learn at university. It is always a tradeoff of burning a resource you have in abundance to optimize utilization of a resource that is scarce, and older database architectures are quite wasteful of resources that have become relatively scarce on newer hardware.
- aloer 8y agoWhat was the situation like before SSDs? Were the fastest hard drives at the time (15k rpm?) faster than the fastest CPUs?
- insulanus 8y agoLet's take a look at the throughput of a server hard disk vs. a typical server CPU: https://serverfault.com/questions/190451/what-is-the-throughput-of-15k-rpm-sas-drive https://serverfault.com/questions/190451/what-is-the-through... https://www.anandtech.com/show/9185/intel-xeon-d-review-performance-per-watt-server-soc-champion/6 https://www.anandtech.com/show/9185/intel-xeon-d-review-perf... Now, that 45.8 GB/sec can't usually be fully utilized by the application, but it's a lot higher than the 200MB/sec of the server hard drive. There's also the complication of RAID, max memory fetch rate of a single core, etc. And the fact that the database has to do some processing besides moving data around. But it seems to me that server CPU bandwidth of the last generation is significantly higher than spinning disks.
- blihp 8y agoIndividually no, collectively (i.e. storage arrays) yes.
- loeg 8y agoSort of. Intel's server parts top out at on the order of 48 PCIe 3 lanes (Skylake-SP). Those parts have 6-channel DDR4 memory (rated to DDR4-2666), giving a theoretical DRAM throughput limit of about 128 GB/s. Meanwhile, a PCIe 3.0 lane has a theoretical throughput limit of about 985MB/s; 48 lanes makes 47 GB/s. Both are theoretical numbers and in practice will be lower, and you're right that it doesn't give you a lot of free CPU to do any kind of processing on the data, but it seems like the CPU can keep up (as long as Intel continues to have an anemic number of PCIe lanes and until PCIe 4 comes along). (FWIW, on a dual-socket Skylake-SP, Anandtech only achieved 122 GB/s (of theoretical 256 GB/s DRAM maximum). But Intel claims the dual-socket can reach 199 GB/s with an AVX-512 workload.[1] Scaling both of these numbers down by a factor of two gives a very crude estimate for single-socket DRAM bandwidth on a conventional workload (61 GB/s) and AVX-512 (99 GB/s).) Meanwhile, AMD's Zen server parts have more than double the PCIe lanes but only a few more memory channels, so I expect you're correct there — huge/fast PCIe arrays could be bottlenecked on DRAM on that platform. [1]: https://www.anandtech.com/show/11544/intel-skylake-ep-vs-amd-epyc-7000-cpu-battle-of-the-decade/12 https://www.anandtech.com/show/11544/intel-skylake-ep-vs-amd...
- blihp 8y agoThe question was about spinning rust drives ('before SSDs') and the corresponding CPUs that were available at the same time.
- owaislone 8y agoAre you aware of any efforts by existing databaes or entirely new databases built with this realization in mind?
- jchrisa 8y agoNoSQL is based on this realization. Rather than optimize for IO with normalization, denormalize and pipe data more or less directly from storage to clients, simplifying the compute and consistency model to make distributed data easier.
- mike_ivanov 8y agoThe point of normalization is logical consistency, not IO optimization.
- jchrisa 8y agoSure. But the point of (early) NoSQL was IO optimization, not logical consistency.
- bogomipz 8y agoI was curious about the parenthetical "early" in your comment. Is this optimization no longer the case then? Could you elaborate on recent developments regarding this? I've been out of the NoSQL loop for some time and am genuinely curious.
- tracker1 8y agoThe problem is, it varies... you have what are effectively key/value stores to document databases, column stores and everything in between. You have systems built on other systems. RethinkDB and Cockroach have different approaches than Redis, Mongo, Cassandra or others. CockroachDB gives an SQL interface over the top of a distributed data store with better consistency and relations. Cassandra has no real relations over a BigTable/Column-Store solution. It really just depends. I think with what's coming out of SSD/NVMe and even Optain DIMMS, that there will be databases directly tuned to control/set their own data storage in these environments.
- shereadsthenews 8y agoI don't know ... we used to interleave our hard disk formats because a hard disk could stream data faster than an i386 CPU could ingest it, and there was plenty of database research done prior to 1990.
- masklinn 8y agoI assume the IO you're talking about is (/was) sequential. And DBs are specifically engineered to sequentialise their IO (e.g. clustered indexes). But even if SSDs do blow HDDs out the water on sequential IO, it's on random IO that the difference is most stark, the cost difference between sequential and random is much lower on SSD than on HDDs, and both random IO throughput and random IOPS shoot through the roof relative to spinning rust. An SSD might be an order of magnitude better than an HDD on sequential IO, but will often be 2 or 3 on random IO. That means the loss of throughput from sequential IO to random IO (while still there due to commands overhead and the like) is much, much, much smaller on an SSD than on an HDD. And the story is similar on the latency front, an HDD might have a command latency in the low tens milliseconds, an SSD in the low-mid tens microseconds.
- briffle 8y agoPostgresql allows you to tune the 'cost' of sequential and random IO, to help the planner figure out the best way to get the data. https://www.postgresql.org/docs/11/runtime-config-query.html#RUNTIME-CONFIG-QUERY-CONSTANTS https://www.postgresql.org/docs/11/runtime-config-query.html...
- dorlaor 8y agoWith LSM approach (log structured merge tree) we have at Scylla, we make all the writes batched/sequential and receive an awesome boost. Some reads can be sequential too if you scan through the clustering keys.
- temac 8y ago> low-mid tens microseconds From a modern CPU point of view, that's an eternity.
- sytelus 8y agoThere are plenty of in-memory database architectures that takes full advantage of fast random access offered by DRAMs. I think paper focuses on OS design which is still tied with assumption of slow disks.
- jandrewrogers 8y agoIn-memory databases are not economically scalable. To a first approximation, database volume is the integral of database throughput, so databases that require extreme I/O throughput also tend to require extremely high volumes of online data. Many typical sensor data models, which drive much of the need for high-throughput I/O, grow by petabytes per day so the multiple orders of magnitude difference in media cost make DRAM storage infeasible. If the bottleneck is not storage bandwidth, and it isn't for many recent systems and modern software architectures, then there is no practical performance advantage to keeping your data purely in-memory. This has been demonstrable ever since PCIe HBAs with arrays of cheap SSDs became practical.
- gambler 8y ago>In-memory databases are not economically scalable. SAP HANA is widely used for very large data sets.
- jandrewrogers 8y agoHow are you defining "very large"? I've used HANA and even SAP doesn't make claim of real scalability -- the practical limitations are in their own documentation. And in real scalability testing, it struggles long before you reach those limits. For sensor data, I wouldn't use it for more than a few terabytes, and even then it has a few sharp edges when it comes to performance if you are not careful. No one is going to be putting petabytes into it, which isn't that much operational data these days. And to my point you quoted, SAP HANA is extremely expensive to operate compared to alternatives. The licensing costs alone will kill you, never mind the hardware requirements.
- 8y ago
- riazrizvi 8y agoI can see how the memory hierarchy will have layers removed, but it still exists because registers are where operations on data predominantly take place. So you still need to identify small working sets of variables that you extract return values from. I can see how a lot of caching logic will be obviated but not the principle of memory hierarchy, which says ‘load small batches of problem set, solve, store results and repeat’.
- magicalhippo 8y agoJust skimmed the article, but it reminded me very much of how graphics APIs have evolved, due to similar motivation. Just compare what it takes to draw a triangle in OpenGL 1.x[1] vs Vulkan[2]. [1]: http://nehe.gamedev.net/tutorial/your_first_polygon/13002/ http://nehe.gamedev.net/tutorial/your_first_polygon/13002/ [2]: https://vulkan-tutorial.com/Overview https://vulkan-tutorial.com/Overview
- stallmanifold 8y agoIn what ways do the database architecture change when one discards the assumption that I/O is the bottleneck? Does it obviate the relational model? I always found the relational model to fit working with data sets very well since it's nice and declarative. Does that turn out to be a bottlneck assuming fast I/O?
- garmaine 8y agoWhy would the model be affected?