5 ms·

This was an interesting read for a database novice. It seems like a lot of the quoted stats are about in memory datasets - is that realistic?

dap 6y ago
It depends a lot on what you’re doing. In my last role, keeping the Postgres database in memory was absolutely not an option, and we ran into major issues related to physical I/O.
bmcahren 6y ago
Yes. Memory can reach 768GB on a single instance today and I imagine that to expand. From there you can scale by sharding. In memory provides real-time transactions you can't guarantee when using disk-based storage.
- ahachete 6y ago
  EC2's u-24tb1.metal is 224/448 cores/hyperthreads and 24TB RAM ;) https://aws.amazon.com/sap/instance-types/ https://aws.amazon.com/sap/instance-types/
  - wiredfool 6y ago
    And nearly 2 million for a 3 year reserved instance.
    - atonse 6y ago
      This reminded me of a story from 15 years ago. I once worked for a company that was writing a proposal for a US Homeland Security IT system. This was 2006. I wasn't involved in it but my office-mate was. He randomly turned his chair around and said "hey, can you go on Dell.com and see if you can build the most expensive server imaginable" – so I did and I ended up at around $350k. I don't remember what it was, but at the time the stats just felt absolutely obscene. I do remember that, at that scale we were buying a closet, not just one box. And I told him the price, and he said, "They have set aside $20 million for server hardware, and what you configured is way way overkill for what they think they need" – so we were both so damn perplexed because, at least from what he told me, this didn't need that much. And it wasn't one of those classified projects where they couldn't tell you what it was for. It was a pretty boring line of business type thing (like HR or building access control or something). Maybe that's probably more a story of just how much cash was being poured into that agency during the years after 9/11.
      - jarym 6y ago
        They got to the end of their financial year and had $$$ left to spend... happens all the time.
        atonse 6y ago
        But this was for an RFP. they would’ve had to continue to obligate that money likely for years to come.
- fabian2k 6y ago
  You can get more than that on common servers, see e.g. the recent post by Let's Encrypt on their new database server: https://letsencrypt.org/2021/01/21/next-gen-database-servers.html https://letsencrypt.org/2021/01/21/next-gen-database-servers... They have 2TB RAM in there, and I suspect that is not the largest possible amount if you're willing to spend more money (though probably the largest possible amount for that particular server). I played around a bit on the Dell website, and the largest server I could find supported up to 6TB RAM, with a price at ~300k EUR (I assume nobody pays list price for these).
  - tyingq 6y ago
    4TB seems to be the limit for normal-ish servers. 32 slots filled with 128GB DIMMS. You can find servers with more slots (48 is common, 6TB), but you're going past most people's definition of "commodity".
- deleted 6y ago
  [deleted]
- dekimir 6y ago
  > In memory provides real-time transactions you can't guarantee when using disk-based storage. This is changing as we discover better ways of coding IO operations. https://www.globenewswire.com/news-release/2019/11/05/1941144/0/en/ScyllaDB-Smashes-Performance-Record-Hits-1-000-000-000-RPS.html https://www.globenewswire.com/news-release/2019/11/05/194114... https://itnext.io/modern-storage-is-plenty-fast-it-is-the-apis-that-are-bad-6a68319fbc1a https://itnext.io/modern-storage-is-plenty-fast-it-is-the-ap...
- bcrosby95 6y ago
  How long does it take to get the data off disk and into memory after coming online? A decade ago filling just 64GB of memory with our hot dataset was a painful process. I can't imagine it's any nicer with 768GB.
  - 1996 6y ago
    On a modern raid of NVME, you can do several gigs per second - with 20 PCIe Gen3 lanes on a 3400G about 10 GB/s is reasonable, for a theoretical max of 20GB/s. Assuming your system is very badly configured and you can only do 3 gigs per second, about 400 s, so about 5 minutes, which is not very painful
0x10c0fe11ce 6y ago
Yes. That’s why the OP talked about separating hot/cold data and scale up instead of scale out.

dap 6y ago

It depends a lot on what you’re doing. In my last role, keeping the Postgres database in memory was absolutely not an option, and we ran into major issues related to physical I/O.

bmcahren 6y ago

Yes. Memory can reach 768GB on a single instance today and I imagine that to expand. From there you can scale by sharding. In memory provides real-time transactions you can't guarantee when using disk-based storage.

ahachete 6y ago
EC2's u-24tb1.metal is 224/448 cores/hyperthreads and 24TB RAM ;) https://aws.amazon.com/sap/instance-types/ https://aws.amazon.com/sap/instance-types/
- wiredfool 6y ago
  And nearly 2 million for a 3 year reserved instance.
  - atonse 6y ago
    This reminded me of a story from 15 years ago. I once worked for a company that was writing a proposal for a US Homeland Security IT system. This was 2006. I wasn't involved in it but my office-mate was. He randomly turned his chair around and said "hey, can you go on Dell.com and see if you can build the most expensive server imaginable" – so I did and I ended up at around $350k. I don't remember what it was, but at the time the stats just felt absolutely obscene. I do remember that, at that scale we were buying a closet, not just one box. And I told him the price, and he said, "They have set aside $20 million for server hardware, and what you configured is way way overkill for what they think they need" – so we were both so damn perplexed because, at least from what he told me, this didn't need that much. And it wasn't one of those classified projects where they couldn't tell you what it was for. It was a pretty boring line of business type thing (like HR or building access control or something). Maybe that's probably more a story of just how much cash was being poured into that agency during the years after 9/11.
    - jarym 6y ago
      They got to the end of their financial year and had $$$ left to spend... happens all the time.
      - atonse 6y ago
        But this was for an RFP. they would’ve had to continue to obligate that money likely for years to come.
fabian2k 6y ago
You can get more than that on common servers, see e.g. the recent post by Let's Encrypt on their new database server: https://letsencrypt.org/2021/01/21/next-gen-database-servers.html https://letsencrypt.org/2021/01/21/next-gen-database-servers... They have 2TB RAM in there, and I suspect that is not the largest possible amount if you're willing to spend more money (though probably the largest possible amount for that particular server). I played around a bit on the Dell website, and the largest server I could find supported up to 6TB RAM, with a price at ~300k EUR (I assume nobody pays list price for these).
- tyingq 6y ago
  4TB seems to be the limit for normal-ish servers. 32 slots filled with 128GB DIMMS. You can find servers with more slots (48 is common, 6TB), but you're going past most people's definition of "commodity".
deleted 6y ago
[deleted]
dekimir 6y ago
> In memory provides real-time transactions you can't guarantee when using disk-based storage. This is changing as we discover better ways of coding IO operations. https://www.globenewswire.com/news-release/2019/11/05/1941144/0/en/ScyllaDB-Smashes-Performance-Record-Hits-1-000-000-000-RPS.html https://www.globenewswire.com/news-release/2019/11/05/194114... https://itnext.io/modern-storage-is-plenty-fast-it-is-the-apis-that-are-bad-6a68319fbc1a https://itnext.io/modern-storage-is-plenty-fast-it-is-the-ap...
bcrosby95 6y ago
How long does it take to get the data off disk and into memory after coming online? A decade ago filling just 64GB of memory with our hot dataset was a painful process. I can't imagine it's any nicer with 768GB.
- 1996 6y ago
  On a modern raid of NVME, you can do several gigs per second - with 20 PCIe Gen3 lanes on a 3400G about 10 GB/s is reasonable, for a theoretical max of 20GB/s. Assuming your system is very badly configured and you can only do 3 gigs per second, about 400 s, so about 5 minutes, which is not very painful

0x10c0fe11ce 6y ago

Yes. That’s why the OP talked about separating hot/cold data and scale up instead of scale out.