4 ms·

You have 6 TiB of ram?

by boppo1 2y ago

ninkendo 2y ago
You don’t need that much ram to use mmap(2)
- marginalia_nu 2y ago
  To be fair, mmap doesn't put your data in RAM, it presents it as though it was in RAM and has the OS deal with whether or not it actually is.
  - ninkendo 2y ago
    Right, which is why you can mmap way more data than you have ram, and treat it as though you do have that much ram. It’ll be slower, perhaps by a lot, but most “big data” stuff is already so god damned slow that mmap probably still beats it, while being immeasurably simpler and cheaper.
    - marginalia_nu 2y ago
      Really depends on the shape of the data. mmap can be suboptimal in many cases. For CSV it flat out doesn't matter what you do since the format is so inefficient and needs to be read start to finish, but something like parquet probably benefits from explicit read syscalls, since it's block based and highly structured, where you can predict the read patterns much better than the kernel can.
cess11 2y ago
The "(multiple times)" part probably means batching or streaming. But yeah, they might have that much RAM. At a rather small company I was at we had a third of it in the virtualisation cluster. I routinely put customer databases in the hundreds of gigabytes into RAM to do bug triage and fixing.
- kmarc 2y ago
  Indeed, what I meant to say is that you can load it in multiple batches. However, now thinking, I did play around with servers of TiBs of memory :-)
vitus 2y ago
If you're one of the public clouds targeting SAP use cases, you probably have some machines with 12TB [0, 1, 2]. [0] https://aws.amazon.com/blogs/aws/now-available-amazon-ec2-high-memory-instances-with-6-9-and-12-tb-of-memory-perfect-for-sap-hana/ https://aws.amazon.com/blogs/aws/now-available-amazon-ec2-hi... [1] https://cloud.google.com/blog/products/sap-google-cloud/announcing-the-general-availability-of-6-and-12tb-vms-for-sap-hana-instances-on-gcp https://cloud.google.com/blog/products/sap-google-cloud/anno... [2] https://azure.microsoft.com/en-us/updates/azure-mv2-series-vms-offering-up-to-12tb-memory-are-now-ga-in-eu-west-and-eu-north-regions/ https://azure.microsoft.com/en-us/updates/azure-mv2-series-v...
qaq 2y ago
You can have 8TB RAM in a 2U box for under 100K. grab a couple and it will save you millions a year compared to over-engineered bigdata setup.
- apwell23 2y ago
  Bigquery and snowflake are software. They come with a sql engine, data governance, integration with your ldap, auditing. Loading data into snowflake isn't overegineering. What you described is over-engineering. No business is passing 6tb data around on their laptops.
  - qaq 2y ago
    So is ClickHouse your point being ? Please point out what a server being able to have 8TB of RAM has to do with laptops.
- int_19h 2y ago
  I wonder how much this costs: https://www.ibm.com/products/power-e1080 https://www.ibm.com/products/power-e1080 And how that price would compare to the equivalent big data solution in the cloud.
- chx 2y ago
  1U box too.
chx 2y ago
If my business depended on it? I can click a few buttons and have a 8TiB Supermicro server on my doorstep in a few days if I wanted to colo that. EC2 High Memory instances offer 3, 6, 9, 12, 18, and 24 TiB of memory in an instance if that's the kind of service you want. Azure Mv2 also does 2850 - 11400GiB. So yes, if need to be, I have 6 TiB of RAM.
david_allison 2y ago
https://yourdatafitsinram.net/ https://yourdatafitsinram.net/
- compressedgas 2y ago
  Was posted as https://news.ycombinator.com/item?id=9581862 https://news.ycombinator.com/item?id=9581862 in 2015
bluedino 2y ago
We are decomming our 5-year old 4TB systems this year and could have been ordered with more
lizknope 2y ago
I personally don't but our computer cluster at work as around 50,000 CPU cores. I can request specific configurations through LSF and there are at least 100 machines with over 4TB RAM and that was 3 years ago. By now there are probably machines with more than that. Those machines are usually reserved for specific tasks that I don't do but if I really needed it I could get approval.

You have 6 TiB of ram?

by boppo1 2y ago

You have 6 TiB of ram?

ninkendo 2y ago

You don’t need that much ram to use mmap(2)

marginalia_nu 2y ago
To be fair, mmap doesn't put your data in RAM, it presents it as though it was in RAM and has the OS deal with whether or not it actually is.
- ninkendo 2y ago
  Right, which is why you can mmap way more data than you have ram, and treat it as though you do have that much ram. It’ll be slower, perhaps by a lot, but most “big data” stuff is already so god damned slow that mmap probably still beats it, while being immeasurably simpler and cheaper.
  - marginalia_nu 2y ago
    Really depends on the shape of the data. mmap can be suboptimal in many cases. For CSV it flat out doesn't matter what you do since the format is so inefficient and needs to be read start to finish, but something like parquet probably benefits from explicit read syscalls, since it's block based and highly structured, where you can predict the read patterns much better than the kernel can.

cess11 2y ago

The "(multiple times)" part probably means batching or streaming. But yeah, they might have that much RAM. At a rather small company I was at we had a third of it in the virtualisation cluster. I routinely put customer databases in the hundreds of gigabytes into RAM to do bug triage and fixing.

kmarc 2y ago
Indeed, what I meant to say is that you can load it in multiple batches. However, now thinking, I did play around with servers of TiBs of memory :-)

vitus 2y ago

If you're one of the public clouds targeting SAP use cases, you probably have some machines with 12TB [0, 1, 2]. [0] https://aws.amazon.com/blogs/aws/now-available-amazon-ec2-high-memory-instances-with-6-9-and-12-tb-of-memory-perfect-for-sap-hana/ https://aws.amazon.com/blogs/aws/now-available-amazon-ec2-hi... [1] https://cloud.google.com/blog/products/sap-google-cloud/announcing-the-general-availability-of-6-and-12tb-vms-for-sap-hana-instances-on-gcp https://cloud.google.com/blog/products/sap-google-cloud/anno... [2] https://azure.microsoft.com/en-us/updates/azure-mv2-series-vms-offering-up-to-12tb-memory-are-now-ga-in-eu-west-and-eu-north-regions/ https://azure.microsoft.com/en-us/updates/azure-mv2-series-v...

qaq 2y ago

You can have 8TB RAM in a 2U box for under 100K. grab a couple and it will save you millions a year compared to over-engineered bigdata setup.

apwell23 2y ago
Bigquery and snowflake are software. They come with a sql engine, data governance, integration with your ldap, auditing. Loading data into snowflake isn't overegineering. What you described is over-engineering. No business is passing 6tb data around on their laptops.
- qaq 2y ago
  So is ClickHouse your point being ? Please point out what a server being able to have 8TB of RAM has to do with laptops.
int_19h 2y ago
I wonder how much this costs: https://www.ibm.com/products/power-e1080 https://www.ibm.com/products/power-e1080 And how that price would compare to the equivalent big data solution in the cloud.
chx 2y ago
1U box too.

chx 2y ago

If my business depended on it? I can click a few buttons and have a 8TiB Supermicro server on my doorstep in a few days if I wanted to colo that. EC2 High Memory instances offer 3, 6, 9, 12, 18, and 24 TiB of memory in an instance if that's the kind of service you want. Azure Mv2 also does 2850 - 11400GiB. So yes, if need to be, I have 6 TiB of RAM.

david_allison 2y ago

https://yourdatafitsinram.net/ https://yourdatafitsinram.net/

compressedgas 2y ago
Was posted as https://news.ycombinator.com/item?id=9581862 https://news.ycombinator.com/item?id=9581862 in 2015

bluedino 2y ago

We are decomming our 5-year old 4TB systems this year and could have been ordered with more

lizknope 2y ago

I personally don't but our computer cluster at work as around 50,000 CPU cores. I can request specific configurations through LSF and there are at least 100 machines with over 4TB RAM and that was 3 years ago. By now there are probably machines with more than that. Those machines are usually reserved for specific tasks that I don't do but if I really needed it I could get approval.