6 ms·

Large-scale storage clusters like Hadoop, Cassandra, ElasticSearch are generally slow, expensive, hard to set up properly and require a lot of monitoring to sta

Large-scale storage clusters like Hadoop, Cassandra, ElasticSearch are generally slow, expensive, hard to set up properly and require a lot of monitoring to stay healthy. Use them only when other solutions won't do. If your data will fit in a set of text files or a cluster of relational databases, use those. Even if you plan on storing a gazillion TB of data, it's faster to iterate app logic on nimble storage solutions first, before entertaining something bigger. Where the large scale storage clusters shine is when the sheer scale of data won't fit in anything else, i.e. there's no other (sane) choice.

chockchocschoir 5y ago
> Where the large scale storage clusters shine is when the sheer scale of data won't fit in anything else, i.e. there's no other (sane) choice. And when people say "won't fit in anything else", do explore your options before! RAM storage can be very, very, very big and the cost for using it is minuscule compared to what it used to be! On that note, there is this great website for seeing if there are servers that can handle fitting all your data in RAM: https://yourdatafitsinram.net/ https://yourdatafitsinram.net/ Systems like Power System E980 can handle up to 64TB RAM, which is a lot of data. Just like parent said here, do try to fit things on machines like this before even trying out large-scale storage clusters, because they are a hassle to deal with and generally not worth the cost. Edit: For the Cloud-hosting crowd out there, I took a quick look at https://instances.vantage.sh/ https://instances.vantage.sh/ and sorted it by memory. Largest instances you can get at AWS is "U-24TB1 Metal", which comes with 24TiB of RAM (and 448 vCPU so the computation itself gets as fast as possible too [granted you can parallelize it]), which should fit most cases of "big data" I've seen in the wild today. Unsure about running costs though, but I'm 90% sure it's cheaper to have that instance for a couple of hours than the cost for having to deal with storage clusters, which are also slower.
- phkahler 5y ago
  >> And when people say "won't fit in anything else", do explore your options before! RAM storage can be very, very, very big and the cost for using it is minuscule compared to what it used to be! That's why the "big data" industry also encourages collection of absolutely every bit of data you can find. They want you to need their tools. You may not think there's a use for it, but vague promises of AI finding needles in the haystack are used to get you to keep it and bloat your system.
  - sitkack 5y ago
    This is why the first, possibly most important phase of a data analytics job is to filter down to the subset of the data that you actually need to compute the result. Collecting it all on the front end can be very effective, as long as you can easily filter. If you can't, you are just causing yourself more problems.
- zozbot234 5y ago
  You could also use SSI (single system image) system architectures, including distributed shared memory, to seamlessly (at least wrt. software implementation) scale up from efficient single-node processing to an arbitrarily large, clustered system. AIUI this is how things are traditionally done in mainframe processing, and it is also regarded as a very meaningful possibility in HPC.
  - AitchEmArsey 5y ago
    That concept has fallen out of favour in HPC, but it'll be interesting to see if it makes a comeback. As recently as 2016, an SGI UV3000 rack-scale SMP machine with 16TB of RAM was the sort of thing you'd see on a trade-show floor - now you can get that much memory in a 4U chassis, and things will only improve further if/when Optane DIMMs take off.
- louwrentius 5y ago
  I can share with you that my solar-powered Pi3B+ hosting yourdatafitsinram is holding up quite well, being linked to on HN - a funny contrast to the kind of systems it links to in the table. Even a 10+ year old DL380G8 - can hold 1.5TB+ RAM and 24/48 cores and that hardware is dirt cheap on the second hand market.
  - fivea 5y ago
    > I can share with you that my solar-powered Pi3B+ hosting yourdatafitsinram is holding up quite well, being linked to on HN - a funny contrast to the kind of systems it links to in the table. If all a server does is serve mostly static content from memory then it's not expected to handle a demanding workload beyond networking. Therefore I fail to see the point of trying to downplay someone else's needs for TB of RAM just because you serve a site on a raspberry pi, as if that's the full extent of what anyone needs to do with a computer, particularly when the OP mentions big data.
    - louwrentius 5y ago
      You totally misinterpreted the point of my text.
- fivea 5y ago
  > Systems like Power System E980 can handle up to 64TB RAM, which is a lot of data. Doesn't that kind of system cost millions of dollars? Hadoop and the like are popular because they run on COTS hardware that costs peanuts. It hardly makes any sense to argue that spending over a million dollars is enough to drop a solution that's primarily adopted because you do not have that kind of money to throw at a problem.
pizza234 5y ago
> Where the large scale storage clusters shine is when the sheer scale of data won't fit in anything else, i.e. there's no other (sane) choice. In general, this can be an effective approach, but at least fulltext search is another story. Storing hundreds of MBs (actually, I think even tens of MBs can be problematic) in text files or a db like MySQL (whose FT engine is terrible) will result in slow fulltext searches.
Breza 5y ago
I highly recommend the book Designing Data-Intensive Applications. It talks at length about different complex data systems and then warns readers that they should only use them if absolutely necessary.

chockchocschoir 5y ago

> Where the large scale storage clusters shine is when the sheer scale of data won't fit in anything else, i.e. there's no other (sane) choice. And when people say "won't fit in anything else", do explore your options before! RAM storage can be very, very, very big and the cost for using it is minuscule compared to what it used to be! On that note, there is this great website for seeing if there are servers that can handle fitting all your data in RAM: https://yourdatafitsinram.net/ https://yourdatafitsinram.net/ Systems like Power System E980 can handle up to 64TB RAM, which is a lot of data. Just like parent said here, do try to fit things on machines like this before even trying out large-scale storage clusters, because they are a hassle to deal with and generally not worth the cost. Edit: For the Cloud-hosting crowd out there, I took a quick look at https://instances.vantage.sh/ https://instances.vantage.sh/ and sorted it by memory. Largest instances you can get at AWS is "U-24TB1 Metal", which comes with 24TiB of RAM (and 448 vCPU so the computation itself gets as fast as possible too [granted you can parallelize it]), which should fit most cases of "big data" I've seen in the wild today. Unsure about running costs though, but I'm 90% sure it's cheaper to have that instance for a couple of hours than the cost for having to deal with storage clusters, which are also slower.

phkahler 5y ago
>> And when people say "won't fit in anything else", do explore your options before! RAM storage can be very, very, very big and the cost for using it is minuscule compared to what it used to be! That's why the "big data" industry also encourages collection of absolutely every bit of data you can find. They want you to need their tools. You may not think there's a use for it, but vague promises of AI finding needles in the haystack are used to get you to keep it and bloat your system.
- sitkack 5y ago
  This is why the first, possibly most important phase of a data analytics job is to filter down to the subset of the data that you actually need to compute the result. Collecting it all on the front end can be very effective, as long as you can easily filter. If you can't, you are just causing yourself more problems.
zozbot234 5y ago
You could also use SSI (single system image) system architectures, including distributed shared memory, to seamlessly (at least wrt. software implementation) scale up from efficient single-node processing to an arbitrarily large, clustered system. AIUI this is how things are traditionally done in mainframe processing, and it is also regarded as a very meaningful possibility in HPC.
- AitchEmArsey 5y ago
  That concept has fallen out of favour in HPC, but it'll be interesting to see if it makes a comeback. As recently as 2016, an SGI UV3000 rack-scale SMP machine with 16TB of RAM was the sort of thing you'd see on a trade-show floor - now you can get that much memory in a 4U chassis, and things will only improve further if/when Optane DIMMs take off.
louwrentius 5y ago
I can share with you that my solar-powered Pi3B+ hosting yourdatafitsinram is holding up quite well, being linked to on HN - a funny contrast to the kind of systems it links to in the table. Even a 10+ year old DL380G8 - can hold 1.5TB+ RAM and 24/48 cores and that hardware is dirt cheap on the second hand market.
- fivea 5y ago
  > I can share with you that my solar-powered Pi3B+ hosting yourdatafitsinram is holding up quite well, being linked to on HN - a funny contrast to the kind of systems it links to in the table. If all a server does is serve mostly static content from memory then it's not expected to handle a demanding workload beyond networking. Therefore I fail to see the point of trying to downplay someone else's needs for TB of RAM just because you serve a site on a raspberry pi, as if that's the full extent of what anyone needs to do with a computer, particularly when the OP mentions big data.
  - louwrentius 5y ago
    You totally misinterpreted the point of my text.
fivea 5y ago
> Systems like Power System E980 can handle up to 64TB RAM, which is a lot of data. Doesn't that kind of system cost millions of dollars? Hadoop and the like are popular because they run on COTS hardware that costs peanuts. It hardly makes any sense to argue that spending over a million dollars is enough to drop a solution that's primarily adopted because you do not have that kind of money to throw at a problem.

pizza234 5y ago

> Where the large scale storage clusters shine is when the sheer scale of data won't fit in anything else, i.e. there's no other (sane) choice. In general, this can be an effective approach, but at least fulltext search is another story. Storing hundreds of MBs (actually, I think even tens of MBs can be problematic) in text files or a db like MySQL (whose FT engine is terrible) will result in slow fulltext searches.

Breza 5y ago

I highly recommend the book Designing Data-Intensive Applications. It talks at length about different complex data systems and then warns readers that they should only use them if absolutely necessary.