5 ms·

It's actually an interesting question you can get 192 cores and 12 TB RAM in a single x86 box. At what point does it actually make sense to go for Hadoop.

by qaq 8y ago

obelix_ 8y ago
It makes sense if you are handling millions of requests from all over the world per second and need failovers if machines go down. But...if you just want to run your own personal search engine say... Then for Wikipedia/Stackoverflow/Quora size datasets (50GB with with 10GB worth of every kind of index(regex/geo/full text etc) ) you can run real time indexing on live updates with all the advanced search options you see under their "advanced search" page one any random Dell or HP Desktop with about 6-8GB of RAM. Lots of people do this on Wall Street. People don't get what is possible on desktop cause so much of it has moved to the cloud. It will come back to desktop imho.
- nl 8y ago
  It won't come back to desktops because as they get cheaper the costs of people to maintain them increases. There will always be people who need them and use them but that proportion is going to keep decreasing (I'm somewhat sad about this, but the math is hard to argue with).
  - obelix_ 8y ago
    Just temporary. Nature did not need to invent cloud computing to perform massive computation. The speed and data involved in computation going on in a cell or an ants brain show us how far we can still go on desktop. The breakthroughs will come. In the meantime (unless you are dealing with video) most text and image datasets out there that avg Joe needs can easily be stored/processed entirely locally thanks to cheap terabyte drives/multicore chips these days. People just haven't realized there isn't that much useable textual data OR that local computing doesn't require all the overhead of handling millions of requests a second. This is Google problem not an avg Joe problem that is being solved with cloud compute.
    - nl 8y ago
      You seem to have missed everything the comment you are replying to says. There is no dispute with the size or amount of compute available on desktops.
      - obelix_ 8y ago
        No idea what cost that comment refers too. If you are Facebook or YouTube or Twitter or Amazon sure you need your five thousand man PAID army of content reviewers to keep the data clean cause the data is mostly useless. But if you are running Wikipedia search or Stackoverflow's search or the team at the library of congress please go and take a look at the size of these teams and their budgets and their growth rates.
        nl 8y ago
        Are you conflating "single server" and "desktop computer" here? It sounds a lot like you might be? Because almost everyone already uses the model of co-locating the search index and query code on a single computer (both Wikipedia and Stackoverflow use Elastic Search which does this). They use multiple physical servers because of the number of simultaneous requests they serve. This has never been the use-case for Hadoop. I've built Hadoop based infrastructure for redundantly storing multiple PB of unstructured data with Spark on top for analysis. This is completely different to search. That's very different to the Wall St analyst running desktop analysis in Matlab, or the oil/gas exploration team doing the same thing.
potatoyogurt 8y ago
That's a good point. It's really hard to give a clear cutoff because there is quite a bit that you can potentially do on a single machine now. But there are disadvantages to doing your processing on a single really big machine even when you can find a way to handle it. If you have inconsistent workloads, a single machine may be harder to schedule to be close to fully allocated, and if it goes down, then your workflow is dead, possibly until you take some sort of manual action, whereas a cluster of machines is already built to be resilient to individual node failures. It all depends on the use case, which makes it really tough to give hard and fast rules.
- mmt 8y ago
  > a single machine may be harder to schedule to be close to fully allocated That seems remarkably unlikely if there truly is only the one. Maybe I don't understand what you mean, though. How is that any different from having a single distributed cluster, instead? > if it goes down I hear (read) this quite a bit, especially recently, and I'm a bit mystified that it's even brought up in this day and age. Firstly, having worked with (what is now) commodity hardware for over a decade, I believe that people who haven't grossly over-estimate how often "it goes down", especially today. This overestimation is trotted out as a reason that operating ones own hardware is such a "nightmare" and therefore one must always use cloud or a VPS. With a "large" enough server, the risk goes up, of course. More DIMMs means more memory can fail, but we're still talking about low single digit percent for all errors (including correctable). IIRC, CPUs have even lower failure rates. Everything else tends to be redundant. Even then, your workflow isn't dead. It's just missing a DIMM or a CPU, possibly after a reboot (which won't be, if you configured it right). In many cases, downtime isn't actually caused by hardware but by software (or humans). That's not going to be unique to centralized processing versus distributed. Also, if the single machine is on a cloud provider, many hardware and utilization issues can be abstracted anyway, for a huge price premium.
  - potatoyogurt 8y ago
    Just replying to the top bit, since we're exchanging multiple comment chains. Hadoop, Spark, etc. are built on top of a job scheduler, so if you have multiple competing jobs, it will allocate tasks to resources in a fairly reasonable way. If there is too much contention at some point in time, your jobs will all still run, and portions that can't be scheduled immediately will be delayed while they wait for free nodes. The processing model also generally leads to independence between jobs. You can do job scheduling on a single machine too, of course, but you get it for free with Hadoop/YARN. You don't get it for free with chained bash scripts or a custom single-node solution. And without sophisticated job scheduling, you can't really come too close to fully allocating your machine (when you're not watching carefully), because you'll accidentally exhaust resources and break random jobs. edit: if you're running on a cloud, you also may be able to autoscale to deal with spiky usage patterns.
threeseed 8y ago
Your comment is actually pretty funny as the entire point of Hadoop was to be able to use commodity, cheap, off the shell PC hardware as opposed to the exotic specifications you mention there. Except that of course nowadays such hardware is just a couple of clicks away in AWS.
- beagle3 8y ago
  The point of hadoop might have been that, but it never actually delievered any real value to most users - it's an abysmal failure from a computing efficiency point of view; here's an example http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...
  - threeseed 8y ago
    I've been using Spark for many years going back to 1.0. It is the foundation technology of almost every data science team around the world. And your misguided post (which for some weird reason only focuses on graph algorithms) doesn't change that. And not sure why you think it's inefficient. We run 30 node, autoscaling clusters which stay close to 100% for most of the time.
    - beagle3 8y ago
      Spark is significantly more efficient than Hadoop. I don’t know about your specific workload, but i’ve seen quite a few Hadoop setups that were at 100% load most of the time, and were replaced by relatively simple non Hadoop based code that used 2% to 10% of the hardware and ran about as fast. I didn’t spend much time evaluating the “pre”, but at least one workload spent 90% of the 100% on [de]serialization. It’s not my link, it is Frank McSherry who is commenting in this thread - I hope he can chime in on why he chose this specific example - but it correlates very well with my experience.
    - mmt 8y ago
      > We run 30 node, autoscaling clusters which stay close to 100% for most of the time. I have exactly zero knowledge on Spark's efficiency as well as zero on how representative graph algorithms are, but I can confidently say that the above statement fails to refute the referenced article's thesis (which, arguably, criticizes assertions just like that). Just because your implementation scales (even autoscales) to use more compute resources says nothing about its efficiency (overall or even marginal when adding more nodes, i.e. the shape of the curve). Computer science has struggled with achieving even near linear-scalability ever since the advent of SMP.

It's actually an interesting question you can get 192 cores and 12 TB RAM in a single x86 box. At what point does it actually make sense to go for Hadoop.

by qaq 8y ago

It's actually an interesting question you can get 192 cores and 12 TB RAM in a single x86 box. At what point does it actually make sense to go for Hadoop.

obelix_ 8y ago

It makes sense if you are handling millions of requests from all over the world per second and need failovers if machines go down. But...if you just want to run your own personal search engine say... Then for Wikipedia/Stackoverflow/Quora size datasets (50GB with with 10GB worth of every kind of index(regex/geo/full text etc) ) you can run real time indexing on live updates with all the advanced search options you see under their "advanced search" page one any random Dell or HP Desktop with about 6-8GB of RAM. Lots of people do this on Wall Street. People don't get what is possible on desktop cause so much of it has moved to the cloud. It will come back to desktop imho.

nl 8y ago
It won't come back to desktops because as they get cheaper the costs of people to maintain them increases. There will always be people who need them and use them but that proportion is going to keep decreasing (I'm somewhat sad about this, but the math is hard to argue with).
- obelix_ 8y ago
  Just temporary. Nature did not need to invent cloud computing to perform massive computation. The speed and data involved in computation going on in a cell or an ants brain show us how far we can still go on desktop. The breakthroughs will come. In the meantime (unless you are dealing with video) most text and image datasets out there that avg Joe needs can easily be stored/processed entirely locally thanks to cheap terabyte drives/multicore chips these days. People just haven't realized there isn't that much useable textual data OR that local computing doesn't require all the overhead of handling millions of requests a second. This is Google problem not an avg Joe problem that is being solved with cloud compute.
  - nl 8y ago
    You seem to have missed everything the comment you are replying to says. There is no dispute with the size or amount of compute available on desktops.
    - obelix_ 8y ago
      No idea what cost that comment refers too. If you are Facebook or YouTube or Twitter or Amazon sure you need your five thousand man PAID army of content reviewers to keep the data clean cause the data is mostly useless. But if you are running Wikipedia search or Stackoverflow's search or the team at the library of congress please go and take a look at the size of these teams and their budgets and their growth rates.
      - nl 8y ago
        Are you conflating "single server" and "desktop computer" here? It sounds a lot like you might be? Because almost everyone already uses the model of co-locating the search index and query code on a single computer (both Wikipedia and Stackoverflow use Elastic Search which does this). They use multiple physical servers because of the number of simultaneous requests they serve. This has never been the use-case for Hadoop. I've built Hadoop based infrastructure for redundantly storing multiple PB of unstructured data with Spark on top for analysis. This is completely different to search. That's very different to the Wall St analyst running desktop analysis in Matlab, or the oil/gas exploration team doing the same thing.

potatoyogurt 8y ago

That's a good point. It's really hard to give a clear cutoff because there is quite a bit that you can potentially do on a single machine now. But there are disadvantages to doing your processing on a single really big machine even when you can find a way to handle it. If you have inconsistent workloads, a single machine may be harder to schedule to be close to fully allocated, and if it goes down, then your workflow is dead, possibly until you take some sort of manual action, whereas a cluster of machines is already built to be resilient to individual node failures. It all depends on the use case, which makes it really tough to give hard and fast rules.

mmt 8y ago
> a single machine may be harder to schedule to be close to fully allocated That seems remarkably unlikely if there truly is only the one. Maybe I don't understand what you mean, though. How is that any different from having a single distributed cluster, instead? > if it goes down I hear (read) this quite a bit, especially recently, and I'm a bit mystified that it's even brought up in this day and age. Firstly, having worked with (what is now) commodity hardware for over a decade, I believe that people who haven't grossly over-estimate how often "it goes down", especially today. This overestimation is trotted out as a reason that operating ones own hardware is such a "nightmare" and therefore one must always use cloud or a VPS. With a "large" enough server, the risk goes up, of course. More DIMMs means more memory can fail, but we're still talking about low single digit percent for all errors (including correctable). IIRC, CPUs have even lower failure rates. Everything else tends to be redundant. Even then, your workflow isn't dead. It's just missing a DIMM or a CPU, possibly after a reboot (which won't be, if you configured it right). In many cases, downtime isn't actually caused by hardware but by software (or humans). That's not going to be unique to centralized processing versus distributed. Also, if the single machine is on a cloud provider, many hardware and utilization issues can be abstracted anyway, for a huge price premium.
- potatoyogurt 8y ago
  Just replying to the top bit, since we're exchanging multiple comment chains. Hadoop, Spark, etc. are built on top of a job scheduler, so if you have multiple competing jobs, it will allocate tasks to resources in a fairly reasonable way. If there is too much contention at some point in time, your jobs will all still run, and portions that can't be scheduled immediately will be delayed while they wait for free nodes. The processing model also generally leads to independence between jobs. You can do job scheduling on a single machine too, of course, but you get it for free with Hadoop/YARN. You don't get it for free with chained bash scripts or a custom single-node solution. And without sophisticated job scheduling, you can't really come too close to fully allocating your machine (when you're not watching carefully), because you'll accidentally exhaust resources and break random jobs. edit: if you're running on a cloud, you also may be able to autoscale to deal with spiky usage patterns.

threeseed 8y ago

Your comment is actually pretty funny as the entire point of Hadoop was to be able to use commodity, cheap, off the shell PC hardware as opposed to the exotic specifications you mention there. Except that of course nowadays such hardware is just a couple of clicks away in AWS.

beagle3 8y ago
The point of hadoop might have been that, but it never actually delievered any real value to most users - it's an abysmal failure from a computing efficiency point of view; here's an example http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...
- threeseed 8y ago
  I've been using Spark for many years going back to 1.0. It is the foundation technology of almost every data science team around the world. And your misguided post (which for some weird reason only focuses on graph algorithms) doesn't change that. And not sure why you think it's inefficient. We run 30 node, autoscaling clusters which stay close to 100% for most of the time.
  - beagle3 8y ago
    Spark is significantly more efficient than Hadoop. I don’t know about your specific workload, but i’ve seen quite a few Hadoop setups that were at 100% load most of the time, and were replaced by relatively simple non Hadoop based code that used 2% to 10% of the hardware and ran about as fast. I didn’t spend much time evaluating the “pre”, but at least one workload spent 90% of the 100% on [de]serialization. It’s not my link, it is Frank McSherry who is commenting in this thread - I hope he can chime in on why he chose this specific example - but it correlates very well with my experience.
  - mmt 8y ago
    > We run 30 node, autoscaling clusters which stay close to 100% for most of the time. I have exactly zero knowledge on Spark's efficiency as well as zero on how representative graph algorithms are, but I can confidently say that the above statement fails to refute the referenced article's thesis (which, arguably, criticizes assertions just like that). Just because your implementation scales (even autoscales) to use more compute resources says nothing about its efficiency (overall or even marginal when adding more nodes, i.e. the shape of the curve). Computer science has struggled with achieving even near linear-scalability ever since the advent of SMP.