7 ms·

This was nicely foreseen in the original Map - Reduce paper, where the authors write: > The issues of how to parallelize the computation, distribute the data

• dijit 3y ago
  There is an incentive for people to go horizontal rather than permitting themselves to go vertical. Makes sense, we are told that vertical has limits in university and we should prioritise horizontal; but I feel a little like the "mid-wit" meme, once we realise how vertical we can go then we can end up using significantly fewer resources in aggregate (as there is overhead in distributed systems of course). I also think we are disincentivised from going vertical as most cloud providers prioritise splitting workloads, most people don't have 16TiB of RAM available to them, but they might have a credit card on file for a cloud provider/hyperscaler. *EDIT*: Largest AWS Instance is, I think, the x2iedn.metal ith 128vCPU and 4TiB RAM *EDIT2*: u-24tb1.metal seems larger; 448vCPU and 24TiB Memory, but I'm not sure if you can actually use it for anything that's not SAP HANA.
  • oblio 3y ago
    The problem is that you do want some horizontal scaling regardless, just to avoid SPOFs as much as you can.
    • dijit 3y ago
      If you can't handle 99.97% uptime for data processing then probably there's a larger issue at play.
      • jcgrillo 3y ago
        That's about 13.4min/mo of downtime, every month. That seems likely to cause all kinds of havoc at scale.
        • dijit 3y ago
          Maybe we're talking about different things then. My laptop is a SPoF in exactly the same way. If my laptop is closed then data collection will still happen, as collection and processing are different systems; but my ability to mutate the data hands-on is affected. Thus any downtime of my laptop is not really a problem. See also: Jupyter notebooks, Excel, etc; I will also point out that robustness in distributed systems is not as cut and dry for two reasons: 1: These are not considered hot-path systems that are mission critical so will be neglected by SRE. 2: Complexity is increased in distributed systems, thus you have more likelihood of failure until you have a lot of effort put into it.
          • jcgrillo 3y ago
            Yes, I believe we are talking about different things. In my experience the hadoop (or mapR) cluster ended up getting used for a bunch of heterogenous workloads running simultaneously at different priorities. High priority workloads were production impacting batch jobs where downtime would be noticed by users. Lower priority workloads were as you describe--analysts running ad-hoc jobs to support BI, data science, operations, etc. Hbase also ran on that infrastructure serving real-time workloads. Downtime on any of the Hbase clusters would be a high severity outage. So minutes/mo of downtime would certainly have unacceptable business impact. Another important thing is replication. Drives do fail, and if a single drive failure brings down prod how long would that take to fix? To be clear in general my opinions are aligned with the article, I think using the whole machine at high utilization is the only environmentally (and financially) responsible way. But I don't believe it's true that purely vertical scaling is realistic for most businesses. EDIT: there are also security and compliance concerns that rule out the scenario of copying data onto an employee laptop. I guess what I'm trying to get at is the scenario seems a little contrived.
    • rcxdude 3y ago
      If you just need high availability, then you don't need to scale horizontally, you just need redundant systems.
  • zzbn00 3y ago
    Horizontal scaling did have specific incentives when Map Reduce got going and today also in the right parameter space. For example, I think Dean & Ghemawat reasonably describe what were their incentives: saving capital by reusing an already distributed set of machines while conserving network bandwidth. In table 1 they write average job duration was around 10 minutes involving 150 computers and that on average 1.2 workers died per such job! The computers had 2-4 GiB memory, 100megabit ethernet and ISA HDDs. In 2003 when they got map reduce going Google's total R&D budget was $90million. There was no cloud so if you wanted a large machine you had to pay up front. What they did with Map Reduce is a great achievement. But I would advise against scaling horizontally right from the start because we may need to scale horizontally at some time in future. If it will fit on one machine, do it on one.
    • em500 3y ago
      Maybe it was a great achievement for Google, but outside of Google I guess approximately nobody rolling out MapReduce or Hadoop read Dean & Ghemawat, resulting in countless analysts waiting 10 minutes to view a few spreadsheet sized tables that used to open in Excel in a few seconds.
  • jonstewart 3y ago
    MapReduce came along at a moment in time where going horizontal was -essential-. Storage had kept increasing faster than CPU and memory, and CPUs in the aughts encountered two significant hitches: the 32-bit to 64-bit transition and the multicore transition. As always, software lagged these hardware transitions; you could put 8 or 16GB of RAM in a server, but good luck getting Java to use it. So there was a period of several years where the ceiling on vertical scalability was both quite low and absurdly expensive. Meanwhile, hard drives and the internet got big.
    • nickpsecurity 3y ago
      I’ll add context that NUMA machines with high CPU’s and RAM used to cost six to seven digits. Some setups were eight figures. They had proprietary software, too. People came up with horizontal scaling across COTS hardware, often called Beowulf clusters, to have more computing for cheaper. They’d run UNIX or Linux with a growing collection of open-source tools. They’d be able to get the most out of their compute by customizing it to their needs. So, vertical scaling being exorbitantly expensive and less flexible at the time, too.
    • zzbn00 3y ago
      For the Map Reduce specifically the one of the big issues was the speed at which you could read data from a HDD and transfer across the network. The MapReduce paper benchmarks were done with computers with 160 GB HDDs (so 3x smaller than typical NVMe SSD today) which had sequential read of maybe 40MB/s (100x smaller than a NVMe Drive today!) and random reads of <1MB/s (also very much smaller than a NVMe Drive today). On the other hand they had 2GHz Xeon CPUs! Table 1 in the paper suggests that average read throughput per worker for typical jobs was around 1MB/s.
      • jonstewart 3y ago
        Maybe more like 80MB/s? But yeah, good point, sequential reads were many times faster than random access, yet on a single disk the sequential transfer rate increases were still not keeping up with storage rate increases, nor CPU speed increases. MapReduce/Hadoop gave you a way to have lots of disks operating sequentially.
  • edgyquant 3y ago
    Plus horizontal scaling is sexier
    • pas 3y ago
      horizontal is Real Scaling™, vertical is just preparing a bigger and brighter sacrifice on the altar of SPoF. (note, comparing a 2 node active-passive hot-spare setup with a bazillion node horizontal hellscape is not in scope.)
      • roncesvalles 3y ago
        SPoF is only a problem if failure is a problem.