4 ms·

Arguable if you can keep everything on one box it will almost always be faster (and cheaper!) than any soft of distributed system. That said, scalability is gen

by edude03 10y ago

Arguable if you can keep everything on one box it will almost always be faster (and cheaper!) than any soft of distributed system. That said, scalability is generally more important than speed because once a task can be distributed you can add performance by adding hardware. As well, depending on your use case you can often get fault tolerance "for free".

TickleSteve 10y ago
No. Show that you need scalability first. Chances are you don't. When you do, scale the smallest possible part of your system that is the bottleneck, not the whole thing.
- lmm 10y ago
  An established, standardised, existing platform is often more maintainable than a custom solution, even if that platform includes a bit more scalability than you actually need.
  - HelloNurse 10y ago
    But straightforward use of cat, grep, xargs and gawk is much less "custom" than depending on specific versions of exotic tools and committing servers for use as part of the current "platform". If I want to run a simple pipeline of UNIX tools on a random system, the worst case prerequisite scenario is reclaiming disk space and installing less common tools (e.g. mawk).
    - lmm 10y ago
      > But straightforward use of cat, grep, xargs and gawk is much less "custom" than depending on specific versions of exotic tools and committing servers for use as part of the current "platform". There are often subtle incompatibilities between the versions of those tools found on different systems (e.g. GNU vs BSD). Worse, there's no test suite or way to declare the dependency, and you may not even notice until the job fails halfway through. Whereas on a Spark job built with maven or similar the dependencies are at least explicit and very easy to reproduce. "Exotic" is relative. At my current job I would expect a greater proportion of my colleagues to know Spark than gawk. Unix is a platform too - and an underspecified one with poor release/version discipline at that. In an organization where Unix is standard and widely understood and Spark is not, use Unix; where the reverse applies, it's often more maintainable to use Spark, even if you don't need distribution. But my original point was: if you need a little more scalability than Unix offers then it may well be worth going all the way to Spark (even though it's far more scalability than you likely need) rather than hacking up some custom job-specific solution that scales just as far as you need, just because Spark is standard, documented and all that.
      - qwertyuiop924 10y ago
        You can look for the GNU utils, you know. And you can do explicit dependancy declaration by grepping the version output to see if it's GNU or bsd, and the version number. It's not convenient, and it could be done better, but it is by no means impossible. Also, how do your colleagues not know AWK? In a primarily UNIX world, that's something that everybody should know. Besides, you can learn the basics in about 15 minutes.
        lmm 10y ago
        > You can look for the GNU utils, you know. And you can do explicit dependancy declaration by grepping the version output to see if it's GNU or bsd, and the version number. > It's not convenient, and it could be done better, but it is by no means impossible. In principle it may be possible, but in practice it's vaporware at best. There's no standard, established way to do this, which means there's no way that other people could maintain - the closest is probably autoconf, but I never saw anyone ship an aclocal.m4 with their shell script. Since unix utilities tend to be installed system-wide, it's not really practical to develop or test (not that the unix platform is at all test-friendly to start with) against old target versions (docker may eventually get there, but it's currently immature) - if your development machine has version 2.6 of some utility, you'll probably end up accidentally using 2.6-only features without noticing. > Also, how do your colleagues not know AWK? In a primarily UNIX world, that's something that everybody should know. Besides, you can learn the basics in about 15 minutes. My organization isn't primarily-unix, and people don't use awk often enough to learn it. I'm sure one could pick up the basics fairly quickly, but that's true of Spark too.
        qwertyuiop924 10y ago
        Okay, I guess. For utilities, it's usually safe to assume that whoever is running it is running a comparable version - the most commonly used options are decades old at this point, and it's relatively unlikely you'll use the new stuff. As for checking for GNU tooling, a grep against <tool> -v does the trick. This can also get you the version number. You can probably even write a command to do it. It's nonstandard and suboptimal, but it is, once again, possible.
zzzcpan 10y ago
There is nothing preventing distributed systems to be faster than one box for this kind of thing. But they don't always bother to pursue efficiency on that level, because things are very different once you have a lot of boxes and something that used to look important for a couple of boxes doesn't anymore.
- kuschku 10y ago
  Yes, there is, you have a lot of overhead in any case for the same tools.
  - zzzcpan 10y ago
    You don't have the same tools. You are probably thinking about emulating POSIX filesystem API and things like that and using those command-line tools on top of that in a single-box kind of way. That's not how you treat your distributed system. EDIT: For something that beats a single box easily I envision an interpreter with JIT running on each node in a distributed system and on the same process that stores data, having pretty much no overhead to access and process it.
    - qwertyuiop924 10y ago
      >You are probably thinking about emulating POSIX filesystem API and things like that and using those command-line tools on top of that in a single-box kind of way. That's not how you treat your distributed system. Yeah, but Manta's mapreduce does something close, and it seems to work okay.
- arielb1 10y ago
  Fancy highly-scalable distributed algorithms have that annoying tendency of starting at 10x slower than the most naïve single-machine algorithm.
toast0 10y ago
My hadoop experience is dated (circa 2011), do the work nodes still poll the scheduler to see if they have work to do? If so, that's still a giant impediment to speed for smaller tasks. Especially if poll times are in the range of minutes. If hadoop put effort into making small tasks time efficient, I think your argument has merit, if there's a reasonable chance of actually needing to scale, or to pick up ancillary benefits (fault tolerance, access to other data that needs to be processed with hadoop etc)

by edude03 10y ago

TickleSteve 10y ago

No. Show that you need scalability first. Chances are you don't. When you do, scale the smallest possible part of your system that is the bottleneck, not the whole thing.

lmm 10y ago
An established, standardised, existing platform is often more maintainable than a custom solution, even if that platform includes a bit more scalability than you actually need.
- HelloNurse 10y ago
  But straightforward use of cat, grep, xargs and gawk is much less "custom" than depending on specific versions of exotic tools and committing servers for use as part of the current "platform". If I want to run a simple pipeline of UNIX tools on a random system, the worst case prerequisite scenario is reclaiming disk space and installing less common tools (e.g. mawk).
  - lmm 10y ago
    > But straightforward use of cat, grep, xargs and gawk is much less "custom" than depending on specific versions of exotic tools and committing servers for use as part of the current "platform". There are often subtle incompatibilities between the versions of those tools found on different systems (e.g. GNU vs BSD). Worse, there's no test suite or way to declare the dependency, and you may not even notice until the job fails halfway through. Whereas on a Spark job built with maven or similar the dependencies are at least explicit and very easy to reproduce. "Exotic" is relative. At my current job I would expect a greater proportion of my colleagues to know Spark than gawk. Unix is a platform too - and an underspecified one with poor release/version discipline at that. In an organization where Unix is standard and widely understood and Spark is not, use Unix; where the reverse applies, it's often more maintainable to use Spark, even if you don't need distribution. But my original point was: if you need a little more scalability than Unix offers then it may well be worth going all the way to Spark (even though it's far more scalability than you likely need) rather than hacking up some custom job-specific solution that scales just as far as you need, just because Spark is standard, documented and all that.
    - qwertyuiop924 10y ago
      You can look for the GNU utils, you know. And you can do explicit dependancy declaration by grepping the version output to see if it's GNU or bsd, and the version number. It's not convenient, and it could be done better, but it is by no means impossible. Also, how do your colleagues not know AWK? In a primarily UNIX world, that's something that everybody should know. Besides, you can learn the basics in about 15 minutes.
      - lmm 10y ago
        > You can look for the GNU utils, you know. And you can do explicit dependancy declaration by grepping the version output to see if it's GNU or bsd, and the version number. > It's not convenient, and it could be done better, but it is by no means impossible. In principle it may be possible, but in practice it's vaporware at best. There's no standard, established way to do this, which means there's no way that other people could maintain - the closest is probably autoconf, but I never saw anyone ship an aclocal.m4 with their shell script. Since unix utilities tend to be installed system-wide, it's not really practical to develop or test (not that the unix platform is at all test-friendly to start with) against old target versions (docker may eventually get there, but it's currently immature) - if your development machine has version 2.6 of some utility, you'll probably end up accidentally using 2.6-only features without noticing. > Also, how do your colleagues not know AWK? In a primarily UNIX world, that's something that everybody should know. Besides, you can learn the basics in about 15 minutes. My organization isn't primarily-unix, and people don't use awk often enough to learn it. I'm sure one could pick up the basics fairly quickly, but that's true of Spark too.
        qwertyuiop924 10y ago
        Okay, I guess. For utilities, it's usually safe to assume that whoever is running it is running a comparable version - the most commonly used options are decades old at this point, and it's relatively unlikely you'll use the new stuff. As for checking for GNU tooling, a grep against <tool> -v does the trick. This can also get you the version number. You can probably even write a command to do it. It's nonstandard and suboptimal, but it is, once again, possible.

zzzcpan 10y ago

There is nothing preventing distributed systems to be faster than one box for this kind of thing. But they don't always bother to pursue efficiency on that level, because things are very different once you have a lot of boxes and something that used to look important for a couple of boxes doesn't anymore.

kuschku 10y ago
Yes, there is, you have a lot of overhead in any case for the same tools.
- zzzcpan 10y ago
  You don't have the same tools. You are probably thinking about emulating POSIX filesystem API and things like that and using those command-line tools on top of that in a single-box kind of way. That's not how you treat your distributed system. EDIT: For something that beats a single box easily I envision an interpreter with JIT running on each node in a distributed system and on the same process that stores data, having pretty much no overhead to access and process it.
  - qwertyuiop924 10y ago
    >You are probably thinking about emulating POSIX filesystem API and things like that and using those command-line tools on top of that in a single-box kind of way. That's not how you treat your distributed system. Yeah, but Manta's mapreduce does something close, and it seems to work okay.
arielb1 10y ago
Fancy highly-scalable distributed algorithms have that annoying tendency of starting at 10x slower than the most naïve single-machine algorithm.

toast0 10y ago

My hadoop experience is dated (circa 2011), do the work nodes still poll the scheduler to see if they have work to do? If so, that's still a giant impediment to speed for smaller tasks. Especially if poll times are in the range of minutes. If hadoop put effort into making small tasks time efficient, I think your argument has merit, if there's a reasonable chance of actually needing to scale, or to pick up ancillary benefits (fault tolerance, access to other data that needs to be processed with hadoop etc)