3 ms·

It is all about efficiency. Can you dig a canal with a tried and true shovel? Yes, you can, but you will need a lot of time and many shovels. Can you cook up

It is all about efficiency. Can you dig a canal with a tried and true shovel? Yes, you can, but you will need a lot of time and many shovels. Can you cook up bioinformatics analysis in Shell and Awk. Yes, you can, but you if you want a large scale analysis to be done in a reasonable amount of time, you roll up your sleeves and reach for compiled languages, distributed systems and so on. There are downsides in introducing new systems, but it should be weighed against the upside of suitability and efficiency.

TeMPOraL 10y ago
It would be cool if it was true in our industry. People digging canals know that to do that well, there are those-and-those types of tools useful depending on the constraints of a particular canal design and environment. In our industry? Your bioinformatic analysis can probably be done well with shell and AWK (however idiotic those tools are, but that's a topic for another day[0]), and this solution will be more efficient and can be even scaled up. But no, most people will pick the "Big Data" solutions requiring a person full-time for just configuring it and managing the machine clusters it used, all because it's trendy these days. There's this saying that "if it fits on a consumer-grade hard drive, it's not big data", and it's absolutely true. What we need is more focus on efficiency across the board, and that means things like how much it will cost (in money and time) to develop / update / maintain the solution, taking into account the actual problem that's being solved, and not some imaginary version of it. -- Also, a food for thought: over the past 15 years, computers got a million times faster. The software we use maxes-out the hardware, and it offers still essentially the same set of features we had 15 years ago, or 30 years ago. Where did that million-fold power increase go? It definitely didn't go into valuable work we can do with our computers[1]. -- [0] - coming back to probably the single biggest fail introduced by UNIX culture - doing everything in unstructured text. Every tool you use thus has to contain a half-assed parser for the input data, and produces output in a random, undocumented and often inconsistent format; a lot of shell scripting is just adding additional parsers to glue those tools together. Sad thing is, people know how to deal with structured data long before UNIX, but it became forgotten, save for the PowerShell team and the web folks who try to backport JSON into UNIX-land. [1] - except in scientific computing, video games and maybe CAD tools.
chadgeidel 10y ago
It's interesting that you use the analogy "can you dig a canal with a tried and true shovel". The attempts at building the Panama Canal [1] were basically just this, and failed. It's a great point though, and on-topic. I think a lot of times large-scale software projects are approached in exactly this way. "Let's just hire a bunch of mid-level developers (shovelers) and get them writing. We can always add more developers." It seems to be the prevailaing attitude with outsourcing as well. [1] https://en.wikipedia.org/wiki/History_of_the_Panama_Canal https://en.wikipedia.org/wiki/History_of_the_Panama_Canal
nickpsecurity 10y ago
Your Shell and Awk example is funny since simple, single-CPU tools are often faster than the Big Data tools the industry is standardizing on: http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html http://aadrake.com/command-line-tools-can-be-235x-faster-tha... http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/assets/COST.pdf It's true, though, that you should use a more complex tool if it's necessary to get the job done. It's just not clear that the complexity users or developers are dealing with are intrinsic to the problems being solved. It seems more social than anything. :)

TeMPOraL 10y ago

It would be cool if it was true in our industry. People digging canals know that to do that well, there are those-and-those types of tools useful depending on the constraints of a particular canal design and environment. In our industry? Your bioinformatic analysis can probably be done well with shell and AWK (however idiotic those tools are, but that's a topic for another day[0]), and this solution will be more efficient and can be even scaled up. But no, most people will pick the "Big Data" solutions requiring a person full-time for just configuring it and managing the machine clusters it used, all because it's trendy these days. There's this saying that "if it fits on a consumer-grade hard drive, it's not big data", and it's absolutely true. What we need is more focus on efficiency across the board, and that means things like how much it will cost (in money and time) to develop / update / maintain the solution, taking into account the actual problem that's being solved, and not some imaginary version of it. -- Also, a food for thought: over the past 15 years, computers got a million times faster. The software we use maxes-out the hardware, and it offers still essentially the same set of features we had 15 years ago, or 30 years ago. Where did that million-fold power increase go? It definitely didn't go into valuable work we can do with our computers[1]. -- [0] - coming back to probably the single biggest fail introduced by UNIX culture - doing everything in unstructured text. Every tool you use thus has to contain a half-assed parser for the input data, and produces output in a random, undocumented and often inconsistent format; a lot of shell scripting is just adding additional parsers to glue those tools together. Sad thing is, people know how to deal with structured data long before UNIX, but it became forgotten, save for the PowerShell team and the web folks who try to backport JSON into UNIX-land. [1] - except in scientific computing, video games and maybe CAD tools.

chadgeidel 10y ago

It's interesting that you use the analogy "can you dig a canal with a tried and true shovel". The attempts at building the Panama Canal [1] were basically just this, and failed. It's a great point though, and on-topic. I think a lot of times large-scale software projects are approached in exactly this way. "Let's just hire a bunch of mid-level developers (shovelers) and get them writing. We can always add more developers." It seems to be the prevailaing attitude with outsourcing as well. [1] https://en.wikipedia.org/wiki/History_of_the_Panama_Canal https://en.wikipedia.org/wiki/History_of_the_Panama_Canal

nickpsecurity 10y ago

Your Shell and Awk example is funny since simple, single-CPU tools are often faster than the Big Data tools the industry is standardizing on: http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html http://aadrake.com/command-line-tools-can-be-235x-faster-tha... http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/assets/COST.pdf It's true, though, that you should use a more complex tool if it's necessary to get the job done. It's just not clear that the complexity users or developers are dealing with are intrinsic to the problems being solved. It seems more social than anything. :)