5 ms·

Would love to hear some examples of experts decreasing complexity and advanced engineers increasing complexity.

nilkn 7y ago
A hypothetical example would be one person spending a week or two setting up an extensive in-house Spark cluster to solve a problem that a second person in one day realizes can be solved on one machine using some clever shell scripting tricks. The former person knows a lot and may have been fully capable of the solution the second came up with, but they followed the wrong guiding principles in analyzing the problem and arrived at a solution of considerable complexity. An alternate formulation of this would involve the second person above arriving at a new job, observing that they're using Spark on an expensive cluster to regularly perform a computation, and noticing that actually they could do the whole calculation on a single node using simpler tools.
- saberience 7y ago
  You could answer that example problem just by being someone who reads Hackernews a bunch, since the "replace spark cluster with random unix tools" article appears here regularly. IMO, that doesn't really define being a principal engineer, it's basic.
  - nilkn 7y ago
    I don't even agree that this is a defining quality of a principal engineer. This is part of our standards for a senior developer, and it's something we coach people on from the get-go.
  - foobarian 7y ago
    Principal engineer: arrives at company, notices expensive Spark cluster, replaces with one server running shell scripts Double digit-strong data science team: speechless
- oarabbus_ 7y ago
  Thanks for the example. Once I did some analytics consulting with a stealth startup with some very talented engineers. One of them was usingusing rsync to replicate their data from their eng server to the analytics server, and I (not a software engineer, but familiar with shell) was initially like "that's it? You're just using rsync instead of <complex, paid replication tool>?" But it was the simplest and most straightforward tool for the job.
bsder 7y ago
Designing a delta-sigma modulator. Team engineers: we need a high-speed design with advanced adders and someone who can do the difficult static timing analysis. Expert: No, you need someone who actually understands VLSI design who can interleave time and space in an algorithm. At that point, you can basically use a single step up from stupid-simple ripple carry adders. And if you use non-overlapped clock generators you don't even have to do static timing analysis. The biggest wins are in architecture, not implementation. Yeah, 33MHz design in 125nm means VLSI like it's 1999.
inerte 7y ago
There's Juicero https://blog.bolt.io/juicero/ https://blog.bolt.io/juicero/
- oarabbus_ 7y ago
  I don't see the relevance?
  - inerte 7y ago
    The article I linked is about unnecessary increase in complexity with tiny gains. An expert engineer would be able to tell Juicero's machine could be simpler, and cheaper and faster to produce.

A hypothetical example would be one person spending a week or two setting up an extensive in-house Spark cluster to solve a problem that a second person in one day realizes can be solved on one machine using some clever shell scripting tricks. The former person knows a lot and may have been fully capable of the solution the second came up with, but they followed the wrong guiding principles in analyzing the problem and arrived at a solution of considerable complexity. An alternate formulation of this would involve the second person above arriving at a new job, observing that they're using Spark on an expensive cluster to regularly perform a computation, and noticing that actually they could do the whole calculation on a single node using simpler tools.

saberience 7y ago
You could answer that example problem just by being someone who reads Hackernews a bunch, since the "replace spark cluster with random unix tools" article appears here regularly. IMO, that doesn't really define being a principal engineer, it's basic.
- nilkn 7y ago
  I don't even agree that this is a defining quality of a principal engineer. This is part of our standards for a senior developer, and it's something we coach people on from the get-go.
- foobarian 7y ago
  Principal engineer: arrives at company, notices expensive Spark cluster, replaces with one server running shell scripts Double digit-strong data science team: speechless
oarabbus_ 7y ago
Thanks for the example. Once I did some analytics consulting with a stealth startup with some very talented engineers. One of them was usingusing rsync to replicate their data from their eng server to the analytics server, and I (not a software engineer, but familiar with shell) was initially like "that's it? You're just using rsync instead of <complex, paid replication tool>?" But it was the simplest and most straightforward tool for the job.

bsder 7y ago

Designing a delta-sigma modulator. Team engineers: we need a high-speed design with advanced adders and someone who can do the difficult static timing analysis. Expert: No, you need someone who actually understands VLSI design who can interleave time and space in an algorithm. At that point, you can basically use a single step up from stupid-simple ripple carry adders. And if you use non-overlapped clock generators you don't even have to do static timing analysis. The biggest wins are in architecture, not implementation. Yeah, 33MHz design in 125nm means VLSI like it's 1999.

inerte 7y ago

There's Juicero https://blog.bolt.io/juicero/ https://blog.bolt.io/juicero/

oarabbus_ 7y ago
I don't see the relevance?
- inerte 7y ago
  The article I linked is about unnecessary increase in complexity with tiny gains. An expert engineer would be able to tell Juicero's machine could be simpler, and cheaper and faster to produce.