5 ms·
I'm having difficulty reconciling these two statements, His best shot at it took somewhere between 11 to 12 hours to compile the results...or crash sometime du
by scott_s 16y ago
I'm having difficulty reconciling these two statements,
His best shot at it took somewhere between 11 to 12 hours to compile the results...or crash sometime during the night.
the developer, in keeping things simple for himself was storing all of the data for aggregation in associative arrays
With this one,
I say fantastic because he managed to write a rock solid, very scalable e-commerce back-end for a very large and very active site. It was seriously impressive stuff.
My reason is that I have difficulty calling someone a "fantastic developer" if they don't have a basic understanding of algorithms, data structures and systems. (If something takes a long time or crashes, run-away memory usage should be high on the list of suspects if you have basic systems knowledge.)
- Nate75Sanders 16y agoThe problem he had other than the reporting may have already been ridiculously parallelizable. If he wrote stable code and sharded user accounts or something, it might have worked with no real difficulties.
- anamax 16y ago> My reason is that I have difficulty calling someone a "fantastic developer" if they don't have a basic understanding of algorithms, data structures and systems A lot of development is plumbing.
- scott_s 16y agoI think you mean plumbing in the sense of connecting already existed components. I'll grant that. But would you consider someone who can only do that a fantastic developer? And, yes, I know we're discussing almost pure semantics.
- anamax 16y ago> But would you consider someone who can only do that a fantastic developer? In some circumstances yes, in others, no. We don't expect medical doctors to be good at everything, we don't expect jet engine mechanics to understand diesels, and so on, so why would we judge software folk based on some universal standard? Let's look at it from the other side. If someone isn't very good at plumbing (or floating point, or crypto, etc), does mean that said someone is not a fantastic developer?
- scott_s 16y agoBut we do judge doctor's based on a universal standard. They go through medical school, which contains subjects we expect them to at least be familiar with. Over time, formal learning in courses fades, but there is still a core of medical knowledge that we expect any practicing doctor to have. I'm suggesting that the basics of algorithms and data structures is that core for developers.
- bane 16y agoYeah, I can understand why that might seem off. I'm using "fantastic" to mean "built really rock solid stuff, from scratch, on his own without any formal education". The e-commerce site, being a bunch of smallish transactions was much easier and more obvious to scale than a huge numbers crunch on a single system (this all predates the common usage of map-reduce algorithms which probably would have been an even better fit in the end). Plus there's tons of great tech out there that helps with scaling a site (load balancers, sharding databases, etc.) that he didn't have to come up with, but not a lot for building custom reporting tools to spec for business types. Most of the requirements for the reporting tool weren't too bad -- "how many of this sku did we sell in the last 24 hrs, 7 days, 30 days, year, forever?" and could be solved pretty quickly and trivially. But scale something like that up, across all skus, correlated to individual users, or user locations, or supplier/warehouse availability, or whatever of another 100 variables and things get complicated very very fast, then aggregate those results and provide meaningful output to business types, and we're starting to talk Computer Science vs. Development. The story wasn't mean to pump my wife up over her boss, at the end of the day, she solved one problem using her education, but she probably wouldn't be able to sit down and build what he built elsewhere. There's a tremendous amount of practical engineering knowledge that she (and I) just don't have (and don't learn in school) that her boss did have -- mostly things you just have to learn that kind of thing on the job. I guess it really comes down to the semantics of developer. He's a great software engineer type, he can plug components together to build bewilderingly complex systems, but at the time he had a very naive view of algorithm design...which is a very different discipline.
- scott_s 16y agoIt is a semantics issue, but just like I expect mechanical engineers to have a strong grasp of Newtonian mechanics, I expect software engineers to have a strong grasp of algorithms and data structures.
- bane 16y agoI suspect at some point the development community is going to have to come up with better names to call people. A person who is fantastic with web technologies may not be great with real-time audio processing tools or graphics processing tools or Natural Language systems. But a person who is a genius at with NLP may not be able to cobble together a website or write an algorithm for fast approximations of an inverse square function. A developer who understands how to squeeze every last cycle out of an embedded system will likely be lost in a massive enterprise environment and a person who specializes in machine learning may stink at writing efficient database and storage systems. A developer great at crypto systems might be laughed at if he wrote a video game. Someone who specializes in low level cache optimization may have absolutely no clue how to write an e-commerce system. And a lifelong mainframe developer would be lost writing device drivers. I suspect that most people aren't great at everything, even top-notch developers. More likely people are really great at one thing, and pretty good at a couple others and lousy at the rest. There's a few exceptions I'm sure, but if your definition of "great developer" only includes those people who are great at every bit of software development trivia then we're probably talking about only a handful of people on the planet. Algorithm optimization is hard and often not-obvious (how many people really understood the algorithm for the fast inverse square root? I bet thousands of developers used it, even plenty of great ones, but groking it is so rare that a series of articles was actually written to explain it) There are plenty of perfectly competent and solid developers (even great) who can hack together great stuff in their domain, but struggle with even the basics just outside of that. An autodidact will probably be pragmatic with what they learn, and most of that will probably be plumbing work since that's what's strictly required to get something up and working. I see looking at your page that you specialize in high-performance computing and parallel systems. Are only developers who understand those topics great developers? Because most developers don't do well with those topics. Even the dev shop at Valve has struggled with parallelizing their software. Saying "I expect mechanical engineers to have a strong grasp of Newtonian mechanics", do you also expect them to have a strong grasp of non-Newtonian mechanics? Since Newton's equations are simply a special case of relativity, why shouldn't they know how to calculate the load stresses on a mechanism using Einstein's much more precise work? Shouldn't they really have a fantastic understanding of particle physics and quantum mechanics too? I mean, the binding forces that hold together the materials they work with every day are subject to that basic knowledge! Alas! Now I'm concerned that the plucky civil engineer, when designing the overpass I take to get home from work today, didn't quite get the section on QCD Color Charge because he mistook the professor's sloppy handwriting of lambda for an 'A'! Perhaps my downtrodden electrical engineer is hiding a secret shame, he's never understood how to accurately calculate ballistic trajectories over the surface of the Earth! Oh sure it looked so simple in class, but they never went over how to actually do it, taking into account variable atmospheric resistance, and a suitable model of the Earth's actual shape, and GRAVITY! Oh gravity...why do you have to subtly change as the mistargetted missile drifts lazily across your forlorn belly! It's just dif-EQ after all! Why then, did his water ballon slingshot fail to hit his neighbors house precisely where he wanted it? I suddenly feel a great pity for these noble engineers, overcoming such obstacles, such gaps in their basic knowledge! Yet still my overpass stands, and still my car works, and still my lights turn on. Only because of this spark of competence shall their secret, their hidden failings, remain safe with me.