4 ms·

> But those who are not comfortable with the foundational theory stuff just don't see it and don't even think to approach the problem in that way. Or maybe all

> But those who are not comfortable with the foundational theory stuff just don't see it and don't even think to approach the problem in that way. Or maybe all that "foundational theory" is oblivious to cache hierarchies(hello Big O notation) and generates worse performance on constrained devices. Or perhaps there's enough unknown, unknowns that throwing something at the wall is a legit way to get data if it's not a one-way decision. However the attitude on display above is a problem because you've just alienated those people who may have come from a different background or do know the theory but are approaching a problem in a different way.

josephg 6y ago
All approaches aren't equal. Yes, big-O notation hides constant factors like cache coherence but the solution isn't to analyse things less. Its to analyse things more. And thats what a proper CS education should teach. And at least once a year I draw on my CS education to: - Model something using state machine semantics - Use heap-based priority queues, binary search, b-trees or skip lists. And of course I use hash tables and hash sets weekly. - Read and implement something that CS researchers invented (PAXOS, interval tree clocks, CRDT work like RGA & YATA, etc). A lot of self taught programmers also don't seem to have fluency with all the degrees of freedom you have as a software engineer. Off the top of my head: Do you know where the bottlenecks are in this program? Are there better algorithms you could use? Are there different dataflow architectures which would help? Can you trade off CPU for memory with caching or memoization? What are the memory allocation patterns? What do you expect the upper bound on performance to be for this process? What are the fundamental invariants your program / data model should always maintain? Can we use a fuzzer to ensure those variants are always maintained? If those invariants aren't being maintained, would we know about it? What are the single points of failure? (And how could we add redundancy?) How would reliability and performance change if we use a different database, or added or removed indexes or caches? If you wanted to steal our user data, what are all the ways you could you do it? A junior engineer (or an engineer at a feature factory) might never need to ask these questions. But becoming a good senior requires a deeper expertise in seeing a program. And that requires an integrated knowledge of fundamentals, program analysis, tooling, experience and creativity. People learn all that without a degree, but personally? There's no way I would have learned all that stuff as well on my own.
- vvanders 6y ago
  > Yes, big-O notation hides constant factors like cache coherence but the solution isn't to analyse things less. Yet I run into good developers with good CS backgrounds all the time who were never exposed to cache hierarchies in their traditional CS education but had Big-O pounded into their head like it was the gospel on high. Sometimes it seems like revealing how cache misses or prefetchers work is some sort of magic trick that academia skips over. I've also seen where software engineers will stop at the boundaries of "CS" when it comes to problem solving(ex: not consider the hardware because it's a "hardware" problem rather than rolling up their sleeves and digging in). With the pace of development in software you need to continue to learn outside of a school setting and cultivate that curiosity that helps you come at problems from new and interesting ways.
  - josephg 6y ago
    > you need to continue to learn outside of a school setting and cultivate that curiosity This is true of just about everyone in computing. This isn't an industry for people who expect to cruise on the knowledge they were given a decade ago. (Plenty of people do, but thats another rant.) Regarding big-O, one habit / intuition I think its really important to develop is an intuition about how well something will actually perform in practice. Like, you should be able to guess within an order of magnitude or two: - How many simple reads per second and writes per second your database can perform - How many lookups / inserts per second to expect out of some standard data structures. (And how those numbers change as the collection grows) - How many allocations your program does, and how much time is spent in the allocator - About how large the steady state working memory size should be for your program I'm not talking about theoretical O(n) numbers. I mean, right now, if I write a tight loop in nodejs on my laptop writing random numbers to a JS Map(), how fast will it go? How does that compare to C/Rust? How many SET calls per second can a redis instance on my laptop handle? How about SELECT queries to postgres, or find({id:...}) calls to mongodb? Will the database be faster or slower than the nodejs program on my laptop thats issuing those queries? How many HTTP requests per second should I expect out of my express server? How many milliseconds should it take to statically render my web app to HTML? Etc. And arguably more importantly, it shouldn't take you more than ~20 minutes to go and find out the answer to any of those questions. Benchmarking is a delicate art, but if you can't measure, you programming blind. I wrote a fuzzer the other day which did about 100 iterations / second. And I know something was wrong because the number was orders of magnitude lower than I intuitively expected. Some tweaking later its now running about 1000 iterations/second - still too slow. Profiling shows its now spending 99% of the time in a single function. I'm hoping for another 10x when I rewrite that function. Without my intuition whispering in my ear, my program could have ended up 100x slower than it should be for no good reason. Aside: This might make a great interview question for a mid. "I have this simple piece of code. How fast do you think it will run on your laptop? Ok, share your screen and write a program to measure it. Talk to me about your answer!"
imtringued 6y ago
Big O is okay but for some reason when professors tell you that reasoning about universally valid performance characteristics of an algorithm is a lot of work if you are worrying about every tiny implementation detail the students ignore all of it and always disregard the constants even though the professor explicitly says that this is a method for estimating how fast the execution time of a function grows with increasing input. It's about building algorithms or datastructures and comparing them. Once you have a good baseline you can improve it further through engineering. The irony of course is that intuition and experience alone can get you 90% of the way but at a much lower time investment but theory can never replace experience because you have to turn that theory into experience first.

josephg 6y ago

All approaches aren't equal. Yes, big-O notation hides constant factors like cache coherence but the solution isn't to analyse things less. Its to analyse things more. And thats what a proper CS education should teach. And at least once a year I draw on my CS education to: - Model something using state machine semantics - Use heap-based priority queues, binary search, b-trees or skip lists. And of course I use hash tables and hash sets weekly. - Read and implement something that CS researchers invented (PAXOS, interval tree clocks, CRDT work like RGA & YATA, etc). A lot of self taught programmers also don't seem to have fluency with all the degrees of freedom you have as a software engineer. Off the top of my head: Do you know where the bottlenecks are in this program? Are there better algorithms you could use? Are there different dataflow architectures which would help? Can you trade off CPU for memory with caching or memoization? What are the memory allocation patterns? What do you expect the upper bound on performance to be for this process? What are the fundamental invariants your program / data model should always maintain? Can we use a fuzzer to ensure those variants are always maintained? If those invariants aren't being maintained, would we know about it? What are the single points of failure? (And how could we add redundancy?) How would reliability and performance change if we use a different database, or added or removed indexes or caches? If you wanted to steal our user data, what are all the ways you could you do it? A junior engineer (or an engineer at a feature factory) might never need to ask these questions. But becoming a good senior requires a deeper expertise in seeing a program. And that requires an integrated knowledge of fundamentals, program analysis, tooling, experience and creativity. People learn all that without a degree, but personally? There's no way I would have learned all that stuff as well on my own.

vvanders 6y ago
> Yes, big-O notation hides constant factors like cache coherence but the solution isn't to analyse things less. Yet I run into good developers with good CS backgrounds all the time who were never exposed to cache hierarchies in their traditional CS education but had Big-O pounded into their head like it was the gospel on high. Sometimes it seems like revealing how cache misses or prefetchers work is some sort of magic trick that academia skips over. I've also seen where software engineers will stop at the boundaries of "CS" when it comes to problem solving(ex: not consider the hardware because it's a "hardware" problem rather than rolling up their sleeves and digging in). With the pace of development in software you need to continue to learn outside of a school setting and cultivate that curiosity that helps you come at problems from new and interesting ways.
- josephg 6y ago
  > you need to continue to learn outside of a school setting and cultivate that curiosity This is true of just about everyone in computing. This isn't an industry for people who expect to cruise on the knowledge they were given a decade ago. (Plenty of people do, but thats another rant.) Regarding big-O, one habit / intuition I think its really important to develop is an intuition about how well something will actually perform in practice. Like, you should be able to guess within an order of magnitude or two: - How many simple reads per second and writes per second your database can perform - How many lookups / inserts per second to expect out of some standard data structures. (And how those numbers change as the collection grows) - How many allocations your program does, and how much time is spent in the allocator - About how large the steady state working memory size should be for your program I'm not talking about theoretical O(n) numbers. I mean, right now, if I write a tight loop in nodejs on my laptop writing random numbers to a JS Map(), how fast will it go? How does that compare to C/Rust? How many SET calls per second can a redis instance on my laptop handle? How about SELECT queries to postgres, or find({id:...}) calls to mongodb? Will the database be faster or slower than the nodejs program on my laptop thats issuing those queries? How many HTTP requests per second should I expect out of my express server? How many milliseconds should it take to statically render my web app to HTML? Etc. And arguably more importantly, it shouldn't take you more than ~20 minutes to go and find out the answer to any of those questions. Benchmarking is a delicate art, but if you can't measure, you programming blind. I wrote a fuzzer the other day which did about 100 iterations / second. And I know something was wrong because the number was orders of magnitude lower than I intuitively expected. Some tweaking later its now running about 1000 iterations/second - still too slow. Profiling shows its now spending 99% of the time in a single function. I'm hoping for another 10x when I rewrite that function. Without my intuition whispering in my ear, my program could have ended up 100x slower than it should be for no good reason. Aside: This might make a great interview question for a mid. "I have this simple piece of code. How fast do you think it will run on your laptop? Ok, share your screen and write a program to measure it. Talk to me about your answer!"

imtringued 6y ago

Big O is okay but for some reason when professors tell you that reasoning about universally valid performance characteristics of an algorithm is a lot of work if you are worrying about every tiny implementation detail the students ignore all of it and always disregard the constants even though the professor explicitly says that this is a method for estimating how fast the execution time of a function grows with increasing input. It's about building algorithms or datastructures and comparing them. Once you have a good baseline you can improve it further through engineering. The irony of course is that intuition and experience alone can get you 90% of the way but at a much lower time investment but theory can never replace experience because you have to turn that theory into experience first.