10 ms·

Algorithms Are Now Commodities

155 points by EvgeniyZh 6y ago

OliverJones 6y ago
This old dev has been around to witness the transition from in-depth algorithm development to algorithms as commodities. I've pounded out my share of B-tree search, polygon intersection, and numeric stuff. Try doing numeric stuff reliably and quickly on a tiny computer without a Floating Point Unit and you'll know why you might need to learn, and then program, Jack Bresenham's line-drawing and circle-drawing algorithms. And there's Newton and Raphson's sqrt(). Hint: if you understand your data and your precision requirement, you can avoid iterating it all the way out to the end of your integer precision, and get a lot more work done. Try transposing a 4-megabyte matrix before the week is out on a machine with 32KiB of RAM and two 5-megabyte hard drives. Yes, it can be done. When I started doing this work it took deep understanding to get anything done. Now it takes the skills of a good reference librarian. We have to be able to find stuff (via npm, nuget, maven) quickly, assess its quality, and figure out how to use it. So the type of thinking we need has changed from "deep" to "broad".
- C1sc0cat 6y ago
  This was true to an extent in the 70's - I recall us using subroutines from NAG htps://www.nag.co.uk/ Not much point in implementing our own Fourier code.
  - crawlcrawler 6y ago
    Not much chance of us seeing an improvements in Fourier transform implementations, then.
    - C1sc0cat 6y ago
      Depends is it better for NAG to do that research or for us to do it? Our funding was to do xxx research and not go off and build a better Fourier transform in this case. Though I do regret that we couldn't get funding to do Vision /ML on one project in the 80's truly cutting edge stuff
    - derefr 6y ago
      And yet we still do, and it's interesting why we do. Most software, we optimize to make it more legible to the compiler. But some software—software we don't trust ourselves to optimize without changing its meaning—we leave alone, and then instead we optimize the compiler specifically to compile that kind of software into better object code. This is what happens with maths code (usually in FORTRAN) and the FORTRAN compilers that work on them. We don't improve the Fourier-transform implementation; we just teach the compiler how to take the "textbook" implementation we've got, and do ever-fancier things to it. Of course, one could also improve the Fourier transform's design—but that requires a mathematician, not a software engineer. Nobody's going to come up with an algorithm with entirely-better time/space complexities just by noodling around in an editor with the existing one. They've got to start from scratch with new maths, derive new theorems, write a textbook implementation of that, and then contribute it to the codebase. And then we'll start a new cycle of getting the compiler to optimize that textbook code.
      - hef19898 6y ago
        Back in 2008 (?), I was in aerospace. The big thing at that time was to become a "systems integrator" instead of a manufacturer. Obviously, while sound on the surface, the OEMs still are manufacturers today. With a higher degree of integration of third party systems, e.g. radar an so on. Today, the same logic can be applied to all the Xaas business models out there. SaaS, PaaS, algorythms, it seems that there could be two extremes. On the one end you have companies developing the services, on the other you have businesses excelling at combining these services.
- mlthoughts2018 6y ago
  I disagree somewhat. For example in my freshman year of college I took a numerical analysis class that covered Newton-Raphson, Brent’s method, advanced quadrature, Runge-Kutta, preconditioning, all sorts of matrix linsolve, LU, rotation matrix, inversion stuff, floating point byte structure, and various topics in parallel computing like Amdahl’s law. It was intense and very detailed, with a lot of discussion of how these issues arise in and how to write these algorithms in the C language and also in Python (like, in C extension modules of CPython). My first job out of college was in a defense research lab that had projects running on tons of simulation systems, radar systems, hardware and embedded systems for vehicles used in the military. What I learned later when I went back to grad school and worked more mainstream tech product jobs is that in those “hardcore numerical computing” situations, there is often a complete absence of architecture or design, no framing the problem in terms of the customer or success criteria, too much willingness to reinvent wheels, constant belief that build it is always better than buy it. From a technical point of view, there was also a gaping absence of statistics, especially statistical optimization techniques like with Markov chains and simulated annealing. There was a certain chip on people’s shoulder that hard core graph algorithms or hard core data structure based search was “real programming” when you could solve the same problem with simulated annealing in 1/100 the amount of code, as long as you didn’t parochially think your numerics experience was superior and force a solution that way. Even though my peers were really experienced, it felt like there was an inverse relationship between numerical algorithm thinking and creative problem solving thinking. Hard core algorithms just sucked all the air out of the room, without making projects more successful.
- bobmaxup 6y ago
  I think the enterprise to open source software relationship today is somewhat similar to the concept of convergence in economics. https://en.wikipedia.org/wiki/Convergence_(economics) https://en.wikipedia.org/wiki/Convergence_(economics)
- game_the0ry 6y ago
  As a young dev, I would be very interested in your opinions and experiences. Specifically, in book or blog format, maybe tweet format :D
  - hoorayimhelping 6y ago
    I know you meant this harmlessly and maybe even encouragingly, but this post comes off as pretty entitled. It comes off kind of like, "hey, I want access to your knowledge and wisdom, but I want you to do an enormous amount of work to put it into a format that is easy for me to consume at my leisure. plzkthxbye." Usually when you want to learn from someone more experienced than you, you seek them out, and you do the work to understand their wisdom, not the other way around. Plato didn't ask Socrates to write everything down so he could read it and grow, Plato studied under Socrates and did the work of transcribing his words himself for the future.
    - vharuck 6y ago
      I read it as, "Do you have a blog? If not, have you thought about blogging?" Nothing demanding. Especially with the happy emoji at the end.
      - game_the0ry 6y ago
        Basically, this. :)
    - game_the0ry 6y ago
      > It comes off kind of like, "hey, I want access to your knowledge and wisdom, but I want you to do an enormous amount of work to put it into a format that is easy for me to consume at my leisure. plzkthxbye." Kind of. When guys like OJ write books, I buy them. When they blog / tweet, I re-tweet them. Mostly to share knowledge with my industry peers and other newbs, but also to support directly or indirectly. I don't see this as one-sided and I don't see anything wrong with expecting something in returning for sharing knowledge. I will tell you what disappoints me - when good knowledge is lost over time. Or worse, hoarded and lost anyway. And I will tell you what else is disappoints me - people who assume young newbs have an entitlement complex when seeking out knowledge, let alone encouraging others to share knowledge. You strike me as a 'what's in it for me?' type of person. That's ok, I might buy your blog or read your book if you have them.
- GuiA 6y ago
  It all depends what kind of work you do. I do things that touch embedded hardware and graphics and frequently find myself doing exactly the sort of things you described. If you miss it, it's certainly not gone :) (and I made a conscious decision to build my career around this, rather than fishing for npm modules).
- chrisseaton 6y ago
  > Try transposing a 4-megabyte matrix before the week is out on a machine with 32KiB of RAM and two 5-megabyte hard drives. Transposing is just copying bytes from one place to another, isn't it? There's no maths involved, is there? To have that run in a week with the most naive approach possible you'd only need a machine that can copy seven words a second. Are there real relevant machines that can't do that? What algorithms could help you anyway? Something to do with cache and the order in which you read and write? And you've got plenty of space as you can trivially do it in-place.
  - smallnamespace 6y ago
    The RAM here is essentially a cache. You’ll want to transpose a block at a time, something like 150 x 150 blocks, so it all fits. Each block is not contiguous on disk, so you’ll want to load blocks in the right order to avoid extra seeks.
    - deleted 6y ago
      [deleted]
    - deleted 6y ago
      [deleted]
    - chrisseaton 6y ago
      I suspect, since the poster mentioned having two hard drives, that really they're confusing transposition with inversion. These aren't the same things.
  - dekhn 6y ago
    Transposing is copying bytes, but changing the stride order of the array. So a lot of the work is extremely non-local. The poster meant that to do this efficiently, you'll use blocks and stage data between the drives while working. You can also transpose in place: https://en.wikipedia.org/wiki/In-place_matrix_transposition https://en.wikipedia.org/wiki/In-place_matrix_transposition see some of the commentary on efficiency there.
- EGreg 6y ago
  Well, there’s the coming up with an algorithm Then there is the teaching it in schools to new comp sci people, as part of a curriculum designed to help you understand basic principles Then there are implementations on various environments, constant optimizations, witness the JS engine wars for instance People specialize in their own industryon various niches — primitives, or combining primitives, etc. And algorithms are just one part of an industry! How many of you want to invent your own SHA1 or BLS signatures LOL You just use them and they have documented properties and guarantees. These are building blocks people use. This thing about glorifying algorithms is sort of an “applied maths” mentality. And models like Black-Scholes therefore become taught everywhere and become self-fulfilling prophecies. But really, math is just a very simplified version of computing and modeling. People operate with maybe 10-20 symbols throughout the entire thing. There are many things — like the game of life and New Kind of Science - that you can’t describe in analytic form.
enriquto 6y ago
> I date the age of the Algorithm from roughly the 1960s to the late 1980s. During the age of the Algorithms, developers spent a lot of time figuring out the best algorithm to use and writing code to implement algorithms. Today there's orders of magnitude more developers spending time figuring out algorithms and writing core to implement them than in the 80s. The difference, maybe, is that even more developers today do not write any algorithm at all. But if there's an age of the algorithm, it is today. At no point in human history have as many algorithms been written as today. And next week, many more will be written!
- dtech 6y ago
  You're missing the point. There is a larger absolute number of people working on complex algorithms, but as a percentage it is much smaller. In the 60's deep algorithmic knowledge was required for 90%+ of programming tasks, now it is probably closer to 1%. It is perfectly possible to make a living as a programmer nowadays without being able to inverse a binary tree
  - oriolid 6y ago
    What's the obsession with inverting a binary tree? I have never found an use case that particular algorithm, but on the other hand I would be very happy if more programmers were aware that having a where clause in sql statement (and maybe and index) is more efficient than selecting entire database and looping over the results, even if the end result is the same. I understand that in order to come up with the inversion on the fly you need to know about trees and recursion, but is it really that scary?
    - arethuza 6y ago
      Possibly due to the tweet by Max Howell: "Google: 90% of our engineers use the software you wrote (Homebrew), but you can’t invert a binary tree on a whiteboard so fuck off."
      - oriolid 6y ago
oriolid 6y ago
I have the feeling that the article is really about optimization, but the author does not know the difference.
fogetti 6y ago
And that's why HackerRank, Codility and all the other coding sites have zero relevance today in regards of the actual capability of the candidate. The fact that hiring committees, HR departments and software engineering managers rely on them goes to show how detached all those people are/have been from reality for a long time now. Good software engineering skills on the other hand are pretty much more needed than ever.
- collyw 6y ago
  I have needed to do one heavily algorithmic task. Once. In nearly 20 years of programming.
- cyberdrunk 6y ago
  The thing is, in large tech companies like Google, you might be tasked with designing (or modyfing a design) of something that works on Internet scale. For such cases, thinking in terms of algorithms is essential. That's why I think FAANGs still require algorithmic literacy. Unfortunately, other companies, which don't have FAANG-like problems, ape its recruitment process...
  - fogetti 6y ago
    Yes, algorithms are essential for information technology. But the question is not that. And the discussion is also not about that. The question is: are algorithms essential for one's career and for the majority of business too? I don't think so.
    - strikelaserclaw 6y ago
      By definition, any code written to solve a problem is an algorithm (not just the standard stuff found in textbooks), since a lot of people in IT write code to solve problems, then yes they absolutely need to understand some what performance characteristics and the correctness of what they wrote. Even if you don't need to performance characteristics, you certainly need the correctness part!
- signa11 6y ago
nickcw 6y ago
I spent a lot of time reading Knuth's books, especially Seminumerical Algorithms when I was implementing multi precision arithmetic for various architectures. What amazing pieces of work they are. That was back in the time when you could look at assembler (or even C code) and get a good idea of how many cycles it would take. I love the Art of the Algorithm, but I have to say I'm very happy to use off the shelf libraries now. I've implemented binary search dozens of times and each time spent ages fixing all the corner cases. Instructive, but a waste of time when you can import a library which is battle tested and faster than anything you'll ever write. Algorithms are dead - long live the (commoditized) Algorithm!
- chrchang523 6y ago
  Incidentally, I think there is at least one important area where Knuth is actually still ahead of off-the-shelf libraries as of 2020: "broadword computing". When skimming volume 4A for semi-recreational purposes a few years ago, I was surprised to find multiple "bit-hacks" which were better than anything I had found in other sources, and proceeded to use them in my own open-sourced work. Every time I've encountered a situation where bitvectors are useful, I have benefited from rolling my own implementation.
  - klyrs 6y ago
    Came here to say this. It's rare that you want to consume these things through an abstraction layer: the layer is often heavier than the work underneath. Sucks for readability, though, so I usually spend as much time documenting as implementing.
jasode 6y ago
>This was the age of the algorithm. [...] When it comes to algorithm implementation, developers are now spoilt for choice; why waste time implementing the ‘low’ level stuff when there were plenty of other problems waiting to be implemented. Algorithms are now like the bolts in a bridge: very important, but nobody talks about them. Today developers talk about story points, features, business logic, etc. [...] Today, we are in the age of the ecosystem. The blog author doesn't make it explicit but his usage of "algorithm" is actually about "low-level algorithms" which is how it seems to support his claim that "algorithms are now commodities". But "algorithms" in the general sense are not commodities. It's just that we're now free to concentrate on higher-level problems with new algorithms. E.g., in the 1980s & 1990s, the Barnes & Noble section for computers would have books with algorithms showing how to write disk-based B-trees from scratch. But now with SQLite library in wide usage, you can just use their B-trees instead of reinventing your own. Some more examples of low-level algorithms I used to write by hand: - manually uppercase chars by adding "32" to the ASCII code because there was no Upper() function in a standard library. - manually titlecase/propercase/lowercase a person's name from "JOHN DOE" to "John Doe" by looping the the string char-by-char and subtracting "32" from the ASCII code if it doesn't follows a space char because there was no TitleCase() function - adding a drop shadow beneath a window because the operating system GUI didn't render shadows for you - manually writing link lists to create dynamic size memory buffers because there was no C++ STL library yet or built-in associative arrays in Python/C# I don't do have to do any of that low-level tedious work anymore but I still write new algorithms because there is always some [missing functionality] where a standard library doesn't exist or a consensus implementation hasn't yet "won" in the marketplace of ideas. It could be distributed cloud/desk/mobile data sync algorithm, or collaborative recommendation algorithm, etc. Algorithms (general sense) will not be commoditized for decades -- if ever.
- raphlinus 6y ago
  I have a feeling you might enjoy this code: https://github.com/zbraniecki/tinystr/blob/master/src/tinystr8.rs#L221 https://github.com/zbraniecki/tinystr/blob/master/src/tinyst... I wrote this because I wanted (normalizing) locale comparisons to be super fast, as they can be part of the key for cache lookups of text layout. In a language identifier such as "en-Latn-US" the script is in title case. So this kind of thing still does happen, just at the lowest levels of libraries that provide nice clean abstractions for other people to use.
scoutt 6y ago
> There are companies where algorithms are not commodities And there is an entire industry! Embedded (but not the Raspberry-pi kind of embedded that is similar to desktop). Try to do a dead reckoning system, or anything related to real-time gesture/motion analysis. Try reading, processing, mixing and outputting audio in real time. Or any DSP related stuff. What about FPGA, ASICs, etc.? Why oh why is the embedded industry so easily forgotten? It's not all cloud, food apps, SaaS and Electron, ya know? Edit to add: all the above with the limited resources of embedded, that forces you to improve your algorithms.
- vanderZwan 6y ago
  > Why oh why is the embedded industry so easily forgotten? It's not all cloud, food apps, SaaS and Electron, ya know? Because the other industry is inherently linked to the internet. It's like the media telling you that the media is very important, or your brain telling you that it's the most interesting organ in your body (I forget which comedian I stole that from).
- pseudosudoer 6y ago
  Depends on the embedded ecosystem. STM has tons of free software packages for their products that remove the need to write drivers and a variety of algorithms from scratch. Albeit not the same, Xilinx and Altera have a plethora of FPGA IP that can be licensed/bought, although I can see the merit in a custom implementation here to save money.
jcahill 6y ago
Commodities are fully fungible goods and services. Different implementations of a single algorithm should have the same basic formal properties, by definition. That is, they should be roughly fungible. Else you aren't dealing with correct implementations of the same algorithm. If anything, the thesis "algorithms are now commodities" is less true now than ever before. As the complexity of software deliverables increases, the use of complex, proprietary algorithms in software should likely increase in turn.
nirui 6y ago
> why waste time implementing the ‘low’ level stuff when there were plenty of other problems waiting to be implemented. Because: - The package is licensed through GPL - The lib allocates memory in such way that it messed up my program - The last update to the lib is 3 years ago, and it takes at least another 3 years for the author of the package get out of jail which is the min punishment of some NSL - I don't like how it's interface is designed - Because I got paid even I write everything by myself The list goes on. Another way to look at this is, "Do I really want just to implement a Linked-list? Or an sizable & queueable list that allow me to sort items via LRU?". Sometime, you just have to to write some "low level stuff" by yourself to make it exactly as you wanted.
tanilama 6y ago
I am somewhat confused about the definition of Algorithms under the article's context. What is this Algorithm we are talking about? Is it about basic data structures? If that is what we talk about, I feel at least from like 15 years ago, people already giving up writing those things from scratch for new projects. I agree encapsulation is growing. As developers we are in a reality that we lose more and more predictability over the behavior of our code. But that seems to create more demand for nuanced solution to reign the complexity, since commoditized algorithm created this problem, not solving it.
jvanderbot 6y ago
Well that's just not true. I concur: Nowadays, the basic building blocks are done. Good sorting, database indexing, O(1) set-membership, many variants of lists, hash tables, and other higher level data structures are well covered by modern libraries. The foundations of modern system design are algorithms and data structures, and they are largely commoditized, for general use. Certainly, working on general purpose, clever-but-foundational algorithms is becomming increasingly niche. I disagree with the final conclusion that somehow algorithms have faded into obscurity. Anyone writing any program has to consistently track their expected program performance as it shuffles data between commodotized library calls or between network end points. This is algorithm design. Many "foundational" algorithms like max-flow actually make use of "foundational(er)" algorithms and shuffle data between them. This is what the modern programmer does when he decomposes a data-intensive problem into sub-problems and uses commoditized libraries and re-composes those solutions. The push to catalog and archive a wide array of well-implemented algorithms and data structures tracked the emergence of the personal and commodity hardware. The emergence of new hardware will require new solutions. Mobile, webassembly (perhaps), ASIC-based data centers, CUDA, are all exciting new areas where our "foundational" algorithms and canonical solutions need to be revisited, redesigned, and re-tuned. Or at least re-composed.
brushfoot 6y ago
All that's old is new again. The level of abstraction just keeps getting higher. Someday, ecosystems like AWS and Azure will be as standardized as programming languages. You'll still be writing algorithms, but instead of telling one PC what to do, you'll be telling a distributed system what to do. The old design patterns will still be relevant; the implementation will just look different. We're already there to some extent. It's just messy and fragmented right now.
omarhaneef 6y ago
There are probably more algorithms being "figured out" today than ever before. However, the vast majority are used during interviews, and don't see the dark of silicon.
sdenton4 6y ago
One of my favorite mini games at my job is rewriting classic algorithms to run in batched mode on gpu/tpu. The speed improvements often improve model training time by days, and it's always a lovely intellectual challenge. (The basic challenge is to rewrite the algorithms in terms of matrix operations which operate on many examples of the problem at once, while eliminating all branching.)
_bxg1 6y ago
I remember reading an article a while back about how professional photography has become a commodity. You can find and license any imaginable photograph, for pennies, in minutes. From the best photographers in the world. So from a business perspective, photography as a pure art has been devalued to almost nothing. Nobody pays you to take the very best possible photo. So why do people still hire photographers? To put themselves or their product into a photo. Most photographers are no longer paid to take the very best photos, but to make their photos specific to what the particular client cares about. That's something that can't be commoditized. I think that's where software is now as an industry. Very few of us these days get to innovate at a "pure art" level. Instead, we get paid to pull together a bespoke solution that's sensitive to a given business and its needs. It isn't novel at the macro-level, only at the micro-level. This can be kind of depressing, but it's also a little comforting that those micro-level solutions are so hard to fully commoditize. As long as the real world is messy and varied, the leaves in the software tree will have to be too, and there will be work that needs doing.
- ForHackernews 6y ago
  I don't see how that's depressing at all. Another way to phrase what you're saying is "Developers get paid to write software that solves problems people actually need solved.
  - _bxg1 6y ago
    For many of us, writing something pure and generic and beautiful is much more gratifying. And it's depressing to think that we'll probably never write something like that that's good enough to be picked over the version that was developed at Google and then open-sourced.
    - ForHackernews 6y ago
      I mean, nobody's stopping you from writing your perfect, beautiful Haskell implementation of Dijkstra's Algorithm. I guess it's simply a different approach, but I'd rather not waste time re-implementing the wheel when I could be making software that will help somebody fulfill a concrete need.
opportune 6y ago
Personally I found it a bit disappointing moving from academia to industry and finding out that there is not much of a market for algorithm implementation or development/research. Sure, there are specialized products where they are important (DB, OS, languages/libraries, trading systems) and even places for algorithms in business apps, but it doesn’t seem you can really “specialize” in it as a career (like distributed systems, system programming, frontend) unless you’re very picky about which jobs you take.
Const-me 6y ago
I don’t remember last time I implemented sorting, search or trees, these are available in all languages and runtimes. Skill, I implement a lot of my own algorithms. Following reasons. 1. Performance. Custom algorithms and data structures is often a requirement to write performant code, because SIMD, memory hierarchy, and other modern hardware-related shenanigans largely ignored by authors of standard libraries. 2. GPGPU. Due to their massively parallel nature they need completely different algorithms, a naïve implementation is often by an order of magnitude slower than purposely built one, due to things like memory access coalescing and group shared memory. 3. Licensing. In my area of work (CAD/CAM/CAE) some good quality libraries are only available as GPL or even AGPL. Apparently, the commercial licenses are targeted towards companies like GM or Airbus, way too expensive.
softwaredoug 6y ago
Yes commodities, but I still need an intuitive understanding of how the algorithms work to pick the right one. I might even need to read the code when I get deeper into solving my problem. Or implement a dummy version simply for learning. Algorithms are commodities, but algorithm skill is still deeply needed
softwaredoug 6y ago
I wonder if the number of people working on algorithms is about the same proportion of the workforce, it's just that there's way way more developers now, and they need commoditized algorithms that solve 90% of the problem (but that 10% is still out there...). For example, if you summed up all the 'core committers' of open source projects, people working on operating systems, embedded software, systems programming, and other specialized software developers that might think about unique algorithms, you wonder if that's about the same as the number of total software developers in the 80s

155 points by EvgeniyZh 6y ago

This old dev has been around to witness the transition from in-depth algorithm development to algorithms as commodities. I've pounded out my share of B-tree search, polygon intersection, and numeric stuff. Try doing numeric stuff reliably and quickly on a tiny computer without a Floating Point Unit and you'll know why you might need to learn, and then program, Jack Bresenham's line-drawing and circle-drawing algorithms. And there's Newton and Raphson's sqrt(). Hint: if you understand your data and your precision requirement, you can avoid iterating it all the way out to the end of your integer precision, and get a lot more work done. Try transposing a 4-megabyte matrix before the week is out on a machine with 32KiB of RAM and two 5-megabyte hard drives. Yes, it can be done. When I started doing this work it took deep understanding to get anything done. Now it takes the skills of a good reference librarian. We have to be able to find stuff (via npm, nuget, maven) quickly, assess its quality, and figure out how to use it. So the type of thinking we need has changed from "deep" to "broad".

C1sc0cat 6y ago
This was true to an extent in the 70's - I recall us using subroutines from NAG htps://www.nag.co.uk/ Not much point in implementing our own Fourier code.
- crawlcrawler 6y ago
  Not much chance of us seeing an improvements in Fourier transform implementations, then.
  - C1sc0cat 6y ago
    Depends is it better for NAG to do that research or for us to do it? Our funding was to do xxx research and not go off and build a better Fourier transform in this case. Though I do regret that we couldn't get funding to do Vision /ML on one project in the 80's truly cutting edge stuff
  - derefr 6y ago
    And yet we still do, and it's interesting why we do. Most software, we optimize to make it more legible to the compiler. But some software—software we don't trust ourselves to optimize without changing its meaning—we leave alone, and then instead we optimize the compiler specifically to compile that kind of software into better object code. This is what happens with maths code (usually in FORTRAN) and the FORTRAN compilers that work on them. We don't improve the Fourier-transform implementation; we just teach the compiler how to take the "textbook" implementation we've got, and do ever-fancier things to it. Of course, one could also improve the Fourier transform's design—but that requires a mathematician, not a software engineer. Nobody's going to come up with an algorithm with entirely-better time/space complexities just by noodling around in an editor with the existing one. They've got to start from scratch with new maths, derive new theorems, write a textbook implementation of that, and then contribute it to the codebase. And then we'll start a new cycle of getting the compiler to optimize that textbook code.
    - hef19898 6y ago
      Back in 2008 (?), I was in aerospace. The big thing at that time was to become a "systems integrator" instead of a manufacturer. Obviously, while sound on the surface, the OEMs still are manufacturers today. With a higher degree of integration of third party systems, e.g. radar an so on. Today, the same logic can be applied to all the Xaas business models out there. SaaS, PaaS, algorythms, it seems that there could be two extremes. On the one end you have companies developing the services, on the other you have businesses excelling at combining these services.
mlthoughts2018 6y ago
I disagree somewhat. For example in my freshman year of college I took a numerical analysis class that covered Newton-Raphson, Brent’s method, advanced quadrature, Runge-Kutta, preconditioning, all sorts of matrix linsolve, LU, rotation matrix, inversion stuff, floating point byte structure, and various topics in parallel computing like Amdahl’s law. It was intense and very detailed, with a lot of discussion of how these issues arise in and how to write these algorithms in the C language and also in Python (like, in C extension modules of CPython). My first job out of college was in a defense research lab that had projects running on tons of simulation systems, radar systems, hardware and embedded systems for vehicles used in the military. What I learned later when I went back to grad school and worked more mainstream tech product jobs is that in those “hardcore numerical computing” situations, there is often a complete absence of architecture or design, no framing the problem in terms of the customer or success criteria, too much willingness to reinvent wheels, constant belief that build it is always better than buy it. From a technical point of view, there was also a gaping absence of statistics, especially statistical optimization techniques like with Markov chains and simulated annealing. There was a certain chip on people’s shoulder that hard core graph algorithms or hard core data structure based search was “real programming” when you could solve the same problem with simulated annealing in 1/100 the amount of code, as long as you didn’t parochially think your numerics experience was superior and force a solution that way. Even though my peers were really experienced, it felt like there was an inverse relationship between numerical algorithm thinking and creative problem solving thinking. Hard core algorithms just sucked all the air out of the room, without making projects more successful.
bobmaxup 6y ago
I think the enterprise to open source software relationship today is somewhat similar to the concept of convergence in economics. https://en.wikipedia.org/wiki/Convergence_(economics) https://en.wikipedia.org/wiki/Convergence_(economics)
game_the0ry 6y ago
As a young dev, I would be very interested in your opinions and experiences. Specifically, in book or blog format, maybe tweet format :D
- hoorayimhelping 6y ago
  I know you meant this harmlessly and maybe even encouragingly, but this post comes off as pretty entitled. It comes off kind of like, "hey, I want access to your knowledge and wisdom, but I want you to do an enormous amount of work to put it into a format that is easy for me to consume at my leisure. plzkthxbye." Usually when you want to learn from someone more experienced than you, you seek them out, and you do the work to understand their wisdom, not the other way around. Plato didn't ask Socrates to write everything down so he could read it and grow, Plato studied under Socrates and did the work of transcribing his words himself for the future.
  - vharuck 6y ago
    I read it as, "Do you have a blog? If not, have you thought about blogging?" Nothing demanding. Especially with the happy emoji at the end.
    - game_the0ry 6y ago
      Basically, this. :)
  - game_the0ry 6y ago
    > It comes off kind of like, "hey, I want access to your knowledge and wisdom, but I want you to do an enormous amount of work to put it into a format that is easy for me to consume at my leisure. plzkthxbye." Kind of. When guys like OJ write books, I buy them. When they blog / tweet, I re-tweet them. Mostly to share knowledge with my industry peers and other newbs, but also to support directly or indirectly. I don't see this as one-sided and I don't see anything wrong with expecting something in returning for sharing knowledge. I will tell you what disappoints me - when good knowledge is lost over time. Or worse, hoarded and lost anyway. And I will tell you what else is disappoints me - people who assume young newbs have an entitlement complex when seeking out knowledge, let alone encouraging others to share knowledge. You strike me as a 'what's in it for me?' type of person. That's ok, I might buy your blog or read your book if you have them.
GuiA 6y ago
It all depends what kind of work you do. I do things that touch embedded hardware and graphics and frequently find myself doing exactly the sort of things you described. If you miss it, it's certainly not gone :) (and I made a conscious decision to build my career around this, rather than fishing for npm modules).
chrisseaton 6y ago
> Try transposing a 4-megabyte matrix before the week is out on a machine with 32KiB of RAM and two 5-megabyte hard drives. Transposing is just copying bytes from one place to another, isn't it? There's no maths involved, is there? To have that run in a week with the most naive approach possible you'd only need a machine that can copy seven words a second. Are there real relevant machines that can't do that? What algorithms could help you anyway? Something to do with cache and the order in which you read and write? And you've got plenty of space as you can trivially do it in-place.
- smallnamespace 6y ago
  The RAM here is essentially a cache. You’ll want to transpose a block at a time, something like 150 x 150 blocks, so it all fits. Each block is not contiguous on disk, so you’ll want to load blocks in the right order to avoid extra seeks.
  - deleted 6y ago
    [deleted]
  - deleted 6y ago
    [deleted]
  - chrisseaton 6y ago
    I suspect, since the poster mentioned having two hard drives, that really they're confusing transposition with inversion. These aren't the same things.
- dekhn 6y ago
  Transposing is copying bytes, but changing the stride order of the array. So a lot of the work is extremely non-local. The poster meant that to do this efficiently, you'll use blocks and stage data between the drives while working. You can also transpose in place: https://en.wikipedia.org/wiki/In-place_matrix_transposition https://en.wikipedia.org/wiki/In-place_matrix_transposition see some of the commentary on efficiency there.
EGreg 6y ago
Well, there’s the coming up with an algorithm Then there is the teaching it in schools to new comp sci people, as part of a curriculum designed to help you understand basic principles Then there are implementations on various environments, constant optimizations, witness the JS engine wars for instance People specialize in their own industryon various niches — primitives, or combining primitives, etc. And algorithms are just one part of an industry! How many of you want to invent your own SHA1 or BLS signatures LOL You just use them and they have documented properties and guarantees. These are building blocks people use. This thing about glorifying algorithms is sort of an “applied maths” mentality. And models like Black-Scholes therefore become taught everywhere and become self-fulfilling prophecies. But really, math is just a very simplified version of computing and modeling. People operate with maybe 10-20 symbols throughout the entire thing. There are many things — like the game of life and New Kind of Science - that you can’t describe in analytic form.

enriquto 6y ago

> I date the age of the Algorithm from roughly the 1960s to the late 1980s. During the age of the Algorithms, developers spent a lot of time figuring out the best algorithm to use and writing code to implement algorithms. Today there's orders of magnitude more developers spending time figuring out algorithms and writing core to implement them than in the 80s. The difference, maybe, is that even more developers today do not write any algorithm at all. But if there's an age of the algorithm, it is today. At no point in human history have as many algorithms been written as today. And next week, many more will be written!

dtech 6y ago
You're missing the point. There is a larger absolute number of people working on complex algorithms, but as a percentage it is much smaller. In the 60's deep algorithmic knowledge was required for 90%+ of programming tasks, now it is probably closer to 1%. It is perfectly possible to make a living as a programmer nowadays without being able to inverse a binary tree
- oriolid 6y ago
  What's the obsession with inverting a binary tree? I have never found an use case that particular algorithm, but on the other hand I would be very happy if more programmers were aware that having a where clause in sql statement (and maybe and index) is more efficient than selecting entire database and looping over the results, even if the end result is the same. I understand that in order to come up with the inversion on the fly you need to know about trees and recursion, but is it really that scary?
  - arethuza 6y ago
    Possibly due to the tweet by Max Howell: "Google: 90% of our engineers use the software you wrote (Homebrew), but you can’t invert a binary tree on a whiteboard so fuck off."
    - oriolid 6y ago

oriolid 6y ago

I have the feeling that the article is really about optimization, but the author does not know the difference.

fogetti 6y ago

And that's why HackerRank, Codility and all the other coding sites have zero relevance today in regards of the actual capability of the candidate. The fact that hiring committees, HR departments and software engineering managers rely on them goes to show how detached all those people are/have been from reality for a long time now. Good software engineering skills on the other hand are pretty much more needed than ever.

collyw 6y ago
I have needed to do one heavily algorithmic task. Once. In nearly 20 years of programming.
cyberdrunk 6y ago
The thing is, in large tech companies like Google, you might be tasked with designing (or modyfing a design) of something that works on Internet scale. For such cases, thinking in terms of algorithms is essential. That's why I think FAANGs still require algorithmic literacy. Unfortunately, other companies, which don't have FAANG-like problems, ape its recruitment process...
- fogetti 6y ago
  Yes, algorithms are essential for information technology. But the question is not that. And the discussion is also not about that. The question is: are algorithms essential for one's career and for the majority of business too? I don't think so.
  - strikelaserclaw 6y ago
    By definition, any code written to solve a problem is an algorithm (not just the standard stuff found in textbooks), since a lot of people in IT write code to solve problems, then yes they absolutely need to understand some what performance characteristics and the correctness of what they wrote. Even if you don't need to performance characteristics, you certainly need the correctness part!
signa11 6y ago

nickcw 6y ago

I spent a lot of time reading Knuth's books, especially Seminumerical Algorithms when I was implementing multi precision arithmetic for various architectures. What amazing pieces of work they are. That was back in the time when you could look at assembler (or even C code) and get a good idea of how many cycles it would take. I love the Art of the Algorithm, but I have to say I'm very happy to use off the shelf libraries now. I've implemented binary search dozens of times and each time spent ages fixing all the corner cases. Instructive, but a waste of time when you can import a library which is battle tested and faster than anything you'll ever write. Algorithms are dead - long live the (commoditized) Algorithm!

chrchang523 6y ago
Incidentally, I think there is at least one important area where Knuth is actually still ahead of off-the-shelf libraries as of 2020: "broadword computing". When skimming volume 4A for semi-recreational purposes a few years ago, I was surprised to find multiple "bit-hacks" which were better than anything I had found in other sources, and proceeded to use them in my own open-sourced work. Every time I've encountered a situation where bitvectors are useful, I have benefited from rolling my own implementation.
- klyrs 6y ago
  Came here to say this. It's rare that you want to consume these things through an abstraction layer: the layer is often heavier than the work underneath. Sucks for readability, though, so I usually spend as much time documenting as implementing.

jasode 6y ago

>This was the age of the algorithm. [...] When it comes to algorithm implementation, developers are now spoilt for choice; why waste time implementing the ‘low’ level stuff when there were plenty of other problems waiting to be implemented. Algorithms are now like the bolts in a bridge: very important, but nobody talks about them. Today developers talk about story points, features, business logic, etc. [...] Today, we are in the age of the ecosystem. The blog author doesn't make it explicit but his usage of "algorithm" is actually about "low-level algorithms" which is how it seems to support his claim that "algorithms are now commodities". But "algorithms" in the general sense are not commodities. It's just that we're now free to concentrate on higher-level problems with new algorithms. E.g., in the 1980s & 1990s, the Barnes & Noble section for computers would have books with algorithms showing how to write disk-based B-trees from scratch. But now with SQLite library in wide usage, you can just use their B-trees instead of reinventing your own. Some more examples of low-level algorithms I used to write by hand: - manually uppercase chars by adding "32" to the ASCII code because there was no Upper() function in a standard library. - manually titlecase/propercase/lowercase a person's name from "JOHN DOE" to "John Doe" by looping the the string char-by-char and subtracting "32" from the ASCII code if it doesn't follows a space char because there was no TitleCase() function - adding a drop shadow beneath a window because the operating system GUI didn't render shadows for you - manually writing link lists to create dynamic size memory buffers because there was no C++ STL library yet or built-in associative arrays in Python/C# I don't do have to do any of that low-level tedious work anymore but I still write new algorithms because there is always some [missing functionality] where a standard library doesn't exist or a consensus implementation hasn't yet "won" in the marketplace of ideas. It could be distributed cloud/desk/mobile data sync algorithm, or collaborative recommendation algorithm, etc. Algorithms (general sense) will not be commoditized for decades -- if ever.

raphlinus 6y ago
I have a feeling you might enjoy this code: https://github.com/zbraniecki/tinystr/blob/master/src/tinystr8.rs#L221 https://github.com/zbraniecki/tinystr/blob/master/src/tinyst... I wrote this because I wanted (normalizing) locale comparisons to be super fast, as they can be part of the key for cache lookups of text layout. In a language identifier such as "en-Latn-US" the script is in title case. So this kind of thing still does happen, just at the lowest levels of libraries that provide nice clean abstractions for other people to use.

scoutt 6y ago

> There are companies where algorithms are not commodities And there is an entire industry! Embedded (but not the Raspberry-pi kind of embedded that is similar to desktop). Try to do a dead reckoning system, or anything related to real-time gesture/motion analysis. Try reading, processing, mixing and outputting audio in real time. Or any DSP related stuff. What about FPGA, ASICs, etc.? Why oh why is the embedded industry so easily forgotten? It's not all cloud, food apps, SaaS and Electron, ya know? Edit to add: all the above with the limited resources of embedded, that forces you to improve your algorithms.

vanderZwan 6y ago
> Why oh why is the embedded industry so easily forgotten? It's not all cloud, food apps, SaaS and Electron, ya know? Because the other industry is inherently linked to the internet. It's like the media telling you that the media is very important, or your brain telling you that it's the most interesting organ in your body (I forget which comedian I stole that from).
pseudosudoer 6y ago
Depends on the embedded ecosystem. STM has tons of free software packages for their products that remove the need to write drivers and a variety of algorithms from scratch. Albeit not the same, Xilinx and Altera have a plethora of FPGA IP that can be licensed/bought, although I can see the merit in a custom implementation here to save money.

jcahill 6y ago

Commodities are fully fungible goods and services. Different implementations of a single algorithm should have the same basic formal properties, by definition. That is, they should be roughly fungible. Else you aren't dealing with correct implementations of the same algorithm. If anything, the thesis "algorithms are now commodities" is less true now than ever before. As the complexity of software deliverables increases, the use of complex, proprietary algorithms in software should likely increase in turn.

nirui 6y ago

> why waste time implementing the ‘low’ level stuff when there were plenty of other problems waiting to be implemented. Because: - The package is licensed through GPL - The lib allocates memory in such way that it messed up my program - The last update to the lib is 3 years ago, and it takes at least another 3 years for the author of the package get out of jail which is the min punishment of some NSL - I don't like how it's interface is designed - Because I got paid even I write everything by myself The list goes on. Another way to look at this is, "Do I really want just to implement a Linked-list? Or an sizable & queueable list that allow me to sort items via LRU?". Sometime, you just have to to write some "low level stuff" by yourself to make it exactly as you wanted.

tanilama 6y ago

I am somewhat confused about the definition of Algorithms under the article's context. What is this Algorithm we are talking about? Is it about basic data structures? If that is what we talk about, I feel at least from like 15 years ago, people already giving up writing those things from scratch for new projects. I agree encapsulation is growing. As developers we are in a reality that we lose more and more predictability over the behavior of our code. But that seems to create more demand for nuanced solution to reign the complexity, since commoditized algorithm created this problem, not solving it.

jvanderbot 6y ago

Well that's just not true. I concur: Nowadays, the basic building blocks are done. Good sorting, database indexing, O(1) set-membership, many variants of lists, hash tables, and other higher level data structures are well covered by modern libraries. The foundations of modern system design are algorithms and data structures, and they are largely commoditized, for general use. Certainly, working on general purpose, clever-but-foundational algorithms is becomming increasingly niche. I disagree with the final conclusion that somehow algorithms have faded into obscurity. Anyone writing any program has to consistently track their expected program performance as it shuffles data between commodotized library calls or between network end points. This is algorithm design. Many "foundational" algorithms like max-flow actually make use of "foundational(er)" algorithms and shuffle data between them. This is what the modern programmer does when he decomposes a data-intensive problem into sub-problems and uses commoditized libraries and re-composes those solutions. The push to catalog and archive a wide array of well-implemented algorithms and data structures tracked the emergence of the personal and commodity hardware. The emergence of new hardware will require new solutions. Mobile, webassembly (perhaps), ASIC-based data centers, CUDA, are all exciting new areas where our "foundational" algorithms and canonical solutions need to be revisited, redesigned, and re-tuned. Or at least re-composed.

brushfoot 6y ago

All that's old is new again. The level of abstraction just keeps getting higher. Someday, ecosystems like AWS and Azure will be as standardized as programming languages. You'll still be writing algorithms, but instead of telling one PC what to do, you'll be telling a distributed system what to do. The old design patterns will still be relevant; the implementation will just look different. We're already there to some extent. It's just messy and fragmented right now.

omarhaneef 6y ago

There are probably more algorithms being "figured out" today than ever before. However, the vast majority are used during interviews, and don't see the dark of silicon.

sdenton4 6y ago

One of my favorite mini games at my job is rewriting classic algorithms to run in batched mode on gpu/tpu. The speed improvements often improve model training time by days, and it's always a lovely intellectual challenge. (The basic challenge is to rewrite the algorithms in terms of matrix operations which operate on many examples of the problem at once, while eliminating all branching.)

_bxg1 6y ago

I remember reading an article a while back about how professional photography has become a commodity. You can find and license any imaginable photograph, for pennies, in minutes. From the best photographers in the world. So from a business perspective, photography as a pure art has been devalued to almost nothing. Nobody pays you to take the very best possible photo. So why do people still hire photographers? To put themselves or their product into a photo. Most photographers are no longer paid to take the very best photos, but to make their photos specific to what the particular client cares about. That's something that can't be commoditized. I think that's where software is now as an industry. Very few of us these days get to innovate at a "pure art" level. Instead, we get paid to pull together a bespoke solution that's sensitive to a given business and its needs. It isn't novel at the macro-level, only at the micro-level. This can be kind of depressing, but it's also a little comforting that those micro-level solutions are so hard to fully commoditize. As long as the real world is messy and varied, the leaves in the software tree will have to be too, and there will be work that needs doing.

ForHackernews 6y ago
I don't see how that's depressing at all. Another way to phrase what you're saying is "Developers get paid to write software that solves problems people actually need solved.
- _bxg1 6y ago
  For many of us, writing something pure and generic and beautiful is much more gratifying. And it's depressing to think that we'll probably never write something like that that's good enough to be picked over the version that was developed at Google and then open-sourced.
  - ForHackernews 6y ago
    I mean, nobody's stopping you from writing your perfect, beautiful Haskell implementation of Dijkstra's Algorithm. I guess it's simply a different approach, but I'd rather not waste time re-implementing the wheel when I could be making software that will help somebody fulfill a concrete need.

opportune 6y ago

Personally I found it a bit disappointing moving from academia to industry and finding out that there is not much of a market for algorithm implementation or development/research. Sure, there are specialized products where they are important (DB, OS, languages/libraries, trading systems) and even places for algorithms in business apps, but it doesn’t seem you can really “specialize” in it as a career (like distributed systems, system programming, frontend) unless you’re very picky about which jobs you take.

Const-me 6y ago

I don’t remember last time I implemented sorting, search or trees, these are available in all languages and runtimes. Skill, I implement a lot of my own algorithms. Following reasons. 1. Performance. Custom algorithms and data structures is often a requirement to write performant code, because SIMD, memory hierarchy, and other modern hardware-related shenanigans largely ignored by authors of standard libraries. 2. GPGPU. Due to their massively parallel nature they need completely different algorithms, a naïve implementation is often by an order of magnitude slower than purposely built one, due to things like memory access coalescing and group shared memory. 3. Licensing. In my area of work (CAD/CAM/CAE) some good quality libraries are only available as GPL or even AGPL. Apparently, the commercial licenses are targeted towards companies like GM or Airbus, way too expensive.

softwaredoug 6y ago

Yes commodities, but I still need an intuitive understanding of how the algorithms work to pick the right one. I might even need to read the code when I get deeper into solving my problem. Or implement a dummy version simply for learning. Algorithms are commodities, but algorithm skill is still deeply needed

softwaredoug 6y ago

I wonder if the number of people working on algorithms is about the same proportion of the workforce, it's just that there's way way more developers now, and they need commoditized algorithms that solve 90% of the problem (but that 10% is still out there...). For example, if you summed up all the 'core committers' of open source projects, people working on operating systems, embedded software, systems programming, and other specialized software developers that might think about unique algorithms, you wonder if that's about the same as the number of total software developers in the 80s