7 ms·

This remains a serious problem on my team- Over half my team members don’t have CS degrees. They get an enormous amount done, but there is often a perf or maint

This remains a serious problem on my team- Over half my team members don’t have CS degrees. They get an enormous amount done, but there is often a perf or maintainability cost. I have no intention of swapping them out because they ultimately provide enormous value, but it’s noticeable.

peterwwillis 8y ago
Is it the algorithmic efficiency that's the problem, or the ability to scale the application? I've seen things some people wouldn't believe. mod_perl 1.4 on Apache 2.0 handling 350,000 hps dynamic traffic on twenty archaic 1U's. Five tiers of caching, serialized objects on local disks, NFS in production. C-beams glittering in the dark near the Tannhäuser gate. Who cares about inefficiency if it scales?
- anyfoo 8y ago
  Putting aside the damage that issues with correctness can cause, the problem is exactly that if you e.g. wrote something that grows with n^2 instead of n log n, you cannot scale by "brute force" anymore, as the inefficiency very quickly outgrows the amount of resources you can add. Seriously, your thinking that "algorithmic inefficiency" can be countered by "scaling" is almost proving the point. CS is by far not just about runtime complexity, but it's one thing that can bite you. Also, all the components you mentioned were likely written, at least in significant parts, by people knowledgeable about computer science.
  - peterwwillis 8y ago
    Scaling an application's operation does not directly relate to its algorithm's efficiency, is my point. Of course a really inefficient application can eventually become unusable, but even terribly performing applications can be "worked around" and thus scaled higher. Say your app is using bubble sort. Egads!!! What a shit design. Clearly this is going to become a nightmare in real world scenarios. But wait - we can theoretically get to O(n) if the list is already nearly sorted. How can we achieve this? By monkeying with the dataset, queueing and batching operations, invalidating operations that take too long, artificially limiting the number of requests per second, or just passing the request to a completely different app depending on use case. It sounds insane, but if you can perform any of these things quicker than redesigning your app, so that it can continue performing under load, that's an example of "scaling" despite the application's poor performance. Another example is the "scalability" of application development. Say you have an application which is basically O(1), but one day you find a bug in it. Even after you write and commit the one-line fix for the bug, if it takes you between four hours and two days to deploy it to production, or the validation process takes a month... You still have a bug in production for hours, days, or weeks. "Scaling" the development process can significantly reduce the amount of time needed to solve problems, or complete new features. It can be more beneficial to be able to ship code faster and more reliably, even if it isn't the most efficient code.
    - anyfoo 8y ago
      but even terribly performing applications can be "worked around" and thus scaled higher. [...] How can we achieve this? By monkeying with the dataset, queueing and batching operations, invalidating operations that take too long, artificially limiting the number of requests per second, or just passing the request to a completely different app depending on use case. And that's a good thing? Part of my point is that this can be avoided, or at least have a higher probability of being unnecessary, if the proper basics were learned. Wouldn't you wish the author would have just known how to properly sort a list[1] in the first place? "Scaling" the development process can significantly reduce the amount of time needed to solve problems, or complete new features. It can be more beneficial to be able to ship code faster and more reliably, even if it isn't the most efficient code Agreed to that, but as said, computer science is not just about runtime complexity. Knowing computer science might help you avoid those situations, or resolving them quicker. I know that shitty stuff can work, even work "well enough", but going back to my original comment: Maybe it could work much better, simpler, more profitable if that's your favorite metric, with just the application of some basics, if they are known. [1] Noting that "sorting a list" is a stand-in for all sorts of tasks that benefit from CS-knowledge.
      - peterwwillis 8y ago
        Yes, absolutely it all could work better with CS knowledge. But CS is also definitely not a prerequisite to having functioning apps that can scale well. CS can help you design the software equivalent of a Formula 1 race car. But most of us only need to work on things ranging from a Honda Civic to a semi truck. It's interesting how little of an education you need to work on a semi truck, even though it's a complex machine that does a tremendous amount of work, compared to the education you need to work on Formula 1 cars. In the US, we have a big lack of tech talent, and I think part of the reason is the way people propose the requirements or education required to do these jobs. If it's a Honda Civic-level programming task, people shouldn't go through the expense and difficulty of a CS degree.
        anyfoo 8y ago
        I think we managed to get on the same page then, I can agree to all of that.
- topkai22 8y ago
  It’s algorithmic efficiency and at a basic level- things like 9 levels of nested loops, each loop pass generating an http call (admitadly some of this is the platform we are using fault)

peterwwillis 8y ago

Is it the algorithmic efficiency that's the problem, or the ability to scale the application? I've seen things some people wouldn't believe. mod_perl 1.4 on Apache 2.0 handling 350,000 hps dynamic traffic on twenty archaic 1U's. Five tiers of caching, serialized objects on local disks, NFS in production. C-beams glittering in the dark near the Tannhäuser gate. Who cares about inefficiency if it scales?

anyfoo 8y ago
Putting aside the damage that issues with correctness can cause, the problem is exactly that if you e.g. wrote something that grows with n^2 instead of n log n, you cannot scale by "brute force" anymore, as the inefficiency very quickly outgrows the amount of resources you can add. Seriously, your thinking that "algorithmic inefficiency" can be countered by "scaling" is almost proving the point. CS is by far not just about runtime complexity, but it's one thing that can bite you. Also, all the components you mentioned were likely written, at least in significant parts, by people knowledgeable about computer science.
- peterwwillis 8y ago
  Scaling an application's operation does not directly relate to its algorithm's efficiency, is my point. Of course a really inefficient application can eventually become unusable, but even terribly performing applications can be "worked around" and thus scaled higher. Say your app is using bubble sort. Egads!!! What a shit design. Clearly this is going to become a nightmare in real world scenarios. But wait - we can theoretically get to O(n) if the list is already nearly sorted. How can we achieve this? By monkeying with the dataset, queueing and batching operations, invalidating operations that take too long, artificially limiting the number of requests per second, or just passing the request to a completely different app depending on use case. It sounds insane, but if you can perform any of these things quicker than redesigning your app, so that it can continue performing under load, that's an example of "scaling" despite the application's poor performance. Another example is the "scalability" of application development. Say you have an application which is basically O(1), but one day you find a bug in it. Even after you write and commit the one-line fix for the bug, if it takes you between four hours and two days to deploy it to production, or the validation process takes a month... You still have a bug in production for hours, days, or weeks. "Scaling" the development process can significantly reduce the amount of time needed to solve problems, or complete new features. It can be more beneficial to be able to ship code faster and more reliably, even if it isn't the most efficient code.
  - anyfoo 8y ago
    but even terribly performing applications can be "worked around" and thus scaled higher. [...] How can we achieve this? By monkeying with the dataset, queueing and batching operations, invalidating operations that take too long, artificially limiting the number of requests per second, or just passing the request to a completely different app depending on use case. And that's a good thing? Part of my point is that this can be avoided, or at least have a higher probability of being unnecessary, if the proper basics were learned. Wouldn't you wish the author would have just known how to properly sort a list[1] in the first place? "Scaling" the development process can significantly reduce the amount of time needed to solve problems, or complete new features. It can be more beneficial to be able to ship code faster and more reliably, even if it isn't the most efficient code Agreed to that, but as said, computer science is not just about runtime complexity. Knowing computer science might help you avoid those situations, or resolving them quicker. I know that shitty stuff can work, even work "well enough", but going back to my original comment: Maybe it could work much better, simpler, more profitable if that's your favorite metric, with just the application of some basics, if they are known. [1] Noting that "sorting a list" is a stand-in for all sorts of tasks that benefit from CS-knowledge.
    - peterwwillis 8y ago
      Yes, absolutely it all could work better with CS knowledge. But CS is also definitely not a prerequisite to having functioning apps that can scale well. CS can help you design the software equivalent of a Formula 1 race car. But most of us only need to work on things ranging from a Honda Civic to a semi truck. It's interesting how little of an education you need to work on a semi truck, even though it's a complex machine that does a tremendous amount of work, compared to the education you need to work on Formula 1 cars. In the US, we have a big lack of tech talent, and I think part of the reason is the way people propose the requirements or education required to do these jobs. If it's a Honda Civic-level programming task, people shouldn't go through the expense and difficulty of a CS degree.
      - anyfoo 8y ago
        I think we managed to get on the same page then, I can agree to all of that.
topkai22 8y ago
It’s algorithmic efficiency and at a basic level- things like 9 levels of nested loops, each loop pass generating an http call (admitadly some of this is the platform we are using fault)