5 ms·

My experience, as someone who isn't a professional full-time developer (systems engineer / devops focused), but who personally geeks out on CS theory, is I find

My experience, as someone who isn't a professional full-time developer (systems engineer / devops focused), but who personally geeks out on CS theory, is I find it fairly rare for developers that I work with that think of Big O considerations when writing their code. A prime example, was a piece of code that has to process incoming results that get stuck in a transnational database table. Instead of selecting, then processing, the records that the module cares about, it does the selection (which typically returns 1 - 10 records), sorts them, processes the record from the top of the list, then throws the rest of the initial select results away. It then goes back to the DB an does the same select, and repeats the process. End result: if the queue gets backed up to 100, or 1000 records, the process never catches up. You have to temporarily change the status of the inbound records to something else, then put maybe 10 at a time back to pending. There are countless other times I've seen similar types of issues where on small data sets the code runs fine, but the performance degrades to n^2, or even worse is when it degrades in a factorial manor. And the worst of it is that if I talk to the developers using common CS notation (such as, your algorithm should have logarithmic or at worse linear degradation as the input queue grows, and it is behaving exponentially), their eyes glaze over.

eindiran 8y ago
That's not my experience as a software engineer at all. In fact, I would say that I see people worrying about writing algorithms that are asymptotically optimal (well before that should ever be a consideration) 10x as often as I see people failing to consider the efficiency of the algorithms they write.
- derekp7 8y ago
  Do you think this is due to a difference between working at a place that has software requiring a specific non-computer related skill set (such as medical diagnostics software), vs being at a place that hires primarily CS grads for regular software? Other developers that I've worked around at previous jobs were in manufacturing (Informix 4GL devs), and an IT services company, among others. The one common thread I've seen though is they tend to be impressed by what I can do in C, for example, or helping them get to the root of an issue they have (I will typically reverse engineer what they are doing from the outside looking in, using things like strace, then writing up an analysis of what I see from the systems side). Another recent example is a case where the app needed to start a number of sub-processes. So the dev used an equiv of a system() call, which spawned a bash shell, to do a ps -ef |grep process_name |grep -v grep |grep -v vi |grep -v vim |grep -v less, to see if the process was running, then doing another system() call to start the sub process. The whole string would be repeated for each sub process, and then they wondered why it took 8 minutes to start all their sub interfaces when they deployed to a larger customer that needed 700 interfaces started (instead of the typical 20 - 50). Oh, and after they would start an interface, the code would do the "ps..." commands on all the other interfaces to update an internal table on their status.
  - eindiran 8y ago
    That might very well be there difference. My coworkers are almost exclusively people who studied CS, and several wrote C professionally at some point.
  - mixmastamyk 8y ago
    I don't think one needs to be a CS grad to recognize such a poor design.
User23 8y ago
One of the things I like about the C++ STL is once you learn the idioms, the API tells you whether an operation is "fast." I don't really care if a line programmer can give me the formal definition of O(x), but I sure want him to know if an operation is appropriate to call in a loop. That's speaking as a professional. As someone who finds CS fascinating, I'm appalled by the lack of interest many professional developers have in what is one of the most rigorous and elegant fields of mathematical inquiry. Even so I don't expect others to care about the things I do.
- sidlls 8y ago
  Why are you appalled? As a physicist I'm not appalled by the lack of interest most engineers have in fundamental theory that I know inside out. The reason is precisely because I understand there's a difference between practical applications and theoretical inquiry.
Retric 8y ago
The example you gave is not really a Big O issue. It's the extra DB calls adding a huge constant that's the problem not the underlying algorithm. O(x) can be horrible when the constant is 1/10th of a second each record. This may be a case of the developers not understanding your ORM tooling. They get something that works, and don't quite what's going on under the hood.
chrisseaton 8y ago
> I find it fairly rare for developers that I work with that think of Big O considerations when writing their code Do you not hear people talk about 'accidentally quadratic' quite a bit? I hear practising developers talk about that a lot. They seem to have a good idea of what to avoid.
- pvg 8y ago
  'Accidentally quadratic' happens and is talked about precisely because much of the time, there's not much reason to think about Big O in many practical software development situations.
imtringued 8y ago
It doesn't sound like teaching Big O Notation would help your coworkers write faster software. You would be better off by teaching them how to use databases properly.
- deleted 8y ago
  [deleted]
roel_v 8y ago
" is I find it fairly rare for developers that I work with that think of Big O considerations when writing their code." Do you mean that as in 'people I work with don't care about this, therefore it's not necessary' or as in 'people I work with are clowns who don't even know foundations'? Because (I think) I'm seeing replies to your post interpreting it both ways.
- derekp7 8y ago
  I wouldn't use the word clowns, because they are really good at other areas. In this case, the developers focus on a development environment that isn't widely used in the US, and I believe that they haven't had a formal CS education (more of the same type of self taught that I am).
  - roel_v 8y ago
    "I wouldn't use the word clowns" Nah it was just an expression from my part, didn't mean to say that anyone not thinking about algorithmic complexity when writing any piece of software is a hack :)
p1necone 8y ago
I don't think knowing about big O notation (or really much cs theory) is a requirement for reasoning about this particular case. "network/sql calls are slow" and "don't do the same work multiple times when it isn't necessary" would be enough.
codingdave 8y ago
> their eyes glaze over. Sounds like a communication problem to me. Non-academic developers might not know "Big O notation". But they absolutely know that nested loops are way less efficient than single loops. They know that some code performs better than other code. And while they might be writing inefficient code, that is an opportunity to talk to them and help them improve. If you come at them with academic terms (right though you may be), and their eyes glaze over, try talking about the actual change you are seeking. Especially in the database world, as less experienced developers might not know that database performance can be tuned by how you use it. We all were new once. So educate them. Kindly.
deathanatos 8y ago
> I find it fairly rare for developers that I work with that think of Big O considerations when writing their code. Every time I choose between a set and a vector in code, I'm essentially drawing on my knowledge of their big-O properties to choose the right one for whatever task I have at hand. I wouldn't say that's the only piece of CS theory I use, either. I routinely run across problems that are best modeled as DAGs (any dependency relationship between items boils down to this quickly). I use DB theory any time I design a database table. I draw on language theory to write parsers. Most of it is reflexive, I think, at this point, but it doesn't change that a "vector" is the right structure b/c I don't need O(1) lookup, or I'd like indexing.

eindiran 8y ago

That's not my experience as a software engineer at all. In fact, I would say that I see people worrying about writing algorithms that are asymptotically optimal (well before that should ever be a consideration) 10x as often as I see people failing to consider the efficiency of the algorithms they write.

derekp7 8y ago
Do you think this is due to a difference between working at a place that has software requiring a specific non-computer related skill set (such as medical diagnostics software), vs being at a place that hires primarily CS grads for regular software? Other developers that I've worked around at previous jobs were in manufacturing (Informix 4GL devs), and an IT services company, among others. The one common thread I've seen though is they tend to be impressed by what I can do in C, for example, or helping them get to the root of an issue they have (I will typically reverse engineer what they are doing from the outside looking in, using things like strace, then writing up an analysis of what I see from the systems side). Another recent example is a case where the app needed to start a number of sub-processes. So the dev used an equiv of a system() call, which spawned a bash shell, to do a ps -ef |grep process_name |grep -v grep |grep -v vi |grep -v vim |grep -v less, to see if the process was running, then doing another system() call to start the sub process. The whole string would be repeated for each sub process, and then they wondered why it took 8 minutes to start all their sub interfaces when they deployed to a larger customer that needed 700 interfaces started (instead of the typical 20 - 50). Oh, and after they would start an interface, the code would do the "ps..." commands on all the other interfaces to update an internal table on their status.
- eindiran 8y ago
  That might very well be there difference. My coworkers are almost exclusively people who studied CS, and several wrote C professionally at some point.
- mixmastamyk 8y ago
  I don't think one needs to be a CS grad to recognize such a poor design.

User23 8y ago

One of the things I like about the C++ STL is once you learn the idioms, the API tells you whether an operation is "fast." I don't really care if a line programmer can give me the formal definition of O(x), but I sure want him to know if an operation is appropriate to call in a loop. That's speaking as a professional. As someone who finds CS fascinating, I'm appalled by the lack of interest many professional developers have in what is one of the most rigorous and elegant fields of mathematical inquiry. Even so I don't expect others to care about the things I do.

sidlls 8y ago
Why are you appalled? As a physicist I'm not appalled by the lack of interest most engineers have in fundamental theory that I know inside out. The reason is precisely because I understand there's a difference between practical applications and theoretical inquiry.

Retric 8y ago

The example you gave is not really a Big O issue. It's the extra DB calls adding a huge constant that's the problem not the underlying algorithm. O(x) can be horrible when the constant is 1/10th of a second each record. This may be a case of the developers not understanding your ORM tooling. They get something that works, and don't quite what's going on under the hood.

chrisseaton 8y ago

> I find it fairly rare for developers that I work with that think of Big O considerations when writing their code Do you not hear people talk about 'accidentally quadratic' quite a bit? I hear practising developers talk about that a lot. They seem to have a good idea of what to avoid.

pvg 8y ago
'Accidentally quadratic' happens and is talked about precisely because much of the time, there's not much reason to think about Big O in many practical software development situations.

imtringued 8y ago

It doesn't sound like teaching Big O Notation would help your coworkers write faster software. You would be better off by teaching them how to use databases properly.

deleted 8y ago
[deleted]

roel_v 8y ago

" is I find it fairly rare for developers that I work with that think of Big O considerations when writing their code." Do you mean that as in 'people I work with don't care about this, therefore it's not necessary' or as in 'people I work with are clowns who don't even know foundations'? Because (I think) I'm seeing replies to your post interpreting it both ways.

derekp7 8y ago
I wouldn't use the word clowns, because they are really good at other areas. In this case, the developers focus on a development environment that isn't widely used in the US, and I believe that they haven't had a formal CS education (more of the same type of self taught that I am).
- roel_v 8y ago
  "I wouldn't use the word clowns" Nah it was just an expression from my part, didn't mean to say that anyone not thinking about algorithmic complexity when writing any piece of software is a hack :)

p1necone 8y ago

I don't think knowing about big O notation (or really much cs theory) is a requirement for reasoning about this particular case. "network/sql calls are slow" and "don't do the same work multiple times when it isn't necessary" would be enough.

codingdave 8y ago

> their eyes glaze over. Sounds like a communication problem to me. Non-academic developers might not know "Big O notation". But they absolutely know that nested loops are way less efficient than single loops. They know that some code performs better than other code. And while they might be writing inefficient code, that is an opportunity to talk to them and help them improve. If you come at them with academic terms (right though you may be), and their eyes glaze over, try talking about the actual change you are seeking. Especially in the database world, as less experienced developers might not know that database performance can be tuned by how you use it. We all were new once. So educate them. Kindly.

deathanatos 8y ago

> I find it fairly rare for developers that I work with that think of Big O considerations when writing their code. Every time I choose between a set and a vector in code, I'm essentially drawing on my knowledge of their big-O properties to choose the right one for whatever task I have at hand. I wouldn't say that's the only piece of CS theory I use, either. I routinely run across problems that are best modeled as DAGs (any dependency relationship between items boils down to this quickly). I use DB theory any time I design a database table. I draw on language theory to write parsers. Most of it is reflexive, I think, at this point, but it doesn't change that a "vector" is the right structure b/c I don't need O(1) lookup, or I'd like indexing.