4 ms·

All of these hypotheses will take on the order of days to check. OK, but you can check them, right? How is that different from a regular software bug?

by p1esk 3y ago

janalsncm 3y ago
In software engineering you can test things in something on the order of seconds to minutes. Functions have fixed contracts which can be unit tested. In ML your turnaround time is days. That alone makes things harder. Further, some of the problems I listed are open-ended which makes it very difficult to debug them.
- rezonant 3y ago
  > In software engineering you can test things in something on the order of seconds to minutes. Functions have fixed contracts which can be unit tested. I think this only applies to a certain subset of software engineering, the one that rhymes with "tine of christmas". Implementing bitstream formats is an area I'm very familiar with, and I dance when an issue takes seconds to resolve. Sometimes you need to physically haul a vendor's equipment to the lab. In broadcast we have this thing called "Interops" where tons of software and hardware vendors do just this, but in a more convention-esque style (actually is often done at actual conventions).
  - rjmunro 3y ago
    > rhymes with "tine of christmas". What?
    - rezonant 3y ago
      line of business
- p1esk 3y ago
  I've been an ML researcher for the last 11 years. Last week I spent 3 days debugging an issue in my code which had nothing to do with ML. It was my search algorithm not properly modifying the state of an object in my quantization algorithm. Individually, both algorithms worked correctly but the overall result was incorrect. Looking back at my career, the hardest bugs to spot were not in ML, but in distributed systems, parallel data processing algorithms, task scheduling, network routing, and in dynamic programming. Of course I also had a ton of ML related bugs, but over the years I developed tools and intuition to deal with them, so usually I can narrow down an issue (like impacted accuracy, or training not converging) fairly quickly. I don't think these kind of bugs are fundamentally different from any other bugs in software engineering. You typically try to isolate an issue by simplifying the environment, break it down into parts, test on toy problems, trace program execution and print out or visualize values.
blackbear_ 3y ago
What makes some bugs in ML algorithms hard to spot is that many of then hinder, but do not prevent, the model from learning. They can be really hard to spot because you do see the model learning and getting better, and yet without that bug the predictions could be even more accurate. Only with domain experience you can tell that something might be wrong. Moreover, this kind of issues are usually related to the mathematical aspect of the model, meaning that you need to understand the theoretical motivation of things and check all operations one by one. Just this week for example I was dealing with a bug there where we were normalizing on the wrong dimension of a tensor.
- p1esk 3y ago
  Only with domain experience you can tell that something might be wrong. Obviously. How is this different from any other field of science or engineering? you need to understand the theoretical motivation of things and check all operations one by one. Again, this is true when debugging any complex system. How else would you debug it? a bug there where we were normalizing on the wrong dimension of a tensor If you describe the methodology you used to debug it, it will probably be applicable to debugging a complicated issue in any other SWE domain.
  - janalsncm 3y ago
    Because the difference is that statistical models are by definition somewhat stochastic. Some incorrect answers are to be expected, even if you do everything right. In software engineering you have test code. 100% of your tests should pass. If one doesn’t you can debug it until it does.
  - blackbear_ 3y ago
    > How is this different from any other field of science or engineering? The difference is that in most cases it is not so clear how well any given approach will work in a given scenario. Often the only option is to try, and if performance is not satisfying it is not easy to find a reason for it. Besides bugs or wrong model choice, it could be wrong training parameters, the quality or quantity of the data, and who knows how much more you would need. It's not necessarily different from SWE, problem solving is a general skill, the difficulty comes from the fact that there is no clear definition of "it works" and that there are no guidelines or templates to follow to find out what is wrong, if anything at all. In particular, many issues are not about the code.

All of these hypotheses will take on the order of days to check. OK, but you can check them, right? How is that different from a regular software bug?

by p1esk 3y ago

All of these hypotheses will take on the order of days to check. OK, but you can check them, right? How is that different from a regular software bug?

janalsncm 3y ago

In software engineering you can test things in something on the order of seconds to minutes. Functions have fixed contracts which can be unit tested. In ML your turnaround time is days. That alone makes things harder. Further, some of the problems I listed are open-ended which makes it very difficult to debug them.

rezonant 3y ago
> In software engineering you can test things in something on the order of seconds to minutes. Functions have fixed contracts which can be unit tested. I think this only applies to a certain subset of software engineering, the one that rhymes with "tine of christmas". Implementing bitstream formats is an area I'm very familiar with, and I dance when an issue takes seconds to resolve. Sometimes you need to physically haul a vendor's equipment to the lab. In broadcast we have this thing called "Interops" where tons of software and hardware vendors do just this, but in a more convention-esque style (actually is often done at actual conventions).
- rjmunro 3y ago
  > rhymes with "tine of christmas". What?
  - rezonant 3y ago
    line of business
p1esk 3y ago
I've been an ML researcher for the last 11 years. Last week I spent 3 days debugging an issue in my code which had nothing to do with ML. It was my search algorithm not properly modifying the state of an object in my quantization algorithm. Individually, both algorithms worked correctly but the overall result was incorrect. Looking back at my career, the hardest bugs to spot were not in ML, but in distributed systems, parallel data processing algorithms, task scheduling, network routing, and in dynamic programming. Of course I also had a ton of ML related bugs, but over the years I developed tools and intuition to deal with them, so usually I can narrow down an issue (like impacted accuracy, or training not converging) fairly quickly. I don't think these kind of bugs are fundamentally different from any other bugs in software engineering. You typically try to isolate an issue by simplifying the environment, break it down into parts, test on toy problems, trace program execution and print out or visualize values.

blackbear_ 3y ago

What makes some bugs in ML algorithms hard to spot is that many of then hinder, but do not prevent, the model from learning. They can be really hard to spot because you do see the model learning and getting better, and yet without that bug the predictions could be even more accurate. Only with domain experience you can tell that something might be wrong. Moreover, this kind of issues are usually related to the mathematical aspect of the model, meaning that you need to understand the theoretical motivation of things and check all operations one by one. Just this week for example I was dealing with a bug there where we were normalizing on the wrong dimension of a tensor.

p1esk 3y ago
Only with domain experience you can tell that something might be wrong. Obviously. How is this different from any other field of science or engineering? you need to understand the theoretical motivation of things and check all operations one by one. Again, this is true when debugging any complex system. How else would you debug it? a bug there where we were normalizing on the wrong dimension of a tensor If you describe the methodology you used to debug it, it will probably be applicable to debugging a complicated issue in any other SWE domain.
- janalsncm 3y ago
  Because the difference is that statistical models are by definition somewhat stochastic. Some incorrect answers are to be expected, even if you do everything right. In software engineering you have test code. 100% of your tests should pass. If one doesn’t you can debug it until it does.
- blackbear_ 3y ago
  > How is this different from any other field of science or engineering? The difference is that in most cases it is not so clear how well any given approach will work in a given scenario. Often the only option is to try, and if performance is not satisfying it is not easy to find a reason for it. Besides bugs or wrong model choice, it could be wrong training parameters, the quality or quantity of the data, and who knows how much more you would need. It's not necessarily different from SWE, problem solving is a general skill, the difficulty comes from the fact that there is no clear definition of "it works" and that there are no guidelines or templates to follow to find out what is wrong, if anything at all. In particular, many issues are not about the code.