3 ms·

I don’t think this is true. Security primitives are more widely available in software libraries now than ever before, but you don’t see people believing they ca

by mlthoughts2018 6y ago

I don’t think this is true. Security primitives are more widely available in software libraries now than ever before, but you don’t see people believing they can read a “security for hackers” blog post and then roll their own encryption tool or secrets management tool and use it in production. More than ever, the distinction between a security professional and a layperson who read about RSA algorithms is hugely critical. It’s precisely the same with machine learning and statistical inference. The lower barrier to entry just means the danger of releasing extremely unsafe projects is much higher, and the careful validation from experts is that much more critical.

quonn 6y ago
If the model produces good observable results in practice it works. For security it‘s different: It may seem to work, yet be completely insecure (security being the goal).
- mlthoughts2018 6y ago
  Without statistical expertise, how would you know the model produces good results? (If you say something like, “just look at the business outcome of the model” you are proving my point about the danger, because that would be a catastrophically bad way to judge the performance of a model. What accuracy metric did you use? Why? Did you understand training / serving skew? Did you look at a confusion matrix or study class imbalance? What about missing data? What statistical test did you use? Did you adjust for multiple-testing? Was there peek-ahead bias? Did you test for discontinuities and non-linearities that can render p-values inapplicable for tests of linear models? What was your training convergence like? What simple baselines did you compare to?)
  - quonn 6y ago
    No, it‘s not. Most machine learning is used for practical applications where it‘s easy to judge if it works. For example, if I provide some automatic assignment of let‘s say a related object, reducing the work required to find the correct one and the team using it is much faster, then I can measure this and that‘s enough. Likewise for, let‘s say a recommender system. The BI department will look at conversions and that‘s in fact the only thing that matters. Nobody cares how good the model is. What we care about is whether the recommendations have the desired effect in that case. And besides, what‘s hard about measuring how accurate a model is for predictions? Compared to most things in a typical CS curriculum that‘s rather easy indeed. Finally, what‘s „dangerous“? ML is usually applied in a business context. Better recommendations; automatic assignment of objects. Automatic classification of pictures. Things like that. Sure, in a medical context perhaps or financial. Then it may be dangerous. For everything else, maybe you get a worse result. That‘s not unlike everything else in software engineering where many people with different skill levels get very different results (some better some much worse).
    - zwaps 6y ago
      Careful with this reasoning! I have this anecdote of a industrial sorting/production machine that worked for literally a year until someone opened the door on a windy day and everything went flying thanks to the AI.

quonn 6y ago

If the model produces good observable results in practice it works. For security it‘s different: It may seem to work, yet be completely insecure (security being the goal).

mlthoughts2018 6y ago
Without statistical expertise, how would you know the model produces good results? (If you say something like, “just look at the business outcome of the model” you are proving my point about the danger, because that would be a catastrophically bad way to judge the performance of a model. What accuracy metric did you use? Why? Did you understand training / serving skew? Did you look at a confusion matrix or study class imbalance? What about missing data? What statistical test did you use? Did you adjust for multiple-testing? Was there peek-ahead bias? Did you test for discontinuities and non-linearities that can render p-values inapplicable for tests of linear models? What was your training convergence like? What simple baselines did you compare to?)
- quonn 6y ago
  No, it‘s not. Most machine learning is used for practical applications where it‘s easy to judge if it works. For example, if I provide some automatic assignment of let‘s say a related object, reducing the work required to find the correct one and the team using it is much faster, then I can measure this and that‘s enough. Likewise for, let‘s say a recommender system. The BI department will look at conversions and that‘s in fact the only thing that matters. Nobody cares how good the model is. What we care about is whether the recommendations have the desired effect in that case. And besides, what‘s hard about measuring how accurate a model is for predictions? Compared to most things in a typical CS curriculum that‘s rather easy indeed. Finally, what‘s „dangerous“? ML is usually applied in a business context. Better recommendations; automatic assignment of objects. Automatic classification of pictures. Things like that. Sure, in a medical context perhaps or financial. Then it may be dangerous. For everything else, maybe you get a worse result. That‘s not unlike everything else in software engineering where many people with different skill levels get very different results (some better some much worse).
  - zwaps 6y ago
    Careful with this reasoning! I have this anecdote of a industrial sorting/production machine that worked for literally a year until someone opened the door on a windy day and everything went flying thanks to the AI.