3 ms·

How to evaluate a cybersecurity vendor's ML claims

mrits 6y ago
This seems like a good way to evaluate a Sales Engineer. With a couple of decades in this industry I've noticed that the sales engineers are just as likely to give an answer about their previous company as the current one.
gruez 6y ago
>3) Ask them where on Wikipedia you can read more about the approach they took. If you can't read about it on Wikipedia, ask them where their paper is in the peer-review and on arXiv. If the paper doesn't exist / is a "trade secret", deduct 3 points Is it common for ML companies to publish their algorithms?
- lucb1e 6y ago
  Rather the other way around: if they don't use a standard algorithm, it's probably beyond useless. Maybe it's comparable to hiring a dev company that proudly proclaims they use a proprietary programming language that is a trade secret.
  - monkeybutton 6y ago
    So similar to "We use a proprietary data store instead of an off the shelf RDBMS" or "We wrote our own cryptographic library"?
- site-packages1 6y ago
  I was thinking about this one as well. If the process/algorithm/etc. verifiably works on real data for the real use case, even if that's some ripped off the shelf algo or some process combining multiple algos, I think that's fine. Things can be trade secret and not be bad.
- abhgh 6y ago
  Yes and no. But even for cases, where the algorithm is not published, the general approach can be indicated and hence "Ask them where on Wikipedia you can read more about the approach they took" might be still valid. For ex. you can say you are using a seq2seq model but you don't have to disclose the precise architecture, preprocessing, optimizations and other training nuances.
lucb1e 6y ago
I tried to make a ML system to detect malware on a network. Kept coming back to what is basically the second and fourth points: Rules and honeypots work better and more simply. A machine learning system will need to learn from something, continuously because malware will adapt to what the model learned previously. Every time you figure out a way to make it classify some traffic as malicious and other traffic not, you're basically writing the rule, except it's like beating a particularly retarded rat brain into figuring out and writing the rule for you. For reference, the 2nd and 4th points are > How do they know their test data is anything like real life data? How do they monitor system performance in the field? > Ask them why they didn't take a simpler approach than the approach they took. Making malware trigger rules is simpler and basically the result of #2.
mellosouls 6y ago
This headline reminds me of "Time AI" https://www.vice.com/en/article/8xw9kp/black-hat-talk-about-time-ai-causes-uproar-is-deleted-by-conference https://www.vice.com/en/article/8xw9kp/black-hat-talk-about-... https://www.theregister.com/2019/08/26/black_hat_sued/ https://www.theregister.com/2019/08/26/black_hat_sued/
agentofoblivion 6y ago
This is a little silly. ML is very good at building things much more general than rules. As a person that’s built and deployed a lot of ML solutions in a related space and brought a product to market meant to help others do the same, we consistently ran into two main camps: 1) people that had a long history of rule writing and were unreasonably closed off to the idea that ML could help and 2) people that thought ML was magic and could do things it most certainly cannot. I understand the sentiment of being skeptical in a world of outrageous claims from “fake it till you make it” startups, but the idea that most ML is not better than rule based systems is just wrong. You need both and they solve slightly different problems.
- DarthGhandi 6y ago
  > As a person that’s built and deployed a lot of ML solutions Care to name one?
motohagiography 6y ago
Have been on the other side of that call and those questions. I had straight answers for each of them. But if I worked in a bank, I wouldn't use an ML behavioral security tool to stop bad guys, I'd use it to identify, model, and detect top traders so I could double down on their bets. Seriously, if the model actually worked, why would you waste it catching hackers? Arguably, if the underlying tech is not useful for that, it's not useful for anything. What it comes down to is, "the confusion matrix is the product." The mistake ML tech companies seem to make is thinking that their product applies to a given vertical (banking, health, govt, etc) , but in fact the balance of that confusion matrix applies to companies whose growth stage defines their needs, imo.
resfirestar 6y ago
Working in consulting with a lot of these products, I don’t really see ML claims as indicative of anything one way or another. I know some pretty good products claim to have advanced ML and some better ones that don’t talk about ML at all, and some really bad ones on all parts of the spectrum. Rather than focusing on ways to get a gotcha on some poor sales team that probably doesn’t control the language used in their marketing, if you want to evaluate whether an EDR product is able to detect things the best way is probably to skip to steps 7 and 8 and ask them about testing - has the product undergone a MITRE evaluation? What’s their strategy for discovering new tactics and validating that they are detected? If you’re evaluating the product on a live demo, your red team should ideally be giving a decent battery of tests, not just firing a macro-laden Word doc and Metasploit...even traditional antivirus ought to be catching that stuff in $currentYear, though of course many don’t.
egberts1 6y ago
If they use discriminators like K-Splice, Munkres, Bloom, Kleene Hierarchy, Markov-Chain to provide refined feedback into their ML, then there is little reason to doubt that ML is being used.
gwittel 6y ago
ML and security is tricky. Its a good tool, but shouldn't be the only one. With threats constantly changing, ML alone will fall short. Its great for so called 'easy' stuff or locating 'odd' stuff to prioritize. On top of that you need explainability and reproducibility. Depending on the ML in use, this can be difficult. When a FP or FN occurs, customers want to know what happened and can you fix it. They don't like to be told things like "because the ML said so." Reproducibility is important for support. When things go wrong, they need to know how to right it. If you can't explain or reproduce, its hard to say the misclassification is resolved and won't happen again.

This seems like a good way to evaluate a Sales Engineer. With a couple of decades in this industry I've noticed that the sales engineers are just as likely to give an answer about their previous company as the current one.

gruez 6y ago

>3) Ask them where on Wikipedia you can read more about the approach they took. If you can't read about it on Wikipedia, ask them where their paper is in the peer-review and on arXiv. If the paper doesn't exist / is a "trade secret", deduct 3 points Is it common for ML companies to publish their algorithms?

lucb1e 6y ago
Rather the other way around: if they don't use a standard algorithm, it's probably beyond useless. Maybe it's comparable to hiring a dev company that proudly proclaims they use a proprietary programming language that is a trade secret.
- monkeybutton 6y ago
  So similar to "We use a proprietary data store instead of an off the shelf RDBMS" or "We wrote our own cryptographic library"?
site-packages1 6y ago
I was thinking about this one as well. If the process/algorithm/etc. verifiably works on real data for the real use case, even if that's some ripped off the shelf algo or some process combining multiple algos, I think that's fine. Things can be trade secret and not be bad.
abhgh 6y ago
Yes and no. But even for cases, where the algorithm is not published, the general approach can be indicated and hence "Ask them where on Wikipedia you can read more about the approach they took" might be still valid. For ex. you can say you are using a seq2seq model but you don't have to disclose the precise architecture, preprocessing, optimizations and other training nuances.

lucb1e 6y ago

I tried to make a ML system to detect malware on a network. Kept coming back to what is basically the second and fourth points: Rules and honeypots work better and more simply. A machine learning system will need to learn from something, continuously because malware will adapt to what the model learned previously. Every time you figure out a way to make it classify some traffic as malicious and other traffic not, you're basically writing the rule, except it's like beating a particularly retarded rat brain into figuring out and writing the rule for you. For reference, the 2nd and 4th points are > How do they know their test data is anything like real life data? How do they monitor system performance in the field? > Ask them why they didn't take a simpler approach than the approach they took. Making malware trigger rules is simpler and basically the result of #2.

mellosouls 6y ago

This headline reminds me of "Time AI" https://www.vice.com/en/article/8xw9kp/black-hat-talk-about-time-ai-causes-uproar-is-deleted-by-conference https://www.vice.com/en/article/8xw9kp/black-hat-talk-about-... https://www.theregister.com/2019/08/26/black_hat_sued/ https://www.theregister.com/2019/08/26/black_hat_sued/

agentofoblivion 6y ago

This is a little silly. ML is very good at building things much more general than rules. As a person that’s built and deployed a lot of ML solutions in a related space and brought a product to market meant to help others do the same, we consistently ran into two main camps: 1) people that had a long history of rule writing and were unreasonably closed off to the idea that ML could help and 2) people that thought ML was magic and could do things it most certainly cannot. I understand the sentiment of being skeptical in a world of outrageous claims from “fake it till you make it” startups, but the idea that most ML is not better than rule based systems is just wrong. You need both and they solve slightly different problems.

DarthGhandi 6y ago
> As a person that’s built and deployed a lot of ML solutions Care to name one?

motohagiography 6y ago

Have been on the other side of that call and those questions. I had straight answers for each of them. But if I worked in a bank, I wouldn't use an ML behavioral security tool to stop bad guys, I'd use it to identify, model, and detect top traders so I could double down on their bets. Seriously, if the model actually worked, why would you waste it catching hackers? Arguably, if the underlying tech is not useful for that, it's not useful for anything. What it comes down to is, "the confusion matrix is the product." The mistake ML tech companies seem to make is thinking that their product applies to a given vertical (banking, health, govt, etc) , but in fact the balance of that confusion matrix applies to companies whose growth stage defines their needs, imo.

resfirestar 6y ago

Working in consulting with a lot of these products, I don’t really see ML claims as indicative of anything one way or another. I know some pretty good products claim to have advanced ML and some better ones that don’t talk about ML at all, and some really bad ones on all parts of the spectrum. Rather than focusing on ways to get a gotcha on some poor sales team that probably doesn’t control the language used in their marketing, if you want to evaluate whether an EDR product is able to detect things the best way is probably to skip to steps 7 and 8 and ask them about testing - has the product undergone a MITRE evaluation? What’s their strategy for discovering new tactics and validating that they are detected? If you’re evaluating the product on a live demo, your red team should ideally be giving a decent battery of tests, not just firing a macro-laden Word doc and Metasploit...even traditional antivirus ought to be catching that stuff in $currentYear, though of course many don’t.

egberts1 6y ago

If they use discriminators like K-Splice, Munkres, Bloom, Kleene Hierarchy, Markov-Chain to provide refined feedback into their ML, then there is little reason to doubt that ML is being used.

gwittel 6y ago

ML and security is tricky. Its a good tool, but shouldn't be the only one. With threats constantly changing, ML alone will fall short. Its great for so called 'easy' stuff or locating 'odd' stuff to prioritize. On top of that you need explainability and reproducibility. Depending on the ML in use, this can be difficult. When a FP or FN occurs, customers want to know what happened and can you fix it. They don't like to be told things like "because the ML said so." Reproducibility is important for support. When things go wrong, they need to know how to right it. If you can't explain or reproduce, its hard to say the misclassification is resolved and won't happen again.