3 ms·

I'm not the person you asked, but I'm not sure I understand your question and I'd like to. It whiffed multiple common softballs, to the point it brings into que

by jdiff 3y ago

I'm not the person you asked, but I'm not sure I understand your question and I'd like to. It whiffed multiple common softballs, to the point it brings into question the claims made about its performance. What reasoning is there to trust it?

michaelmior 3y ago
> It whiffed multiple common softballs I must have missed this in the article. Where was this?
- deleted 3y ago
  [deleted]
- jdiff 3y ago
  ...It's in the comment you were responding to. Directly above the section you quoted.
  - michaelmior 3y ago
    I understand that, but it wasn't clear to me where those examples came from.
    - jdiff 3y ago
      It's pretty obvious from the whole comment that they're his own experience. Are you going anywhere with this or are you just saying things?
EnigmaFlare 3y ago
It had 3 failures. How is that a sign it's untrustworthy? I'm sure all alternatives have more than 3 failures. You might be making assumptions about the distribution of successes and failures (GP didn't say how many files they tested to find those 3) or how "soft" they were. In an extreme case, they might even have been crafted adversarial examples. But even if not, they might have features that really do look more like some other file type from the point of view of the classifier even if it's not easily apparent to a human. Being strictly superior to a competent human is a pretty high bar to set.
- epcoa 3y ago
  > or how "soft" they were. From the comment: It identified some simple HTML files (html, head, title, body, p tags and not much else) as "MS Visual Basic source (VBA)", "ASP source (code)", and "Generic text document" where the `file` utility correctly identified all such examples as "HTML document text". That's pretty soft. Nothing "adversarial" claimed either. > Being strictly superior to a competent human is a pretty high bar to set. The bar is the file utility.
  - EnigmaFlare 3y ago
    Those are only soft to a human. I looked at a couple and I picked them correctly but I don't know what details the classifier was seeing which I was blind to. Not to say it was correct, just that we can't call them soft just because they're short and easy for a human. > The bar is the file utility. It has higher accuracy than that. You would reject it just because the failures are different even though they're less?
    - jdiff 3y ago
      Yes. Unpredictable failures are significantly worse than predictable ones. If file messes up, it's because it decided a ZIP-based document was a generic ZIP file. If Magika messes up, it's entirely random. I can work around file's failure modes, especially if it's one I work with often. Magika's failure modes strike at random and are not possible to anticipate. File also bails out when it doesn't know, a very common failure mode in Magika is that it confidently returns a random answer when it wasn't trained on a file type.
      - EnigmaFlare 3y ago
        Your original statement was that having a couple of failures brings into question its claims about performance. It doesn't because it doesn't claim such high performance. 99.31% is lower than perhaps 997 out of 1000 or whatever the GP tested. Of course having unpredictable failures is a worry but it's a different worry.

by jdiff 3y ago

michaelmior 3y ago

> It whiffed multiple common softballs I must have missed this in the article. Where was this?

deleted 3y ago
[deleted]
jdiff 3y ago
...It's in the comment you were responding to. Directly above the section you quoted.
- michaelmior 3y ago
  I understand that, but it wasn't clear to me where those examples came from.
  - jdiff 3y ago
    It's pretty obvious from the whole comment that they're his own experience. Are you going anywhere with this or are you just saying things?

EnigmaFlare 3y ago

It had 3 failures. How is that a sign it's untrustworthy? I'm sure all alternatives have more than 3 failures. You might be making assumptions about the distribution of successes and failures (GP didn't say how many files they tested to find those 3) or how "soft" they were. In an extreme case, they might even have been crafted adversarial examples. But even if not, they might have features that really do look more like some other file type from the point of view of the classifier even if it's not easily apparent to a human. Being strictly superior to a competent human is a pretty high bar to set.

epcoa 3y ago
> or how "soft" they were. From the comment: It identified some simple HTML files (html, head, title, body, p tags and not much else) as "MS Visual Basic source (VBA)", "ASP source (code)", and "Generic text document" where the `file` utility correctly identified all such examples as "HTML document text". That's pretty soft. Nothing "adversarial" claimed either. > Being strictly superior to a competent human is a pretty high bar to set. The bar is the file utility.
- EnigmaFlare 3y ago
  Those are only soft to a human. I looked at a couple and I picked them correctly but I don't know what details the classifier was seeing which I was blind to. Not to say it was correct, just that we can't call them soft just because they're short and easy for a human. > The bar is the file utility. It has higher accuracy than that. You would reject it just because the failures are different even though they're less?
  - jdiff 3y ago
    Yes. Unpredictable failures are significantly worse than predictable ones. If file messes up, it's because it decided a ZIP-based document was a generic ZIP file. If Magika messes up, it's entirely random. I can work around file's failure modes, especially if it's one I work with often. Magika's failure modes strike at random and are not possible to anticipate. File also bails out when it doesn't know, a very common failure mode in Magika is that it confidently returns a random answer when it wasn't trained on a file type.
    - EnigmaFlare 3y ago
      Your original statement was that having a couple of failures brings into question its claims about performance. It doesn't because it doesn't claim such high performance. 99.31% is lower than perhaps 997 out of 1000 or whatever the GP tested. Of course having unpredictable failures is a worry but it's a different worry.