7 ms·

Probably disappointing to the authors but an excellent rebuttal. This is the sort of mistake that's awfully easy to make in ML. The unfortunate thing about the

Probably disappointing to the authors but an excellent rebuttal. This is the sort of mistake that's awfully easy to make in ML. The unfortunate thing about the field is that subtle methodological errors often cause subtle failures rather than catastrophic failures as we're used to in many other branches of engineering or science. You can easily make a system slightly worse or slightly better by contaminating your training set with bad data or accidentally leaking some information about your target and the ML system will take it in stride (but with slightly contaminated results). This result makes sense to me because as much as I would like it to be true, applying existing compression algorithms to ML feels like too much of a "free lunch". If there was any special magic happening in compression algorithms we'd use compression algorithms as encoders instead of using transformers as compressors.

iamflimflam1 3y ago
It's true in many experiments. The desire to get the result you want can often overwhelm the need to validate what you are getting. Especially true when the results confirm any pre-existing thinking you may have.
- mattsan 3y ago
  Yep, confirmation bias. Luckily helped with peer review!
  - catgary 3y ago
    Hasn’t this paper made it through peer review?
    - thomasahle 3y ago
      Yeah it was published at ACL ( https://aclanthology.org/2023.findings-acl.426/ https://aclanthology.org/2023.findings-acl.426/ ) which is one of the most prestigious conferences in NLP. So kinda disappointing. But paper reviewers are usually not supposed to look at the actual source code of the papers, and definitely don't try to reproduce the results. They just read the paper itself, which of course doesn't talk about the error. Not sure what the best solution is, other than having the most "hyped" papers double verified by researchers on Twitter.
      - catgary 3y ago
        Yeah, it’s not (entirely) the students’ faults that this slipped through peer review. I don’t envy the whiplash they’re going to experience over the next few weeks. If I was the graduate chair of their department I might schedule a meeting with their supervisor to sort out how this happened.
        westurner 3y ago
        What about the difference in CPU cost, RAM cost, and GPU training hours, though? What about the comparative Big-E's of the models? Great topic: Model minification and algorithmic omplexity
        thomasahle 3y ago
        These are good research topics, but then you really need to be comparing to other models in the same class. The only other super cheap model they compare with is FastText, and FastText beat them quite substantially.
        westurner 3y ago
        Gzip fails at audio and video compression IIRC? Are time-domain signals better compressed with A/V codecs (compression-decompression); and does this gzip finding suggest that other compression algorithms likely to outperform LLMs at certain tasks like computational complexity, at least?
      - godelski 3y ago
        > paper reviewers are usually not supposed to look at the actual source code of the papers Wait what? I haven't reviewed for ACL but most conferences don't say "don't look at the source code." They will say that reviewers are not required to look at it (as well as the appendix). But generally it just isn't uploaded. I do always look at the main method when it is there but talking to my peers and advisor, this is very uncommon[0]. My experience is that most reviewers do not spend more than an hour on a work and make an opinion within 15 minutes. > Not sure what the best solution is, other than having the most "hyped" papers double verified by researchers on Twitter. I'd say (as a start): 1) Get rid of the conference system. A zero-shot (maybe 1-shot if "rebuttal" is allowed) zero-sum system is just disastrous, especially at scale. There's high incentives to actually reject works you review for. A conference system has a binary outcome and the purpose is to reject 80% of papers based on a rather noisy metric of "top tier." A journal system is a back and forth where reviewers are trying to improve the paper. The purpose of the reviewers here is to determine if the idea is indeed good, and then if the paper meets the requirements or not and must explicitly state what needs to be changed for acceptance. 1.5) An actual rebuttal system could help alleviate some of these issues. Using OpenReview for a conversation between authors and reviewers is critical. A singular 1 page response (the norm) is not adequate to respond to 4 different people who often have low similarities in responses. Reviewers are allowed (though breaks guidelines) to respond in one sentence. 2) ACs need to do a better job at validating reviewers. The number of inane and absolutely unacceptable level of reviews I have gotten is astounding (>25%). I've also seen reviewers often break guidelines and have nothing happen. Examples are comments such as those claiming lack of novelty with no explanation or asking authors to compare to concurrent works (I've had this happen for a work that was put out _after_ submission deadlines. Not mine, but here's an example[1] of this being done publicly). If the reviewer is pushed to update their comment then the authors have no ability to respond to their update without the conversation aspect. If there is high variance in response -- not just scores, but what the critiques are about -- then the ACs need to look closer as something is going wrong. We're in a crisis for reviewers but we also have an undisclosed crisis in quality of reviewers. Benchmarkism is on the rise but benchmarks are extremely limiting for evaluation. There's a certain irony given our frequent discussion of Goodhart's Law or Reward Hacking. I'll even make the claim that the quality crisis influences the quantity crisis as I have seen many peers stop reviewing because it isn't worth their time and they aren't getting a fair shot in return. On a personal note, there is a journal I will no longer review for because in-actionable and unreasonable responses, but I also won't submit to them either. 3) Either get rid of double-blind, or actually do it. Everything is published on arxiv these days, which in general is great for the community as it allows things to move fast. But with this it is incredibly easy to de-anonymize authors. Though for big labs, they de-anonymize themselves actively[2]. In a very noisy process even a very slight edge becomes a significant edge[3]. These biases can even come unconsciously given that we're all reading arxiv papers constantly and it isn't unlikely that we come across some of the works we end up reviewing (yet to knowingly happen to me fwiw). But certain labs do have keywords that they use that can be identified. I think one of the major problems comes down to this: in a small community we have a certain level of accountability, as we all end up knowing one another through minimal connections. But in a large community there is little to no accountability and what depends on good faith can no longer be trusted. This encourages bad actors, especially when the system is highly competitive (see 1)), and creates bad science/evaluation creep. (e.g. now standard to tune HPs on test data results -- this is information leakage. If you don't, you likely can't compete). ====== [0] Here's a prominent researcher explicitly saying they don't read the appendix, calling it trash, and a poll showing most people don't look at it https://twitter.com/david_picard/status/1660293648796340226 https://twitter.com/david_picard/status/1660293648796340226 [1] Here's a prominent researcher criticizing a paper for "not citing his work". I linked the top response which is telling him the submission date was 2 months prior to his arxiv release. This is someone who published >250 papers vs someone with <50. For added reference, paper 2 (prominent researcher) was _published_ June 26th in TMLR, but they did cite the other work (gotta give credit for that) https://twitter.com/RinonGal/status/1667943354670170118 https://twitter.com/RinonGal/status/1667943354670170118 [2] We have 2 scenarios here: either reviewers do not know Chinchila == DeepMind, where I'd argue that they are unfit for reviewing given the prominence of that model or 2) they do know, and thus know this is a DeepMind work, and we have an ethics problem. Neither sound great. https://openreview.net/forum?id=OpzV3lp3IMC&noteId=HXmrWV3lnh https://openreview.net/forum?id=OpzV3lp3IMC&noteId=HXmrWV3ln... [3] The conclusion in this analysis of consistency experiment is that even a small amount of inconsistency leads to a lot of noise given a highly selective standard. Which means that paper acceptance itself is highly stochastic: (2014 experiment) https://inverseprobability.com/talks/notes/the-neurips-experiment-snsf.html https://inverseprobability.com/talks/notes/the-neurips-exper... [3.1] A shorter version: https://blog.mrtz.org/2014/12/15/the-nips-experiment.html https://blog.mrtz.org/2014/12/15/the-nips-experiment.html [3.2] A follow-up on the 2014 experiment tdlr: reviewers are good at identifying bad papers, but not good at identifying good papers (i.e. bias to reject): https://arxiv.org/abs/2109.09774 https://arxiv.org/abs/2109.09774 [3.3] A follow-up 2021 experiment (consistent with 2014 experiment): https://blog.neurips.cc/2021/12/08/the-neurips-2021-consistency-experiment/ https://blog.neurips.cc/2021/12/08/the-neurips-2021-consiste... [3.4] Video form https://www.youtube.com/watch?v=19Q-vMd9bYg https://www.youtube.com/watch?v=19Q-vMd9bYg
    - Karellen 3y ago
      I suspect GP commenter meant "replication study" rather than "peer review". ;-) (Peer review doesn't check if your data is correct. They check your data collection methods make sense given the hypothesis you're testing, and that your conclusions are supported by the data you collected.)
- mananaysiempre 3y ago
  One particular example that I remember from an introductory particle physics class is the History Plots section[1] of the biennial review of experimental data. Knowing these quantities is important, but their particular values largely aren’t; nobody’s funding or career really depended on them being equal to one thing or another. Yet look at all the jumps, where the measurements after the initial very rough ones got stuck in the completely wrong place until the jump to the right value—when it happened—was of a completely implausible magnitude, like four, six, or ten sigma. [1] https://pdg.lbl.gov/2023/reviews/rpp2022-rev-history-plots.pdf https://pdg.lbl.gov/2023/reviews/rpp2022-rev-history-plots.p...
  - godelski 3y ago
    What's also good to see here is that the post '90 numbers usually don't even fall within the error bars of the pre '90 numbers. While uncertainty is great, it isn't the end all. I think a lot of people forget how difficult evaluation actually is. Usually we just look at one or two metrics and judge based on that, but such an evaluation is incredibly naive. Metrics and measures are only guides, they do not provide certainty nor targets.
BSEdlMMldESB 3y ago
now shift fields such that the subtle methodological errors don't come to light in 20 years. which field are you on now? economics!? haahha
godelski 3y ago
> This is the sort of mistake that's awfully easy to make in ML. It is important to remember this! Mistakes are common because they are easy to make. Science is a noisy process, but there is signal there and what we see here is exactly what peer review is about. I tend to argue that open publications are a better form of peer review than conferences/journals because of exactly this. Peer review is about your peers reviewing your work, less about whatever random and noisy standard a conference/journal puts forward. Remember that this was the way things happened for most of our history and that our modern notion of peer review is very recent (mid 70's). Older journals were more about accomplishing the mission that arxiv accomplishes today: disseminating works. https://mitcommlab.mit.edu/broad/commkit/peer-review-a-historical-perspective/ https://mitcommlab.mit.edu/broad/commkit/peer-review-a-histo... [side note] another reason I'd advocate for the abolishment of conferences/journals is that through this we can actively advocate for reproduction papers, failure papers, and many other important aspects since we would not be held to the "novelty" criteria (almost everything is incremental). "Publishing" is about communicating your work to your peers and having them validate or invalidate your results. [edit] I think conferences are good in the fact that they bring people together and that encourages collaboration. That's great. But I should clarify that I'm specifically talking about using these platforms as a means to judge the validity of works. If a conference system wants to just invite works and the community, then I'm totally cool with that. I do also like journals in theory given that there's a conversation happening between authors and reviewers, but I believe this also could just easily be accomplished through arxiv + github or OpenReview (preferred).
_ea1k 3y ago
> The unfortunate thing about the field is that subtle methodological errors often cause subtle failures rather than catastrophic failures as we're used to in many other branches of engineering or science. I've been doing a lot of studying in the ML field lately, and I'm seeing this a lot. It is just another thing that feels like the polar opposite of everything else that I've done as a software engineer. Miss a semicolon? Instant error. Miscalculate some grads on one out of three layers? Every now and then it might even work! But the results will be weird.
- godelski 3y ago
  How about this one: tune your hyper-parameters based on the results on your test data. This is prolific, even the norm, but it is a form of information leakage. You're passing information about the test dataset to the model. The solution to this is to use 3 partitions: train, validation, test. Validation is for HP tuning (you can do cross-validation btw) and test is a single shot.
  - _ea1k 3y ago
    Yep, I've been guilty of that one lately. That and solving problems by simply overfitting a neural net to the data in the problem domain. I mean, it works, but the result is less interesting than what I should have done. :)
    - godelski 3y ago
      Definitely. Problem is that doing this helps you get published, not hurts. I think this is why there's often confusion when industry tries to use academic models, as they don't generalize well due to this overfitting. But also, evaluation is fucking hard, and there's just no way around that. Trying to make it easy (i.e. benchmarkism) just adds up creating more noise instead of the intended decrease.
    - eyegor 3y ago
      What about: add more dropout or noise layers and train an ensemble of models. Submit the best one. Is this considered dirty?
TX81Z 3y ago
Academic research code is largely dogshit written as quickly as possible by amateurs, barely tested whatsoever, and the primary intended output of all such code is accumulating paper citations. A world with half as many scientific papers and twice as much care would produce far more value but the whole enterprise is hopelessly gamified.
achileas 3y ago
Having worked in other sciences (neuroscience for me), I’m not sure what catastrophic obvious errors you’re used to seeing. The vast majority IME are like this, except with even longer feedback loops (on the order of several months).
- AbrahamParangi 3y ago
  Well, biology is probably a lot closer to ML in this way. My experience in chemistry or material science is that 99% of the time if you do anything wrong it totally doesn't work at all. This is fairly typical in software as well.
mindesc 3y ago
Those are used. Search for minimum description principle and entropy based classifier. The performance is poor, but it is definitely there and really easy to deploy. I have seen gzip being used for plagiarism detection as similar text tends to compress better. Use the compression ratio as weights on spring model then for visualisation. Also works with network communication metadata ...

iamflimflam1 3y ago

It's true in many experiments. The desire to get the result you want can often overwhelm the need to validate what you are getting. Especially true when the results confirm any pre-existing thinking you may have.

mattsan 3y ago
Yep, confirmation bias. Luckily helped with peer review!
- catgary 3y ago
  Hasn’t this paper made it through peer review?
  - thomasahle 3y ago
    Yeah it was published at ACL ( https://aclanthology.org/2023.findings-acl.426/ https://aclanthology.org/2023.findings-acl.426/ ) which is one of the most prestigious conferences in NLP. So kinda disappointing. But paper reviewers are usually not supposed to look at the actual source code of the papers, and definitely don't try to reproduce the results. They just read the paper itself, which of course doesn't talk about the error. Not sure what the best solution is, other than having the most "hyped" papers double verified by researchers on Twitter.
    - catgary 3y ago
      Yeah, it’s not (entirely) the students’ faults that this slipped through peer review. I don’t envy the whiplash they’re going to experience over the next few weeks. If I was the graduate chair of their department I might schedule a meeting with their supervisor to sort out how this happened.
      - westurner 3y ago
        What about the difference in CPU cost, RAM cost, and GPU training hours, though? What about the comparative Big-E's of the models? Great topic: Model minification and algorithmic omplexity
        thomasahle 3y ago
        These are good research topics, but then you really need to be comparing to other models in the same class. The only other super cheap model they compare with is FastText, and FastText beat them quite substantially.
        westurner 3y ago
        Gzip fails at audio and video compression IIRC? Are time-domain signals better compressed with A/V codecs (compression-decompression); and does this gzip finding suggest that other compression algorithms likely to outperform LLMs at certain tasks like computational complexity, at least?
    - godelski 3y ago
      > paper reviewers are usually not supposed to look at the actual source code of the papers Wait what? I haven't reviewed for ACL but most conferences don't say "don't look at the source code." They will say that reviewers are not required to look at it (as well as the appendix). But generally it just isn't uploaded. I do always look at the main method when it is there but talking to my peers and advisor, this is very uncommon[0]. My experience is that most reviewers do not spend more than an hour on a work and make an opinion within 15 minutes. > Not sure what the best solution is, other than having the most "hyped" papers double verified by researchers on Twitter. I'd say (as a start): 1) Get rid of the conference system. A zero-shot (maybe 1-shot if "rebuttal" is allowed) zero-sum system is just disastrous, especially at scale. There's high incentives to actually reject works you review for. A conference system has a binary outcome and the purpose is to reject 80% of papers based on a rather noisy metric of "top tier." A journal system is a back and forth where reviewers are trying to improve the paper. The purpose of the reviewers here is to determine if the idea is indeed good, and then if the paper meets the requirements or not and must explicitly state what needs to be changed for acceptance. 1.5) An actual rebuttal system could help alleviate some of these issues. Using OpenReview for a conversation between authors and reviewers is critical. A singular 1 page response (the norm) is not adequate to respond to 4 different people who often have low similarities in responses. Reviewers are allowed (though breaks guidelines) to respond in one sentence. 2) ACs need to do a better job at validating reviewers. The number of inane and absolutely unacceptable level of reviews I have gotten is astounding (>25%). I've also seen reviewers often break guidelines and have nothing happen. Examples are comments such as those claiming lack of novelty with no explanation or asking authors to compare to concurrent works (I've had this happen for a work that was put out _after_ submission deadlines. Not mine, but here's an example[1] of this being done publicly). If the reviewer is pushed to update their comment then the authors have no ability to respond to their update without the conversation aspect. If there is high variance in response -- not just scores, but what the critiques are about -- then the ACs need to look closer as something is going wrong. We're in a crisis for reviewers but we also have an undisclosed crisis in quality of reviewers. Benchmarkism is on the rise but benchmarks are extremely limiting for evaluation. There's a certain irony given our frequent discussion of Goodhart's Law or Reward Hacking. I'll even make the claim that the quality crisis influences the quantity crisis as I have seen many peers stop reviewing because it isn't worth their time and they aren't getting a fair shot in return. On a personal note, there is a journal I will no longer review for because in-actionable and unreasonable responses, but I also won't submit to them either. 3) Either get rid of double-blind, or actually do it. Everything is published on arxiv these days, which in general is great for the community as it allows things to move fast. But with this it is incredibly easy to de-anonymize authors. Though for big labs, they de-anonymize themselves actively[2]. In a very noisy process even a very slight edge becomes a significant edge[3]. These biases can even come unconsciously given that we're all reading arxiv papers constantly and it isn't unlikely that we come across some of the works we end up reviewing (yet to knowingly happen to me fwiw). But certain labs do have keywords that they use that can be identified. I think one of the major problems comes down to this: in a small community we have a certain level of accountability, as we all end up knowing one another through minimal connections. But in a large community there is little to no accountability and what depends on good faith can no longer be trusted. This encourages bad actors, especially when the system is highly competitive (see 1)), and creates bad science/evaluation creep. (e.g. now standard to tune HPs on test data results -- this is information leakage. If you don't, you likely can't compete). ====== [0] Here's a prominent researcher explicitly saying they don't read the appendix, calling it trash, and a poll showing most people don't look at it https://twitter.com/david_picard/status/1660293648796340226 https://twitter.com/david_picard/status/1660293648796340226 [1] Here's a prominent researcher criticizing a paper for "not citing his work". I linked the top response which is telling him the submission date was 2 months prior to his arxiv release. This is someone who published >250 papers vs someone with <50. For added reference, paper 2 (prominent researcher) was _published_ June 26th in TMLR, but they did cite the other work (gotta give credit for that) https://twitter.com/RinonGal/status/1667943354670170118 https://twitter.com/RinonGal/status/1667943354670170118 [2] We have 2 scenarios here: either reviewers do not know Chinchila == DeepMind, where I'd argue that they are unfit for reviewing given the prominence of that model or 2) they do know, and thus know this is a DeepMind work, and we have an ethics problem. Neither sound great. https://openreview.net/forum?id=OpzV3lp3IMC&noteId=HXmrWV3lnh https://openreview.net/forum?id=OpzV3lp3IMC&noteId=HXmrWV3ln... [3] The conclusion in this analysis of consistency experiment is that even a small amount of inconsistency leads to a lot of noise given a highly selective standard. Which means that paper acceptance itself is highly stochastic: (2014 experiment) https://inverseprobability.com/talks/notes/the-neurips-experiment-snsf.html https://inverseprobability.com/talks/notes/the-neurips-exper... [3.1] A shorter version: https://blog.mrtz.org/2014/12/15/the-nips-experiment.html https://blog.mrtz.org/2014/12/15/the-nips-experiment.html [3.2] A follow-up on the 2014 experiment tdlr: reviewers are good at identifying bad papers, but not good at identifying good papers (i.e. bias to reject): https://arxiv.org/abs/2109.09774 https://arxiv.org/abs/2109.09774 [3.3] A follow-up 2021 experiment (consistent with 2014 experiment): https://blog.neurips.cc/2021/12/08/the-neurips-2021-consistency-experiment/ https://blog.neurips.cc/2021/12/08/the-neurips-2021-consiste... [3.4] Video form https://www.youtube.com/watch?v=19Q-vMd9bYg https://www.youtube.com/watch?v=19Q-vMd9bYg
  - Karellen 3y ago
    I suspect GP commenter meant "replication study" rather than "peer review". ;-) (Peer review doesn't check if your data is correct. They check your data collection methods make sense given the hypothesis you're testing, and that your conclusions are supported by the data you collected.)
mananaysiempre 3y ago
One particular example that I remember from an introductory particle physics class is the History Plots section[1] of the biennial review of experimental data. Knowing these quantities is important, but their particular values largely aren’t; nobody’s funding or career really depended on them being equal to one thing or another. Yet look at all the jumps, where the measurements after the initial very rough ones got stuck in the completely wrong place until the jump to the right value—when it happened—was of a completely implausible magnitude, like four, six, or ten sigma. [1] https://pdg.lbl.gov/2023/reviews/rpp2022-rev-history-plots.pdf https://pdg.lbl.gov/2023/reviews/rpp2022-rev-history-plots.p...
- godelski 3y ago
  What's also good to see here is that the post '90 numbers usually don't even fall within the error bars of the pre '90 numbers. While uncertainty is great, it isn't the end all. I think a lot of people forget how difficult evaluation actually is. Usually we just look at one or two metrics and judge based on that, but such an evaluation is incredibly naive. Metrics and measures are only guides, they do not provide certainty nor targets.

BSEdlMMldESB 3y ago

now shift fields such that the subtle methodological errors don't come to light in 20 years. which field are you on now? economics!? haahha

godelski 3y ago

> This is the sort of mistake that's awfully easy to make in ML. It is important to remember this! Mistakes are common because they are easy to make. Science is a noisy process, but there is signal there and what we see here is exactly what peer review is about. I tend to argue that open publications are a better form of peer review than conferences/journals because of exactly this. Peer review is about your peers reviewing your work, less about whatever random and noisy standard a conference/journal puts forward. Remember that this was the way things happened for most of our history and that our modern notion of peer review is very recent (mid 70's). Older journals were more about accomplishing the mission that arxiv accomplishes today: disseminating works. https://mitcommlab.mit.edu/broad/commkit/peer-review-a-historical-perspective/ https://mitcommlab.mit.edu/broad/commkit/peer-review-a-histo... [side note] another reason I'd advocate for the abolishment of conferences/journals is that through this we can actively advocate for reproduction papers, failure papers, and many other important aspects since we would not be held to the "novelty" criteria (almost everything is incremental). "Publishing" is about communicating your work to your peers and having them validate or invalidate your results. [edit] I think conferences are good in the fact that they bring people together and that encourages collaboration. That's great. But I should clarify that I'm specifically talking about using these platforms as a means to judge the validity of works. If a conference system wants to just invite works and the community, then I'm totally cool with that. I do also like journals in theory given that there's a conversation happening between authors and reviewers, but I believe this also could just easily be accomplished through arxiv + github or OpenReview (preferred).

_ea1k 3y ago

> The unfortunate thing about the field is that subtle methodological errors often cause subtle failures rather than catastrophic failures as we're used to in many other branches of engineering or science. I've been doing a lot of studying in the ML field lately, and I'm seeing this a lot. It is just another thing that feels like the polar opposite of everything else that I've done as a software engineer. Miss a semicolon? Instant error. Miscalculate some grads on one out of three layers? Every now and then it might even work! But the results will be weird.

godelski 3y ago
How about this one: tune your hyper-parameters based on the results on your test data. This is prolific, even the norm, but it is a form of information leakage. You're passing information about the test dataset to the model. The solution to this is to use 3 partitions: train, validation, test. Validation is for HP tuning (you can do cross-validation btw) and test is a single shot.
- _ea1k 3y ago
  Yep, I've been guilty of that one lately. That and solving problems by simply overfitting a neural net to the data in the problem domain. I mean, it works, but the result is less interesting than what I should have done. :)
  - godelski 3y ago
    Definitely. Problem is that doing this helps you get published, not hurts. I think this is why there's often confusion when industry tries to use academic models, as they don't generalize well due to this overfitting. But also, evaluation is fucking hard, and there's just no way around that. Trying to make it easy (i.e. benchmarkism) just adds up creating more noise instead of the intended decrease.
  - eyegor 3y ago
    What about: add more dropout or noise layers and train an ensemble of models. Submit the best one. Is this considered dirty?

TX81Z 3y ago

Academic research code is largely dogshit written as quickly as possible by amateurs, barely tested whatsoever, and the primary intended output of all such code is accumulating paper citations. A world with half as many scientific papers and twice as much care would produce far more value but the whole enterprise is hopelessly gamified.

achileas 3y ago

Having worked in other sciences (neuroscience for me), I’m not sure what catastrophic obvious errors you’re used to seeing. The vast majority IME are like this, except with even longer feedback loops (on the order of several months).

AbrahamParangi 3y ago
Well, biology is probably a lot closer to ML in this way. My experience in chemistry or material science is that 99% of the time if you do anything wrong it totally doesn't work at all. This is fairly typical in software as well.

mindesc 3y ago

Those are used. Search for minimum description principle and entropy based classifier. The performance is poor, but it is definitely there and really easy to deploy. I have seen gzip being used for plagiarism detection as similar text tends to compress better. Use the compression ratio as weights on spring model then for visualisation. Also works with network communication metadata ...