7 ms·
Bad numbers in the “gzip beats BERT” paper?
- fnands 3y agoJust a note: your blog seems to be stuck in 2022. Date of post is 17 July 2022
- ks2048 3y agoThanks, should be fixed in a minute. That's what I get for writing dates by hand...
- bjord 3y agoIf this is true, I'm looking forward to seeing how all the people who made grandiose statements about that paper now quietly scrub them. LinkedIn and Twitter influencers, I'm looking at you in particular. If it's not true, I guess I'll be the one looking stupid—I only skimmed the article.
- refulgentis 3y agoReally happy to see this: KNN + classification task + doing classification that's based on pure text similarity is a recipe for stacked results. Schaudenfreude responses to this paper misunderstand that the natural language stuff is crucially important for embeddings: sure, phrases that share words will classify well and GZIP well, so GZIP can be used as ersatz classification. The miracle of BERT / embeddings is _not_ having to share words: for instance, "what is my safe passcode?" has a strong match with "my lockbox pin is 1234", but not "my jewelry is stored safely in the safe". This is also an important thing to consider with LLMs: people are using embeddings intended to do text similarity, whereas you want to use an SBERT model: that is trained to correlate a question to a document that will answer it. https://www.sbert.net/ https://www.sbert.net/ for the full rabbit hole. Previously: Should you use OpenAI's embeddings? Probably not, and here's why. https://iamnotarobot.substack.com/p/should-you-use-openais-embeddings https://iamnotarobot.substack.com/p/should-you-use-openais-e.... HN discussion: https://news.ycombinator.com/item?id=35377935 https://news.ycombinator.com/item?id=35377935
- jabbany 3y ago> The miracle of BERT / embeddings is _not_ having to share words To be fair, the original task is specifically chosen where something like knn+compression has a chance of being good: i.e. out of domain + low resource. Under these conditions the training inputs could be too sparse for a highly parameterized model to learn good embeddings from. In traditional in domain + big data classification settings there's no chance that non-parametric methods like compression would beat a learned representation.
- renewiltord 3y agoGreat job replicating and fixing. So easy to accidentally create results that are statistical artifacts.
- chaxor 3y agoI appreciate that this blog post was made. This is like so, so many little projects that I do (even specifically showing problems in papers like this) that never see the light of day. I usually just make a small noise, and then it sits on my hard drive and that's it. So thank you for putting this out there.
- thomasahle 3y agoI've started using Twitter as a "lower effort blogging" platform. After I spend a day on something like this, I'm usually too tired to actually write a blog post about it, which feels like a waste. But then writing a small Twitter thread is usually within my capabilities.
- AbrahamParangi 3y agoProbably disappointing to the authors but an excellent rebuttal. This is the sort of mistake that's awfully easy to make in ML. The unfortunate thing about the field is that subtle methodological errors often cause subtle failures rather than catastrophic failures as we're used to in many other branches of engineering or science. You can easily make a system slightly worse or slightly better by contaminating your training set with bad data or accidentally leaking some information about your target and the ML system will take it in stride (but with slightly contaminated results). This result makes sense to me because as much as I would like it to be true, applying existing compression algorithms to ML feels like too much of a "free lunch". If there was any special magic happening in compression algorithms we'd use compression algorithms as encoders instead of using transformers as compressors.
- iamflimflam1 3y agoIt's true in many experiments. The desire to get the result you want can often overwhelm the need to validate what you are getting. Especially true when the results confirm any pre-existing thinking you may have.
- mattsan 3y agoYep, confirmation bias. Luckily helped with peer review!
- catgary 3y agoHasn’t this paper made it through peer review?
- thomasahle 3y agoYeah it was published at ACL ( https://aclanthology.org/2023.findings-acl.426/ https://aclanthology.org/2023.findings-acl.426/ ) which is one of the most prestigious conferences in NLP. So kinda disappointing. But paper reviewers are usually not supposed to look at the actual source code of the papers, and definitely don't try to reproduce the results. They just read the paper itself, which of course doesn't talk about the error. Not sure what the best solution is, other than having the most "hyped" papers double verified by researchers on Twitter.
- skrebbel 3y agoCan anyone explain to me how a compression algorithm can beat an LLM at anything? Isn’t that like saying horses are better than graffiti? I’m sure the answer is in there somewhere but I’m not well versed in AI and I simply can’t figure it out.
- ozr 3y agoThe intuition about how the gzip method works goes like so: If you compress `ABC`, it will be X bytes. If you then compress `ABCABC`, it will not take 2x bytes. The more similar the two strings that you concatenate, the less bytes it will take. `ABCABD` will take more than `ABCABC`, but less than `ABCXYZ`. BERT is, by todays standards, a very small LLM, which we know has weaker performance than the billion-param scale models most of us are interacting with today.
- jabbany 3y ago> very small LLM Heh. So does that make it a MLM (medium)? I've always found it funny that we've settled on a term for a class of models that has a size claim... Especially given how fast things are evolving...
- ks2048 3y agoIt's a very limited task: take a document and classify it into one of (for example) 10 or so categories. Things like detecting certain words can do pretty well in some cases. Things that compress well have the occurrence of common substrings.
- kachnuv_ocasek 3y agoOne way to interpret/understand language models is as quite involved compression algorithms.
- refulgentis 3y agoOther reply is great, more in-depth on details from me here: https://news.ycombinator.com/item?id=36758681 https://news.ycombinator.com/item?id=36758681 Plain english TL;DR: - if you limit your task to binning snippets of text - and the snippets are very well-defined (ex. code vs. Filipino text) - the snippets _bytes_ could be compared and score well, no text understanding needed - size delta of a GZIP after adding one more sentence acts as an ersatz way to compare sets of bytes to eachother (ex. you can imagine a GZIP containing 0xFFABCDEF that has 0xFFABCDEF added to it will have a size delta of 0)
- ks2048 3y agoHi, that's my blog post. I'm pretty sure about what I wrote here, but may need the authors to chime-in in case I am missing something. I just submited an issue on github, https://github.com/bazingagin/npc_gzip/issues/3 https://github.com/bazingagin/npc_gzip/issues/3
- lalaland1125 3y agoJust wanted to say, thanks for your work debugging this. You have probably saved other researchers an unfathomable amount of time
- deleted 3y ago[deleted]
- syats 3y agoThanks for the replication, this is important. One question, did you try to replicate the other result table (Table 3)? If I understand correctly, top-2 accuracy would be 1 if you have only 2 classes, but it will differ from "normal" accuracy less and less as the number of classes increases (on average). So this shouldn't change the results for table 3 thaaat much as the datasets have large amounts of classes (see table 1). In any case, top-2 accuracy of 0.685 for the 20-newsgroups dataset is pretty neat for a method that doesn't even consider characters as characters[1], let alone tokens, n-grams, embeddings and all the nice stuff that those of use working on NLP have been devoting years to. [1] In my understanding of gzip, it considers only bit sequences, which are not necessarily aligned with words (aka. bytes).
- ks2048 3y agoI haven't yet replicated Table 3 because most of those datasets are much larger and it will take awhile to run (they said the YahooAnswers database took them 6 days). Also, I have only tried the "gzip" row because that is all that is in the github repo they referenced. Yeah, you're right, the more classes there are, probably the lower the effect this will have.
- cs702 3y ago
- puttycat 3y agoThis is interesting, however, why not first discuss with authors directly to make sure that you're right?
- pinko 3y agoThey did. See: https://github.com/bazingagin/npc_gzip/issues/3 https://github.com/bazingagin/npc_gzip/issues/3
- Dayshine 3y agoOne hour ago?
- ks2048 3y agoYes, that could be a better idea. I am mainly trying something new to work more "in the open" and write blogs about things as a do them. I could be wrong and that would be pretty embarrassing for me. I just published the code I used to double-check things, now linked on the page near the bottom.
- netdur 3y agoI have hacked javascript port https://gist.github.com/netdur/a777f75fb70e0abc19c407c2ff7f9e0c https://gist.github.com/netdur/a777f75fb70e0abc19c407c2ff7f9... and it seems to work!!! regardless Best matched answer for 'How do I start a new project?' is 'To create a new project, go to the dashboard and click on 'New Project'. Then, fill out the details and click 'Save'.' Best matched answer for 'How can I delegate tasks to others in my team?' is 'To invite team members to your project, open the project and click on 'Invite Members'. Enter their email addresses and click 'Send Invites'.' Best matched answer for 'What do I do when I finish a task?' is 'When a task is completed, it should be marked as 'Complete'. It will then be moved to the 'Completed Tasks' section.'
- adamsmith143 3y agoWhats disturbing to me if this is true is the number of "top voices" in ML on ML-Twitter were head over heals with this paper. How many of them actually read it at all?
- snowstormsun 3y agoSo, couldn't it be that the authors of the paper ran the model with a random tie break and got lucky? This blog post seems to assume they had the "rand" flag deactivated. Please correct me if I am wrong.
- expensive_news 3y agoFrom what I understand in the post getting lucky enough to see that big of a change in this situation would be like getting 1000 head flips in a row. It’s not luck you could expect to ever get.
- snowstormsun 3y agoI see
- marcinzm 3y agoThe code of the paper is all on Github so you can verify that line of thinking if you want to.
- abecedarius 3y agoOne thing many people are missing: the simple gzip-for-text-classification hack is not the contribution of this paper. (They reference the standard intro AI textbook for that hack.) The contribution is to use the gzip numbers together with k-nearest-neighbors. In section 6.2 they compare gzip-distance+kNN vs. gzip-distance on its own on four problems: it was better on two, and worse on two others. Another bit of background I guess is worth saying: language models are pretrained with a compression objective. That is, the loss function in pretraining is the cross entropy of the input text, which means "minimize the compressed length of this input if you fed it to this LM driving an arithmetic coder".
- AtNightWeCode 3y agoIn current times *zip mostly cripples things like the web and Docker. To find out it is the best at something in 2023 is not very likely. Nice find btw.
- usgroup 3y agoIt wasn't obvious to me why the authors chose kNN for the classifier. Since they produce a distance matrix, they could have used multi-dimensional scaling to convert the matrix to factors, and then used a tree algorithm such as xgboost which would likely make use of more information than kNN and produce significantly better results. They could have also used a PAQ compression algorithm which are much better than the LZ compressors -- all of which could have significantly improved the results and possibly delivered on their original conclusions. what i liked about the subject paper is that the compression algorithm is abstracted away and it led me to consider what else one could do with compression via the p(x) ~ K^(-|x|) relationship, where K is the alphabet size and |x| is the length of the string x, and assuming optimal coding. For example it occurred to me that one could do traditional classification by packing the factors for each response into separate documents and then proceeding as the paper does to classify which document best compresses the next sample in order to determine its class: a sort of supervised classification using compression algorithms. The closer the compressor is to an optimal code for the dataset, the better it will work. Schemes for sequence prediction are equally straightforward to implement. I found it to be a pleasant surprise.
- softwaredoug 3y agoWhen we did search relevance experimentation at Shopify we made lots of mistakes. I can empathize with the authors. I’ve had a lot of my own public screw ups. At the end of my time at Shopify I learned good science requires good software engineering. It’s really easy to make mistakes at so many places in the stack. We spent a lot of time on creating rigorous, heavily tested and high quality software for our experiments so we could trust our numbers and reproduce each others experiments. We tried to discourage one-off evaluation methods, but if we created a new one, to add it to our suite and test the metric to understand what it meant. It seems obvious, but sadly not as common as I wish it were in my experience with this kind of experimentation. Companies want velocity, and thinking deeply statistically, and building internal tools, is not in the interest of most higher ups.
- thomasahle 3y ago> At the end of my time at Shopify I learned good science requires good software engineering. This is a positive side of industry research. First, you tend to have more software engineering expertise available, and secondly you have more of an insentive to not exaggerate your claims, as if you say it works, you'll be expected to put it into production.
- antonoo 3y agoMy take on this: https://twitter.com/antonosika/status/1679423272541192196?s=46 https://twitter.com/antonosika/status/1679423272541192196?s=... Regardless, great work digging into the code, and great work by authors publishing the code.
- Jimmc414 3y agoI had some difficulty reproducing results that were better than word2vec embeddings + cosine similarity. See my comment on the announcement thread here. https://news.ycombinator.com/item?id=36706078 https://news.ycombinator.com/item?id=36706078
- mercenario 3y agoGreat. So a researcher publishes their code, we can replicate it, find flaws, and correct it. If only every paper were like this. Congrats to the authors for making it easily accessible and the OP for trying to replicate and fix the problem.
- schmaett 3y agoIMO their results have nothing to do with compression: When I replace gzip with lz4 (lower compression ratio, just does the "substring replacement" part of gzip), I get the same performance. When explicitly just comparing the substrings, I still get the same performance. All experiments were done in their 100-shot setting. Code: https://github.com/mattminder/npc_gzip https://github.com/mattminder/npc_gzip (for more details, see the readme)
- mindesc 3y agoGzip as a classifier is surprisingly good and you should use it as a benchmark for your neural network.