6 ms·
A commit history of BERT and its forks
- goose847 6y agoA really interesting take! Makes scientific papers feel more like a tool rather than an article or book. Certainly nice in the case of Computational work. Also allows the potential for long term projects that get incremented on in what would’ve been separate papers. Additionally if new, relevant information comes to light long after a paper has been published, the authors could reference this to give a more complete story.
- amitness 6y agoAgree. This makes a lot of sense for incremental papers.
- hprotagonist 6y ago“shingling” is the (fairly rude) term of art.
- pbhjpbhj 6y agoI don't know how it's rude, shingling means 'many small roof tiles' to me.
- hprotagonist 6y ago'term of art' means that there's a specific meaning inside a particular sphere of discussion for this term (which has normal sense elsewhere). To accuse a research group of shingling means that you think that that group releases a lot of papers one after the other which have a lot of overlap between them and back-cite each other-- and that this is done to artificially boost publication count and citation count to make the group look prominent.
- pbhjpbhj 6y agoCool, thanks, I did a quick search but nothing came up - I assumed it was something scatalogical :o)
- mimixco 6y agoWe are working on a similar concept[0] using a programming language designed for this exact purpose[1]. One of our key ideas is "fact diffing" between two papers with different narrative text. We think this will be useful in all kinds of scientific and academic work. [0] http://mimix.io/recipes http://mimix.io/recipes [1] http://mimix.io/specs http://mimix.io/specs
- nfc 6y agoHi mimixco, I just wrote a comment https://news.ycombinator.com/item?id=23124707 https://news.ycombinator.com/item?id=23124707 about some thoughts I had about this kind of ideas, what do you think?, I'll try to read more about what you are doing as soon as I find some free time :)
- mimixco 6y agoOk, I wrote a little something over there, too. :-) Thanks for looking at our stuff and feel free to email me. My contact info is on our site.
- JBiserkov 6y agoSee https://nextjournal.com/#feature-immutability https://nextjournal.com/#feature-immutability for an example with Immutability on all levels of the system - not just the source code, but also the computations/analyses.
- usrusr 6y agoAlmost, but not quite, entirely unlike this? https://en.wikipedia.org/wiki/Special:History/Wikipedia:No_original_research https://en.wikipedia.org/wiki/Special:History/Wikipedia:No_o...
- RobertoG 6y agoRelated to this, I have this idea for a while that, publishing a paper, could mean to appear as a submission to a Hacker News style forum, where the users are identified members of the scientific community. It would allow a kind of public review process of the paper and it would float the more interested papers for the community to the "first page".
- hobofan 6y agoIn theory for arxiv preprints this exists with arxiv-sanity[0]. In practice nobody is discussing anything there. Same thing for most other preprint sites that have integrated comment sections. [0]: http://www.arxiv-sanity.com http://www.arxiv-sanity.com
- n4r9 6y agoWhen I was doing my PhD there was also SciRate: https://scirate.com/ https://scirate.com/
- slightwinder 6y agoInstead of a Forum, wouldn't a wiki, blog or other type of CMS make more sense? You still can add a comment and interlink-functionality. With a CMS even better, as you can comment directly to a paragraph or even sentence. Or do you mean the river of news-style, the frontpage, maybe also with upvotes, so people can better manage attention to papers?
- MiroF 6y agohttps://openreview.net/ https://openreview.net/
- ivoras 6y agoFirst we need to move away from PDF as a method of distribution of research papers. While it's an immense improvement over the dead-tree medium and images containing scanned paper pages, it's a visual presentation medium instead of a semantic medium. We'll see how long that particular revolution in communication takes. Once / if we have all that information in even a loosely structured but plaintext format (whatever that may be ... JSON? XML? semantic HTML? MHTML? TeX? as long as both images and text can live in the same file), there will be another revolution in accessibility, ease of distribution and accountability. Who knows, we might even stop mentioning GitHub as a best practice for that and use something distributed, blockchain-like. Imagine a future where citations of previous work looks like "As H.G.Wells et all mention in block 4f98aaf1 document #391, the foo is better than the bar." with hyperlinks of course. Peddling my own wares, it can be done with something like https://github.com/ivoras/daisy https://github.com/ivoras/daisy .
- op03 6y agoWikipedia/WikiData seem to have already all solved all these problems. Which is why I think a lot of burnt out scientists spend their time there editing articles.
- ConradKilroy 6y agoLikes for mentioning Wikipedia. (by a Wikipedian)
- bitforger 6y agoI totally agree, but I think the state of a research field is too malleable / moves too quickly to be a Wikipedia article (where people expect things to be "true"). A few weeks ago I spun up wikisurvey.org to try and make a place for Wikipedia-like collaboration on research. There's nothing on there yet, but I'm hoping it will enable low-latency collaboration once we get the ball rolling.
- gumby 6y ago> Imagine a future where citations of previous work looks like "As H.G.Wells et all mention in block 4f98aaf1 document #391, the foo is better than the bar." I dunno, it’s easier for me to describe and to remember “Radin’s 801 paper” or “the ultimate papers” or such.
- nfc 6y agoI've given some thought to this and a related idea. Let's say that for a subset of scientific papers you have the possibility of specifying both the premises and results in a way that can be composed. Let's say for example that your result is that the rate of expansion of the universe is N. Other papers could cite this result let's say through a URL of the result. Other papers could then use this URL as premise for their results and we could create an automatic system that would notify all this papers if the result has changed after new data for any of the elements of the chain. Scientists could be notified that they should revise their own papers to see if their conclusions change with the new data so other papers depending on them can be notified. A paper could be even marked stale if after the change of one of the premises the authors have not confirmed that the conclusions are still valid, this staleness would propagate down the chain A very simplified structure of the data would be sth like this: { premises: ['urlOfResult1', 'urlOfResult2',...], conclusions: [RateOfExpansion: > N] } This is obviously terribly simplified, I suppose it'd take me a lot of time to clearly explain in more detail how such a system could work and I thought about it a long time ago. It could be interesting to apply this to other fields, for example public policy: In this case let's say that we have created this law because of this piece of data. The data changes, we could be notified that perhaps the law could benefit from a new look. { premises: ['urlOfResultLeadIsNotDangerous'], conclusions: ['PeopleCanUseLeadInPipesAsMuchAsTheyWant'] } Such a system could be made even more generic. Probably people have already worked in this kind of systems but I never took the time to investigate it. If someone knows of examples of this kind of systems applied I'd be very happy to know more
- mimixco 6y agoHi, nfc. This is along the lines of the problem we are trying to solve with Mimix[0]. Like your JSON structure, we are using s-expressions to denote facts in text narratives so we can detect when they have changed between papers. Instead of a centralized, URL-based solution, we do the fact comparisons locally, allowing each user to decide what source of facts he or she considers "canonical." The facts and their sources are recorded in that user's writing going forward so the next user has the same access to sources. [0] http://mimix.io http://mimix.io
- 6y ago
- mjw1007 6y agoI wish research papers had a much stronger notion of "effective date". By that, I mean the date to use to interpret any wording like "current" or "recently" or "yet" used inside the paper. For some reason preprints often have no visible date on them at all, and automated datestamps can be misleadingly recent, if someone makes a minor change without rewriting the whole thing.
- activatedgeek 6y agoI think this nature of retro-fitting models used for software engineering into research is ill-targeted. Research is non-linear by nature. One of the reasons research code is usually not well written is because of this same fact - there is no clear plan (or else it wouldn’t be research). Restricting research cultures to follow this linear progress model also imposes an artificial block that may be detrimental to productivity. Alternatively, this exercise could demand a second pass over research code which cleans it up and puts it in its “place”. This may or may not work because “clean up” is a non-trivial task which the researcher may find limited time and utility for. By no means I’m against writing clean code. In fact, I’ve had multiple discussions advocating it but it’s hard to keep everyone aligned. The primary objective of research in a fast-paced field like Machine Learning is to get the idea out the door. Unfortunately, the exceptionally compounding benefits of writing clean thoughtful code from the beginning are realized much later in this timeline. By then, it’s too late and too hard to retro-fix code. My understanding so far (having written code for multiple research projects) is that the only way to fix this culture is to deeply ingrain the compounding benefits of clean code in new researchers. By exposing junior researchers to the compounding benefits of “clean” code, we can gradually nudge the community into a more favorable culture with respect to this “commit history” of research.
- MiroF 6y agoAgreed - and I would say this is part of a larger phenomenon of SWE hubris that expertism in software automatically translates to process expertise in every other domain under the sun.
- activatedgeek 6y agoTo be fair, SWE patterns are insightful. But here's where the problem lies - the proposal in the article is representative of how the community has become so used to this nice side-effect of modularity of neural networks, giving the impression that a clean linear decomposition of contributions is even conceivable. There are examples where this modularity isn't possible (or not optimal even if possible). The poster child, I would say, is probabilistic programming. It is easy to think about building general purpose inference algorithms in probabilistic programming. However, more often than not, inference heavily relies on context. An example for this case is Gaussian Processes. In theory, it is easy to conceive a chain of ideas that connect threads of research within this tiny field. Unfortunately, in practice it turns out inference with GPs is more efficient when we exploit structural assumptions within the context of modeling choices and general purpose few-liner code changes would be terribly inefficient. You'd find the code will look vastly different to an untrained eye even though it would compose similar sounding (or looking) building blocks.
- canjobear 6y agoA meaningful diff summary like this can only be made in retrospect after some time. There are a lot more differences between, say, BERT and XLNet than the ones listed. At the time of publication, it wasn’t yet clear which particular differences were the important ones (and to some extent it is still not clear).
- 6gvONxR4sf7o 6y agoI think this is missing the point of the research. The point of e.g. ALBERT isn't -Next Sentence Prediction +Sentence Order Prediction +Cross-layer Parameter Sharing +Factorized Embeddings The point was what each of those things do, in terms of theory and experimentation. Factorized embeddings are needed because of the portion of memory used in embeddings, NSP isn't useful because it's too easy, etc. Consider a paper like this Smoothness and Stability in GANs [0] from ICLR the other week. You could summarize it as Wassertstein loss + spectral normalization + gradient penalty. Or you could summarize the same work as a generator's convex conjugate + fenchel-moreau duality + L-smoothness + inf-convolutions. Both would be missing the point. Research isn't code. Ideas, their motivation, their demonstration, their explanation, and their testing are represented in the form they are for a reason. Natural language + mathematical language + tables + graphs + references, etc. It's why we aren't having this discussion in terms of diffs versus comments we're replying to. [0] https://iclr.cc/virtual_2020/poster_HJeOekHKwr.html https://iclr.cc/virtual_2020/poster_HJeOekHKwr.html
- amitness 6y agoI've read the ALBERT paper extensively[0] and agree with what you mean. We can't boil down years of research efforts with just 4 bullet points. The intention in the above blog post isn't to suggest that we discard detailed research papers. It's just thinking about additional mediums we can use to make papers more accessible(a good example is distill.pub and paperswithcode). As mentioned in other threads below, PDF can be a limiting medium. [0] https://amitness.com/2020/02/albert-visual-summary/ https://amitness.com/2020/02/albert-visual-summary/
- octbash 6y agoI seem to find myself in the minority, but I don't think distill.pub is a particularly ideal model for publicizing research. distill.pub heavily favors fancy and interactive visualization over actually meaningful research content. This is not to say that the research publicized on distill.pub is not meaningful, but that it is biased to research that can have fancy visualizations. So you end up seeing a lot of tweakable plots, image-augmentations, and attention weights visualizations. It is also further biased towards research groups that have the resources to create a range of D3 plots with sliders, carved out of actual research time. For instance, I don't think BERT could ever make it into a distill.pub post. Despite completely upending the NLP field over the last 2 years, it has no fancy plots, multi-headed self-attention is too messy to visualize, and its setup is dead simple. You could maybe have one gif explaining how masked language modeling works. The best presentation of the significance of BERT is "here is a table of results showing BERT handily beating every other hand-tweaked implementation for every non-generation NLP task we could find with a dead-simple fine-tuning regime, and all it had was masked language modeling." To give another exmaple: I think it's one of the reasons why a lot of junior researchers spend time trying to extract structure from attention and self-attention mechanisms. As someone who's spent some time looking into this topic, you'll find a ton of one-off analysis papers, and next to no insights that actually inform the field (other than super-trivial observations like "tokens tend to attend to themselves and adjacent tokens).
- kinow 6y agoVery interesting and creative way to visualize related papers.