You are right. I often got told "You don't compare with anything" when proposing something very new. That's true, because if you are literally the first one att

You are right. I often got told "You don't compare with anything" when proposing something very new. That's true, because if you are literally the first one attempting a task, there isn't any benchmark. The trick then is to make up at least a straw man alternative to your method and to compare with that. Since then, I have evolved my thinking, and I now use something that isn't just a straw man: Before I even conceive my own method or model or algorithm, I ask myself "What is the simplest non-trivial way to do this?". For example, when tasked with developing a transformer based financial summarization system we pretrained a BERT model from scratch (several months worth of work), but I also implemented a 2-line grep based mini summarizer as a shell script, which defied the complexity of the BERT transformer yet proved to be a competitor tought to beat: https://www.springerprofessional.de/extractive-summarization-of-financial-earnings-call-transcripts/24634460 https://www.springerprofessional.de/extractive-summarization... I'm inclined to organize a workshop on small models with few parameters, and to organize a shared task as part of it where no model can be larger than 65 kB, a sort of "small is beautiful" workshop in dedication of Occam.