3 ms·
One of the quagmires with systems research is that it makes “stuff” (performance or usability) easier for the user. Research is supposed to be “scientific”, so
by mirker 5y ago
One of the quagmires with systems research is that it makes “stuff” (performance or usability) easier for the user. Research is supposed to be “scientific”, so you need ample evidence of those claims. This is difficult to obtain for very weird systems because usability is in human-computer interaction, which is hard to measure. If it’s so weird to not be portable and you have not measured humans, it must be impractical. That’s the line of thought that I encounter, at least.
The much more publishable paper is on fixing the existing interface and changing the implementation. Look at a paper like Tensorflow; the value add for users is huge, but at the end of the day all the results are on comparing to highly tuned assembly programming. There is some notion that each of the ~millions of ML practitioners that use TF would only use it if it had very good performance. Reality is that most practitioners just want a working implementation. In Pike’s slides, most users are “grandma”, which, in this case, severely distorts relevant metrics.
Also, CS tends to dislike revisiting problems and is seasonal in research topics (see the rise and fall of AI and Big Data frameworks). I’d guess most of this “hot topics” agenda is motivated by possible industrial use. The industrial utility of very new things is near zero (basically what the slides say). You basically see research doing a “I can do that, too” whenever something interesting hits the market.
I think an interesting thought experiment is to take something like databases or distributed systems and simulate inventing them today. It’s unclear how you’d argue for such big ideas. Afterall, you can write database queries by hand and buy bigger computers.
- Ericson2314 5y agoSystems research has become like orthodox economics, doing various marginal empirics but missing the bigger more qualative macro picture. Maybe the megacorp that is going to hire you wants to see some benchmarks improved, but frankly, that's not what the world needs, or even what would best benefit the megacorp? Economics at least has its uncooth grandfather in Political Economy. Computer systems needs to discover it's equivalent, something that is not afraid to go full qualitative and tackle the daemon of accidental complexity head on. There is still great work to be done auditing "supply chains" in dependencies and bootstraps, characterizing bad abstractions and proposing better ones, etc. etc. Per what I wrote the top level comment about capabilities, I think this work can be "practical" in offering realistic incremental improvements and not just more baleful essays like this one, too.
- mirker 5y agoI agree with you, but at the same time, I think it’s easy to dismiss this whole line of thought as quacky. It’s very hard to argue with tables and plots. Economics has a lot of mathematical work for similar “hard science” reasons.
- Ericson2314 5y agoIt's not how academia is structured, but backing up criticism with a protoype of the alternative IMO completely solves the credability problem. From the vantage of the social sciences, where one can only dream of running prototype towns and cities, this is a gift and systems researchers should consider themselves lucky.
- mirker 5y agoWhat if there are many alternatives? Each implementation would be a challenge to optimize fairly. You’d be forced to have some open competition to make it convincing.
- Ericson2314 5y agoI'm confused what you mean? The prototypes need not be performant, they just must be correct and not outrageously naive to demonstrate the better abstractions/layering is working. The questions are: is the code beautiful and dose it work?
- mirker 5y agoAh well that requires a lot of manual inspection. Usually there isn’t enough attention span to do that in practice.
- andrewjl 5y ago> characterizing bad abstractions and proposing better ones Do you know of anything published that tackles this?