2 ms·
>> Thus when using a benchmark, we should also think about and clarify answers to several related questions: Do improvements on the benchmark correspond to prog
by YeGoblynQueenne 5y ago
>> Thus when using a benchmark, we should also think about and clarify answers
to several related questions: Do improvements on the benchmark correspond to
progress on the original problem? (...) How far will we get by gaming the
benchmark rather than making progress towards solving the original problem?
But what is the "original problem" and how do we measure progress towards
solving it? Obviously there's not just one such problem - each community has a
few of its own.
But in general, the reason that we waste so much time and effort on benchmarks
in AI research (and that's AI in general, not just machine learning this time)
is because nobody can really answer this fundamental question: how do we
measure the progress of AI research?
And that in turn is because AI research is not guided by a scientific theory:
an epistemic object that can explain current and past observations according
to current and past knowledge, and make predictions of future observations. We
do not have such a theory of artificial intelligence. Therefore, we do not
know what we are doing, we do not know where we are going and we do not even
know where we are.
This is the sad, sad state of AI research. If AI research has been reduced,
time and again, to a spectacle, a race to the bottom of pointless benchmarks,
that's because AI research has never stopped to take its bearings, figure out
its goals (there are no commonly accepted goals of AI research) and
establish itself as a science, with a theory - rather than a constantly
shifting trip from demonstration to demonstration. 70 years of demonstrations!
I think the paper above manages to go on about benchmarks for 34 pages and
still miss the real limitation of empirical-only evaluations in a field
without a theoretical basis. That no matter what benchmarks you choose and
how, without a theoretical basis, you'll never know what you're doing.
- hazbot 5y agoI don't think there's not a universal theory of AI for lack of trying - it's just really hard and non obvious. e.g. physics and chemistry took centuries of empirical fooling around to develop the highly predictive theories we have now. I think one of the points of this article is that we need to think deeply about what our empirical foolings around are actually telling us.
- YeGoblynQueenne 5y ago>> I don't think there's not a universal theory of AI for lack of trying - it's just really hard and non obvious. I agree, but in that case most of the work done until now is inconsequential and a waste of everybody's time, and we should probably drop all of it and focus on setting down a solid theoretical foundation for future research. But there is lack of any incentives to do that and there are strong incentives to go on as we are now, from demonstration to demonstration of results and techniques that never get anywhere.