3 ms·
These are very serious research level math questions. They are not “Erdős style” questions; they look more like problems or lemmas that I encountered while doin
by _alternator_ 8mo ago
These are very serious research level math questions. They are not “Erdős style” questions; they look more like problems or lemmas that I encountered while doing my PhD. Things that don’t make it into the papers but were part of an interesting diversion along the way.
It seems likely that PhD students in the subfields of the authors are capable of solving these problems. What makes them interesting is that they seem to require fairly high research level context to really make progress.
It’s a test of whether the LLMs can really synthesize results from knowledge that require a human several years of postgraduate preparation in a specific research area.
- clickety_clack 8mo agoSo these are like those problems that are “left for the reader”?
- Jaxan 8mo agoNot necessarily. Even the statements may not appear in the final paper. The questions arose during research, and understanding them was needed for the authors to progress, but maybe not needed for the goal in mind.
- jasonfarnon 8mo agoNo, results in a paper are identified to be "left for the reader" because they are thought to be straightforward to the paper's audience. These are chosen because they are novel. I didn't see any reason to think they are easier than the main results, just maybe not of as much interest.
- data_maan 8mo agoVery serious for mathematicians - not for ML researchers. If the paper would not have had the AI spin, would those 10 questions still have been interesting? It seems to me that we have here a paper that is solely interesting because of the AI spin -- while at the same time this AI spin is really poorly executed from the point of AI research, where this should be a blog post at most, not an arXiv preprint.
- j_maffe 8mo agoThe timed-reveal aspect is also interesting.
- data_maan 8mo agoHow is that interesting for a scientific point of view? This seems more like a social experiment dressed as science. Science should be about reproducibility, and almost nothing here is reproducible.
- thesmtsolver2 8mo agoReproducibility is just one aspect of science, logic + reasoning from principles and data is the major aspect. There are some experiments which cannot be carried out more than once.
- data_maan 8mo ago> There are some experiments which cannot be carried out more than once Yes, in which case a very detailed methodology is required: which hardware, runtimes, token counts etc. This does none of that.
- cowsandmilk 8mo agoDeepmind’s Nobel Prize was primarily for its performance in CASP which is pretty much exactly this. Labs solve structures of proteins, but don’t publish them until after all the computational teams predict structures. So I’m not sure where you’re coming from claiming that this isn’t scientific.
- data_maan 8mo agoIt wasn't like this in any way. CASP relies on a robust benchmark (not just 10 random proteins), and has clear participation criteria, objective metrics how the eval plays out, etc. So I stand by my claim: This isn't scientific. If CASP is Japan, a highly organized & civilized society, this is a banana republic.