3 ms·
The coolest thing about Machine Assisted Proofs is the idea of expanding collaborators on a single large problem by orders of magnitude by breaking up work into
by npunt 3y ago
The coolest thing about Machine Assisted Proofs is the idea of expanding collaborators on a single large problem by orders of magnitude by breaking up work into small machine-verifiable pieces.
This to me is one of the exciting core capabilities that AI unlocks: the ability to verify/enforce a set of standards across many inputs. It's one of the information age's core problems. One example used a lot in social sciences is inter-rater reliability (IRR) [1]. There's probably a lot of domains out there that can benefit from this pattern of machines verifying distributed human inputs, both in crunch the numbers like machine assisted proofs, as well as more subjective domains where subjectivity can be extremely carefully defined.
I'd love to see more AI tooling focused on these kinds of large scale multi-contributor problem solving methods, including the idea of knowing how to correctly state the overall problem (I believe this was an example in the video).
[1] https://en.wikipedia.org/wiki/Inter-rater_reliability https://en.wikipedia.org/wiki/Inter-rater_reliability
- pests 3y agoHow is that something only AI unlocks?
- npunt 3y agoverifying/enforcing standards on inputs is not something that only AI unlocks, but LLMs are an example of huge step forward in this kind of work and I was expressing my general enthusiasm for the new tools we now have for these sorts of problems. We can turn relatively subjective content into much more computable semi-objective ratings through simple prompting (e.g. rate X on a scale of 0.0 to 1.0), which is valuable for all sorts of use cases. I'm using it right now for my work.
- pests 3y agoMay I ask how you are using it or what you are getting it to rate?
- npunt 3y agoRating content in a dataset to determine if we need to review it as I'm standardizing a large dataset right now to improve user experience. It's a relatively mundane activity but I think this is tentatively a good method. I've been exploring ways to use it for a bunch of ideas, like distance between ideas and reflecting back to users on how their messages might land with others. I think this idea of accelerating self reflection is a big use case for AI.
- maroonblazer 3y agoThis sounds interesting. Are there other examples of this out in the wild? Any plans to blog/write about it? I'm curious to learn more.
- npunt 3y agoI haven't seen any tho I haven't really been looking. Mostly it's messing with prompts and looking at data. I hope to write about it eventually, tho probably in the context of the self reflection use case. I don't think a lot of people are barking up this tree yet and it's important.
- itpnotllm 3y agoTao explicitly mentions that it is ITPs that unlock this. The ITPs will keep the LLMs in check, and stop them from bs-ing. The ITPs unlock massive collaboration.
- npunt 3y agoUm yes ITPs are the unlock for specifically math collaboration. I was referring to other more subjective fields that can also benefit from tech that allows collaboration to scale. LLMs have value for those. For math, LLMs suck.
- kaba0 3y agoI haven’t yet seen the video, nor am I familiar with the topic - could you please tell me what ITPs are? Edit: are they interactive theorem provers?
- chabulhwi 3y agoYes.