5 ms·

Here's some criticism of AI for you: • VENDOR LOCKIN • It's possible that OpenAI is managing to establish a permanent competitive edge and will be the next Go

Here's some criticism of AI for you: • VENDOR LOCKIN • It's possible that OpenAI is managing to establish a permanent competitive edge and will be the next Google. There is/was a lot of rather handwavy assumptions about open source models being competitive thanks to Stability AI and the Facebook leak, and last time I raised red flags about this on HN I got dunked on. But right now it looks more likely that LLMs are the new search engines and there won't be any competitive implementations that are both legal and that you can run yourself. It'll be APIs all the way, just like with web search engines. Also, embeddings are model/vendor specific and expensive to compute. If you calculate 10 million embeddings using OpenAI it's going to be expensive to recalculate them all with another vendor. • UNCLEAR PRICING • It seems likely that OpenAI is either selling at below cost, or is at best break-even on compute. It's very unclear right now if prices are going to rise, fall or remain where they are and in fact AI prices have done all these things in just a few months. That makes it difficult to know if it's an acceptable business risk to deeply incorporate AI into your workflow. • HUMAN BOTTLENECKS • A lot of AI use cases that sound initially compelling actually bottleneck on human review, because it's too risky to put AI output straight into production. Some people don't care hence the wave of amusing "As an AI language model" spam, but it shows what can go wrong if you skip reviews. • FACTUALITY • Obvious, but after having studied this more and listened to a talk by one of the OpenAI team I think this will actually go away as a problem in the semi-near term future. • INTEGRATION COMPLEXITY • The limited LLM context window size means a lot of tricky workarounds are required for many use cases, which increases implementation complexity. • TESTING DIFFICULTY • LLMs aren't deterministic, so it's a testing nightmare. You never know when the LLM will just do something unexpected that breaks your use case or integration. • BUSINESS UNCERTAINTY • There's genuinely a ton of potential, the tech deserves the hype. But it can be hard to capitalize because so many people are trying to do things all at once, so where do you go that you aren't immediately drowned in VC-flush competition? And many ideas may be more naturally done as features of existing products, so if the vendor isn't doing them, should you wait or should you try developing something yourself and risk obsolesence? The pricing uncertainty compounds that, of course.

chongli 4y ago
after having studied this more and listened to a talk by one of the OpenAI team I think this will actually go away as a problem in the semi-near term future. Is that talk available online? I’m skeptical that they will ever solve the problem of factuality. I’d love to hear their arguments for why I’m wrong.
- mike_hearn 4y ago
  Sure, here you go: https://www.youtube.com/watch?v=hhiLw5Q_UFg https://www.youtube.com/watch?v=hhiLw5Q_UFg Summarizing: 1. Obviously you can't completely "solve" truthfulness because people disagree on what is and is not true. But you can go a long way. 2. The models do know what they don't know. Their level of uncertainty is not only expressed in the final token logprobs but also seems to be reified somehow, such that they can express their own level of certainty in words. 3. Many of the problems are introduced by subtle issues introduced during training which effectively teach the AI to guess. One example is by training on QA data sets where the answer is never "I don't know". RLHF introduces its own problems because the human trainers don't know what the model knows, so may reward it for getting a correct answer by guessing. 4. Many failures are caused by lack of access to information. Giving the LLM tools to let it search the web and access other databases can help a lot, and it's relatively easy to do. 5. In some cases you want it to guess. Like, it's much more useful when coding for it to spit out a screenful of code that has one or two minor errors, than to just refuse to try at all because the result might not be perfect.
  - standyro 4y ago
    This is complete technical hubris. You're effectively saying that the "truth" problem with AI is solved by letting it guess. Which is probably a fine enough answer, but doesn't actually "solve" the problem at all. To the points: 3) Training on data sets where the answer is never "I don't know" is effectively the same as raising children to believe they're always right and teaching them to be needlessly confident. No one should trust those folks. 4) "Lack of access to information" is not the issue... These models are trained on the entire corpus of Twitter, Wikipedia, and more information than I've ever seen in my lifetime, and some of them already do this (Bing) and produce little more than summaries of blog posts based on keywords. If anything, the issue is that an LLM lacks the real world knowledge to discern any nuance as to whether something is correct or grounded in reality.

chongli 4y ago

after having studied this more and listened to a talk by one of the OpenAI team I think this will actually go away as a problem in the semi-near term future. Is that talk available online? I’m skeptical that they will ever solve the problem of factuality. I’d love to hear their arguments for why I’m wrong.

mike_hearn 4y ago
Sure, here you go: https://www.youtube.com/watch?v=hhiLw5Q_UFg https://www.youtube.com/watch?v=hhiLw5Q_UFg Summarizing: 1. Obviously you can't completely "solve" truthfulness because people disagree on what is and is not true. But you can go a long way. 2. The models do know what they don't know. Their level of uncertainty is not only expressed in the final token logprobs but also seems to be reified somehow, such that they can express their own level of certainty in words. 3. Many of the problems are introduced by subtle issues introduced during training which effectively teach the AI to guess. One example is by training on QA data sets where the answer is never "I don't know". RLHF introduces its own problems because the human trainers don't know what the model knows, so may reward it for getting a correct answer by guessing. 4. Many failures are caused by lack of access to information. Giving the LLM tools to let it search the web and access other databases can help a lot, and it's relatively easy to do. 5. In some cases you want it to guess. Like, it's much more useful when coding for it to spit out a screenful of code that has one or two minor errors, than to just refuse to try at all because the result might not be perfect.
- standyro 4y ago
  This is complete technical hubris. You're effectively saying that the "truth" problem with AI is solved by letting it guess. Which is probably a fine enough answer, but doesn't actually "solve" the problem at all. To the points: 3) Training on data sets where the answer is never "I don't know" is effectively the same as raising children to believe they're always right and teaching them to be needlessly confident. No one should trust those folks. 4) "Lack of access to information" is not the issue... These models are trained on the entire corpus of Twitter, Wikipedia, and more information than I've ever seen in my lifetime, and some of them already do this (Bing) and produce little more than summaries of blog posts based on keywords. If anything, the issue is that an LLM lacks the real world knowledge to discern any nuance as to whether something is correct or grounded in reality.