11 ms·
Things I don't know about AI
- eladgil 3y agoWould love to get any views / counterpoints on these topics. Eg which AI infra approach thrives relative to the big clouds? What new architectures are needed for agents vs transformers? Etc... Lots of stuff to figure out....
- texas2toss 3y agoDo you think there is the possibility of consumer or end-user apps collecting enough specialized data to move downwards on your graph to infra and foundational models?
- eladgil 3y agoI think this is already true for things based on diffusion models - e.g. Midjourney, Pika etc For LLMs, Character and ChatGPT are arguably two vertically integrated consumer apps (with some B2B applications for ChatGPT as well)
- danielmarkbruce 3y agoThere is a decent (<50%, >20%) chance that frontier foundation models are less oligopoly like than it seems. The reason is that there are so many levers to pull, so much low hanging fruit. For example: * Read the Bloomberg GPT paper - they create their own tokenizer. For specialized domains (finance, law, medicine, etc) the vocabulary is very different and there is likely a lot to do here, where individual tokens really need to map to specific concepts and having a concept capture in several tokens makes it too hard to learn on limited domain data. * Data - so many ways to do different data - more/less, cleaner, "better" on some dimension. * Read the recent papers on different decoding strategies - there seems to be a lot to do here. * Model architecture (SSM etc). If you speak to people who aren't even researchers, they have 10 ideas around architecture and some of them are decent sounding ideas - lots of low hanging fruit. * System architecture - ie likely to see more and more "models" served via API which are actually systems of several model calls, and there is a lot to do here. * Hardware, lower precision etc likely to make training much cheaper It's reasonably likely (again, guessing < 50% > 20%) that this large set of levers to pull become ways to see constant leap-frogging for years and years. Or, at least they become choices/trade-offs rather than strictly "better".
- eladgil 3y agoI agree this is a potential outcome. One big question is generalizability versus niche models. For example, is the best legal model a frontier model + a giant context window + RAG? Or is it a niche model trained or fine tuned for law? Right now at least people seem to decouple some measures of how smart the model is from knowledge base, and at least for now the really big models seem smartest. So part of the question is well is how insightful / synthesis centric the model needs to be versus effectively doing regressions....
- danielmarkbruce 3y agoYup, it's unclear. The current ~consensus is "general purpose frontier model + very sophisticated RAG/system architecture" for legal as an example. I'm building something here using this idea and think its 50/50 (at best) I'm on the right path. It's quite easy to build very clever sounding but often wrong insights into various legal agreements (m&a docs for example). When looking at the tokenization, the training data, decode, architecture (lots of guesses) of the big models, there are a lot of things where the knobs seem turned slightly incorrectly for the domain. Some of the domains are so large that a specialized model might seem niche but the value prop is potentially astronomical.
- CuriouslyC 3y agoFrontier model + rag is good when you need cross-discipline abilities and general knowledge, niche models are best when the domain is somewhat self contained (for instance, if you wanted a model that is amazing at role playing certain types of characters). The future is model graphs with networked mixtures of experts, where models know about other models and can call them as part of recursive prompts, with some sort of online training to tune the weights of the model graph.
- sfink 3y ago> The future is model graphs with networked mixtures of experts, where models know about other models and can call them as part of recursive prompts, with some sort of online training to tune the weights of the model graph. What's the difference between that and combining all of the models into a single model? Aren't you just introducing limitations in communication and training between different parts of that über-model, limitations that may as well be encoded into the single model if they're useful? Are you just partitioning for training performance? Which is a big deal, of course, but it just seems like guessing the right partitioning and communication limitations is not going to be straightforward compared to the usual stupid "throw it all in one big pile and let it work itself out" approach.
- jchonphoenix 3y agoArguably one of the earliest consumer use cases that found footing was AI girlfriend/boyfriend. Large amounts of revenue spread across many small players are generated here but it's glossed over due to the category.
- notpachet 3y agoI think one related area we'll start seeing more of in the future is "resurrected" companions. You have a terminally ill family member, so you train a model on a bunch of video recordings of them, then you can talk to "them" after they've shuffled off this mortal coil.
- swozey 3y agoLicensing my soul as closed source now ..
- datadrivenangel 3y agoDNR: Do Not Retrain
- babyshake 3y agoBe right back.
- rockostrich 3y agoWeird how close your user name is to Joaquin Phoenix, star of the film "Her" centered around an AI girlfriend.
- PeterisP 3y agoGiven that how widespread romance scam schemes already are (the "market" is at least $0.5 billion/year), I would expect any reasonably functioning AI girlfriend/boyfriend model to be massively (ab)used also against unwilling/unwitting "partners".
- doctorpangloss 3y agoI haven't listen to your great podcasts so hard to say what is not covered. Architectures matter a lot less than data. "Knowledge" and "reasoning" in LLMs is a manifestation of instruct-style data. It won't matter how much cheaper training gets if there is limited instruct data for use cases. How do you make 100k context window data for example? Still need thousands of people. Same with so-called niches. Maybe it turns out to be a complex coordination problem to share the data. That's bad for equity investors and giant companies. Anyway, all of this would cost less than the moon landing so it's practicable, you don't need cheaper, you just need risk-taking. The obviousness of the path from here to there means it's not about innovation. It's all about strategy. If Google could marshal $10b for Stadia it could spend $10b on generating 100k context window instruct style data and have the best model. It could also synthesize videos from Unity/Unreal for Sora-style generation. It would just be very hard in an org with 100,000+ people to spend $10b on 10 developers and 10,000 writers compared to 400 developers and 3,600 product managers and other egos. At the end of the day you are revisiting the weaknesses that brought Google and other big companies to this mess in the first place. Anyway I personally think the biggest weakness with ChatGPT and the chat-style UX is that it feels like work. Netflix, TikTok, etc. don't feel like work. Nobody at Google (or OpenAI for that matter) knows how to make stuff that doesn't feel like work. And you can't measure "fun." So the biggest thing to figure out is how much technical stuff matters in a world where people can be poached here and there and walk out with the whole architecture in their heads, versus the non-technical stuff that takes decades of hard-worn personal experiences and strongly held opinions like answers to questions "How do you make AI fun?"
- mywittyname 3y ago> "How do you make AI fun?" Bring back text-based adventure games.
- doctorpangloss 3y agoPeople go for this level of obviousness and it doesn't work. I have no doubt that a meme text game will find some level of literal objective success. But it will still suck. Meme games are a terrible business, both in terms of profits and equity. This also speaks to why OpenAI, Google and the other developers will struggle to create anything that feels like fun: they will chase obvious stuff like this, they will think its similar to all problems. And in reality, you don't need any testing or data or whatever to know that people hate reading in video games, the best video game writing is worse than the average movie's screenplay, that most dialogue is extremely tedious, so why are you going to try to make it even worse by making it generated by an AI?
- 3abiton 3y agoAgents and transformers are kinda different things, built on different layers.
- xianshou 3y agoFrom the article: "It is important to note that the scale of investments being made by these cloud providers is dwarfed by actual cloud revenue. For example, Azure from Microsoft generates $25B in revenue a quarter. The ~$10B OpenAI investment by Microsoft is roughly 6 weeks of Azure revenue. This suggests the cloud business (at least for now) is more important than any one model set for Azure." The most interesting implication in the short term is that the impact of model availability on cloud provider choice is more important than the models themselves. Will organizations choose Azure because it offers GPT-4, or AWS because it offers Claude? This also explains reciprocal VC structures whereby e.g. Amazon invests billions in Anthropic while Anthropic in turn promises to spend billions on AWS cloud resources. Anthropic gets a higher valuation and broader distribution, while AWS gets more customers. It would be a fascinating outcome if the predicted oligopoly in fact ends up as several exclusive partnerships.
- infecto 3y agoDoubtful, at the end of the day the models are just like any other API. No reason not to signup for all the big players (GCP/Azure/AWS) and have access to the various models. Should have no weight as to where you should build out your infra.
- simpletone 3y ago> The ~$10B OpenAI investment by Microsoft is roughly 6 weeks of Azure revenue. Revenue isn't what is important. Profit is since profit is the money you have after costs to invest. But since cloud is highly profitable as it is, the point still stands I suppose. > It would be a fascinating outcome if the predicted oligopoly in fact ends up as several exclusive partnerships. I thought people were predicting independent silos? Where highly profitable and cash rich companies like apple, microsoft, google, facebook, etc own their entire stack or exist as their own ecosystem. Where the likes of apple does everything in-house - AI, cloud, OS, chips, etc. The whole she-bang.
- bbor 3y agoOpened the comments for exactly this. This person seems very nice but also convinced that they’re a business maven, which I think leads them to making “clear” observations like that. Microsoft hasn’t invested a full year of revenue into OpenAI, so it’s not that invested? Meh
- bane 3y agoThe confusion that everybody feels right now about "what do I do in this rapidly evolving space" is likely more an artifact of one of the most vertical lead-edges to a hype curve in a generation. There is so much piling on, at absolutely tremendous investment amounts, that it's not really even possible to pick a launch point to a business plan that would survive even months at this point. The second problem is more obvious, but not talked about in concrete terms yet. As these technologies mature, the legal, policy, and social constructs around them will also mature, and will shape the future of what is coming as much as any technical or business decision. Some examples: - Imagine if courts really find that these models are in gross copyright violation and that model builders will need to work through a scheme similar to sampling in music in order to do their work into the future. Not only will it dramatically increase the cost of model development, but will drive some of the work to alternative sources of data if it's cheaper. What effect will this have on these models? - Suppose these models are found to be reliable enough for certain mission critical work (healthcare, defense, etc.) and they lead a user down a path that results in harm or death? Who's at fault? Suppose it becomes a liability of the company offering the model? How does the insurance market respond? - What if these models become good enough, and the automation around them as well, that we start turning them into fully automated agents that make up a significant part of our economy? Do people still have jobs? Or will these things just become the equivalent of hiring more people into an economy and we work in a blended AI/Human Intelligence world? What happens when these things decide to tank capital markets or choke supply chains? - What about the long-term use of these models by individuals with severe mental impairments or psychological issues? What happens when those models go down, or produce a response that's not liked, or there's a loss of historical conversational data? What about otherwise normal people who end up down a path of social isolation? - There's an entire legal market about to be created for remote-work fraud where somebody just replaces themself with one of these models for 90% of their work. It's sci-fi utopia, but fraudulent if you misrepresent who's doing the work in many cases. - Some company, somewhere, will crack the nut on figuring out how to improve the speed and performance of these models, and build custom hardware that offloads the entire effort onto rapidly commoditizing systems. When multi-modal LLMs and diffusion models come preloaded on disposable $.10 SOCs that are put on cereal boxes and in Birthday cards, we'll be well beyond every one of these questions. But we'll be firmly in a different future. We have yet to encounter most of these things to the point that as a society we have to build responses, and that itself points to the immaturity of both these technologies and the use cases for which they will be adopted. Which cloud to pick and why aren't consumers spending thousands of dollars getting the complex systems in place to use this stuff aren't even the problems that are going to become interesting in the next 10 years.
- cs702 3y agoI would add to the list a few questions about the evolution of cost dynamics going forward, given the advent of new sequence modeling architectures. As we all know, Transformers are very expensive to train and run because their compute cost is quadratic in context length: O(N²). If newer model architectures like RWKV, Mamba, and various others, which incur cost that is linear in context length, O(N), prove as successful as Transformers, the demand for compute for a query of N tokens would decline from O(N²) to O(N). For a sequence with N = 100,000 tokens, it would mean cost dropping by a factor of 100,000×. That's not peanuts! The implications for all market participants would be significant.
- nyrikki 3y agoIn the general case that would be the same as hierarchy collapse of PH, which would prove P=NP
- cs702 3y agoWell, it would depend on how many tokens from the vocabulary are required, in the worst case, to solve NP problems. Think of the tokens as symbols, and of the vocabulary as a finite alphabet of symbols, in a formal system. In other words, no, it would NOT necessarily prove P = NP.
- nyrikki 3y agoThe Singleton case is NP-complete If you consider NP second-order existential logic, meaning that given x, you will get a y, While there may be many true values that aren't y, it will help Attention, which requires feed forward networks, which approximate a DAG, you can think of attention as run time reweighing to change what y value is returned. PH is a query expressible in second-order logic, which would allow for 'for any' at the second level. The reduction in the general case to just logarithmic time: O(n log n), would cause issue, there will not be any movement from quadratic time to linear time without massive new discoveries. It is inherent from the restrictions of ANNs having binary output. Considering FP and FNP vs P and NP may help as they make that property clearer than the concept of second-order existential logic does in some contexts.
- bbor 3y agoThe Google CEO said that this (unexpected!) invention is more important than fire or electricity. In that light, I think it’s arrogant to try to soothsay like this. Which is tough advice cause this is otherwise great analysis, and acknowledging the great unknown feels scary and of dubious value. But I think it’s the only honest way forward
- galaxyLogic 3y agoWould it be possible to start an Open-Source AI project that works like SETI that runs on people's PCs on their spare time and collaborate to train an AI model. People could also help by providing feedback to the models?
- haberdasher 3y agoBasically: https://rendernetwork.com/ https://rendernetwork.com/
- m0llusk 3y agoOr this one: How is AI not outsourcing all over again? People pay almost as much for work that needs to be checked before it can be relied on. Sure, there is value, but there is also a lot of risk and questionable margins depending on the context.
- mr_toad 3y agoOne difference is that we can expect models to improve, become more reliable, cheaper and need less hand-holding.
- RyanShook 3y agoSurely we are in an AI bubble not completely unlike the dotcom bubble, right? The promise and potential for AI to be a long-term growth driver exists but the current market euphoria can’t be sustained in the short-term… at least that’s what I’ve told myself every time NVDA jumps and I feel like a loser.
- causal 3y agoIf it's a bubble, it's early days and not likely to pop soon. If it's a dotcom bubble, that means a road bump for the tech as it continues to take over the world.