6 ms·

The trendline is the only argument. What other predictor of the future is there? Given the available information there is no condition where one would bet aga

• RugnirViking 3y ago
  > What other predictor of the future is there? Typically, experts actually thinking about how technology works, on a deep level, does a pretty good job. Consider for example moore's law. A trendline everyone in the know knew couldn't continue long before it eventually failed. It wasn't a case of "well there are always some naysayers, they're right sometimes", it was anyone with any reasonable experience in building chips knew that each innovation is more hard won than the last, with physical barriers looming. Is AI like that? inevitably. There are invisible physical barriers to all fields and all technologies. The only way to find them is to try. But the discussion here is essentially hypothesising where and when they will show up. We may well be able to run an AGI. Using current techniques, it will need to be trained on a vastly more powerful compute stack than gpt4. Its difficult to impart to you just how big their current one is. They are going to have to mobilise non-trivial segments of entire industries and supply chains just to be big enough for gpt5. There will also be neat tricks that will be found to reduce requirements. But eventually, some wall will be hit, gains will slow. The bet is whether we get to AGI or whatever before then
  • corethree 3y ago
    Right a trendline typically follows a curve before it reaches the apex. If anything based on achievements we've had a speed up in the trendline. We are seeing acceleration. Predicting a limit like in Moore's law means seeing a slow down before we hit that limit. You can make analogies but analogies aren't proof. An analogy to Moore's law ending does not mean it is the same thing happening in AI. You need evidence. I agree that a limit will eventually be hit. That will always be the case but we haven't hit that limit yet. It's only been roughly a year since the release of chatGPT. Additionally compute isn't the main story here. The main story is the algorithm. Improvements in that dimension are likely not at a limit yet such that a more efficient algorithm in the future will need less compute.
    • saltcured 3y ago
      You're begging the question that error rate is a simple metric we can analyze and predict. That there are not other qualitative factors that can vary independently and be more significant for strategic planning. If there's one trend I recognize, it's the near-tautology that increasingly complex systems become increasing complex, as do their failure modes. An accurate predictive model has an expanding cone of unknowns and chaotic risk. Not some curve that paints a clear target. Look beyond today's generative AI fabrication or confabulation (hallucination is a misnomer), where naive users are already prone to taking text outputs as factual rather than fictive. To my eye, it's closely linked to the current "disinformation" cultural phenomena. People are gleefully conflating a flood of low-effort, shallow engagement with real investigation, learning, and knowledge. And tech entrepreneurs have already been exploiting this for decades, pitching products that seem more capable than they are, depending on the charitable interpretation of mass consumers to ignore errors and omissions. How will human participants react if AI get more complex and can exhibit more human-like error modes. Imagine future tools capable of gullibility, delusion, or malice. Seeing passionate, blind faith for LLMs today makes me more worried for that future. I do not expect that AI will effectively replace me in my work. I admit the possibility that the economy could disrupt my employer and hence my career. I worry that our shared socio-technological environment could be poisoned by snake oil application of AI-beyond-its-means, where the disruption could be more negative than positive. And, that upheaval could extend through too much of my remaining lifetime. But, these worries are too abstract to be actionable. To function, I think we have to assume things will continue much as they are now, with some hedging/insurance for the unpredictable. There could just as easily be a new AI winter as the spring you imagine, if the current hype curve finds its asymptote and there is funding backlash against the unfulfilled dreams and promises.
      • corethree 3y ago
        You're right. It is unpredictable. The amount of information available is too complex to fully summarize into a clear and accurate prediction. However the brute force simplistic summary that is analyzable is the trendline. If I had to make a bet: improvement, plateau, or regression I would bet on improvement. Think of it like the weather. Yes the weatherman made a prediction. And yes the chaos surrounding that prediction makes it highly inaccurate. But even so that prediction is still the best one we got. Additionally your comment about complexity was not fully correct. That was the surprising thing. These LLMs weren't even complex. The model is still a feed forward network that is fundamentally much simpler then anticipated. Douglas hofstadter predicted agi would involve neural networks with tons of feedback and recursion and the resulting LLM is much simpler then that. The guy is literally going through a crisis right now because of how wrong he was.