10 ms·
Chain of Thought empowers transformers to solve inherently serial problems
- nopinsight 2y agoIn the words of an author: "What is the performance limit when scaling LLM inference? Sky's the limit. We have mathematically proven that transformers can solve any problem, provided they are allowed to generate as many intermediate reasoning tokens as needed. Remarkably, constant depth is sufficient. http://arxiv.org/abs/2402.12875 http://arxiv.org/abs/2402.12875 (ICLR 2024)" https://x.com/denny_zhou/status/1835761801453306089 https://x.com/denny_zhou/status/1835761801453306089
- candiddevmike 2y ago> We have mathematically proven that transformers can solve any problem, provided they are allowed to generate as many intermediate reasoning tokens as needed. That seems like a bit of a leap here to make this seem more impressive than it is (IMO). You can say the same thing about humans, provided they are allowed to think across as many years/generations as needed. Wake me up when a LLM figures out stable fusion or room temperature superconductors.
- WalterSear 2y ago> You can say the same thing about humans 1. Holy shit. 2. You can't apply Moore's law to humans.
- Tostino 2y agoYou can't to chips any more either. Density has continued to increase, but so have prices. The 'law' was tied to the price to density ratio, and it's been almost a decade now since it died.
- gryn 2y ago> 2. You can't apply Moore's law to humans. not with that attitude. /s if you take reproduction into account and ignore all the related externalities you can definitely double your count of transistors (humans) every two years.
- Horffupolde 2y agoIt is actually impressive. One could argue that writing enabled chain of thought across generations.
- krackers 2y agoI think you're misrepresenting the study. It builds upon previous work that examines the computation power of the transformer architecture from a circuit-complexity perspective. Previous work showed that the class of problems that a "naive" Transformer architecture could compute was within TC0 [1, 2] and as a consequence it was fundamentally impossible for transformers to solve certain classes of mathematical problems. This study actually provides a more realistic bound of AC0 (by analyzing the finite-precision case) which rules out even more problems, including such 'simple' ones as modular parity. We also had previous work that hinted that part of the reason why chain-of-thought works from a theoretical perspective is that it literally allows the model to perform types of computations it could not under the more limited setting (in the same way jumping from FSMs to pushdown automata allows you to solve new types of problems) [3]. [1] https://news.ycombinator.com/item?id=35609652 https://news.ycombinator.com/item?id=35609652 [2] https://blog.computationalcomplexity.org/2023/02/why-cant-little-chatty-do-math.html https://blog.computationalcomplexity.org/2023/02/why-cant-li... [3] https://arxiv.org/abs/2305.15408 https://arxiv.org/abs/2305.15408
- shawntan 2y agoGenerally, literature on the computational power of the SAME neural architecture can differ on their conclusions based on their premises. Assuming finite precision will give a more restrictive result, and assuming arbitrary precision can give you Turing completeness. From a quick skim this seems like it's making finite precision assumptions? Which doesn't actually tighten previous bounds, it just makes different starting assumptions. Am author of [1].
- krackers 2y agoAh my bad, great catch! I've updated my comment accordingly.
- shawntan 2y agoYou can't really be blamed though, the language in the paper does seem to state what you originally said. Might be a matter of taste but I don't think it's quite accurate. The prior work they referenced actually did account for finite precision cases and why they didn't think it was useful to prove the result with those premises. In this work they simply argued from their own perspective why finite precision made more sense. The whole sub-field is kinda messy and I get quoted differing results all the time. Edit: Also, your original point stands, obviously. Sorry for nitpicking on your post, but I also just thought people should know more about the nuances of this stuff.
- aurareturn 2y ago> You can say the same thing about humans, provided they are allowed to think across as many years/generations as needed. Isn’t this a good thing since compute can be scaled so that the LLM can do generations of human thinking in a much shorter amount of time? Say humans can solve quantum gravity in 100 years of thinking by 10,000 really smart people. If one AGI is equal to 1 really smart person. Scale enough compute for 1 million AGI and we can solve quantum gravity in a year. The major assumption here is that transformers can indeed solve every problem humans can.
- wizzwizz4 2y ago> Isn’t this a good thing since compute and be scaled so that the LLM can do generations of human thinking in a much shorter amount of time? But it can't. There isn't enough planet. > The major assumption here is that transformers can indeed solve every problem humans can. No, the major assumptions are (a) that ChatGPT can, and (b) that we can reduce the resource requirements by many orders of magnitude. The former assumption is highly-dubious, and the latter is plainly false. Transformers are capable of representing any algorithm, if they're allowed to be large enough and run large enough. That doesn't give them any special algorithm-finding ability, and finding the correct algorithms is the hard part of the problem!
- aurareturn 2y ago> But it can't. There isn't enough planet. How much resource are you assuming an AGI would consume?
- wizzwizz4 2y agoAre we talking about "an AGI", or are we talking about overfitting large transformer models with human-written corpora and scaling up the result? "An AGI"? I have no idea what that algorithm might look like. I do know that we can cover the majority of cases with not too much effort, so it all depends on the characteristics of that long tail. ChatGPT-like transformer models? We know what that looks like, despite the AI companies creatively misrepresenting the resource use (ref: https://www.bnnbloomberg.ca/business/technology/2024/08/21/how-tech-companies-are-obscuring-ais-real-carbon-footprint/ https://www.bnnbloomberg.ca/business/technology/2024/08/21/h...). Look at https://arxiv.org/pdf/2404.06405 https://arxiv.org/pdf/2404.06405: > Combining Wu’s method with the classic synthetic methods of deductive databases and angle, ratio, and distance chasing solves 21 out of 30 methods by just using a CPU-only laptop with a time limit of 5 minutes per problem. AlphaGeometry had an entire supercomputer cluster, and dozens of hours. GOFAI approaches have a laptop and five minutes. Scale that inconceivable inefficiency up to AGI, and the total power output of the sun may not be enough.
- whimsicalism 2y agoit's a TCS result. seems like many commenting don't know about computability
- Veedrac 2y ago> Wake me up when a LLM figures out stable fusion or room temperature superconductors. Man, the goalposts these days.
- WalterSear 2y agoShh!! It's working! It's working!
- FeepingCreature 2y ago"I love [goalposts]. I love the whooshing noise they make as they go by." --Douglas Adams, slightly adjusted
- ec109685 2y agoIs this the infinite monkey Shakespeare trope?
- CamperBob2 2y agoYeah. Monkeys. Monkeys that write useful C and Python code that needs a bit less revision every time there's a model update. Can we just give the "stochastic parrot" and "monkeys with typewriters" schtick a rest? It made for novel commentary three or four years ago, but at this point, these posts themselves read like the work of parrots. They are no longer interesting, insightful, or (for that matter) true.
- visarga 2y agoIf you think about it, humans necessarily use abstractions, from the edge detectors in retina to concepts like democracy. But do we really understand? All abstractions leak, and nobody knows the whole stack. For all the poorly grasped abstractions we are using, we are also just parroting. How many times are we doing things because "that is how they are done" never wondering why? Take ML itself, people are saying it's little more than alchemy (stir the pile). Are we just parroting approaches that have worked in practice without real understanding? Is it possible to have centralized understanding, even in principle, or is all understanding distributed among us? My conclusion is that we have a patchwork of partial understanding, stitched together functionally by abstractions. When I go to the doctor, I don't study medicine first, I trust the doctor. Trust takes the place of genuine understanding. So humans, like AI, use distributed and functional understanding, we don't have genuine understanding as meant by philosophers like Searle in the Chinese Room. No single neuron in the brain understands anything, but together they do. Similarly, no single human understands genuinely, but society together manages to function. There is no homunculus, no centralized understander anywhere. We humans are also stochastic parrots of abstractions we don't really grok to the full extent.
- kaechle 2y agoGreat points. We're pattern-matching shortcut machines, without a doubt. In most contexts, not even good ones. > When I go to the doctor, I don't study medicine first, I trust the doctor. Trust takes the place of genuine understanding. The ultimate abstraction! Trust is highly irrational by definition. But we do it all day every day, lest we be classified as psychologically unfit for society. Which is to say, mental health is predicated on a not-insignificant amount of rationalizations and self-deceptions. Hallucinations, even.
- shawntan 2y agoTheoretical results exist that try to quantify the number of CoT tokens needed to reach different levels of computational expressibility: https://arxiv.org/pdf/2310.07923 https://arxiv.org/pdf/2310.07923 TL;DR: Getting to Turing completeness can require polynomial CoT tokens, wrt the input problem size. For a field that constantly harps on parallelism and compute efficiency, this requirement seems prohibitive. We really need to get away from constant depth architectures.
- benkuykendall 2y ago> Getting to Turing completeness can require polynomial CoT tokens, wrt the input problem size. So, as stated, this is impossible since it violates the Time Hierarchy Theorem. The actual result of the paper is that any poly-time computable function can be computed with poly-many tokens. Which is... not a particularly impressive bound? Any non-trivial fixed neural network can, for instance, compute the NAND of two inputs. And any polynomial computable function can be computed with a polynomial number of NAND gates.
- shawntan 2y ago> The actual result of the paper is that any poly-time computable function can be computed with poly-many tokens. You're right. Re: NAND of two inputs. Isn't this doable even by a single layer (no hidden layers) neural network? Re: Polynomial computable function. I'm assuming this makes no assumption of constant-depth. Because my entire point was that the result of this paper is not actually impressive AND covered by a previous paper. Hopefully that's clearer.
- deleted 2y ago[deleted]
- __loam 2y ago> We have mathematically proven that transformers can solve any problem We should require that you've passed an algorithms and a thermodynamics class before you can post.
- nopinsight 2y agoTo be clear I think the tweet is a bit exaggerated (and the word ‘performance’ there doesn’t take into account efficiency, for example) but I don’t have the time to read the full paper (just skimmed the abstract and conclusion). I quoted the tweet by an author for people to discuss since it’s still a fairly remarkable result.
- bonoboTP 2y agoThis is an accepted ICLR paper by authors from Stanford, Toyota and Google. That's not a guarantee for anything, of course, but they likely know basic algorithms and the second law. You can certainly argue against their claims, but you need to put in the legwork.
- __loam 2y agoI don't think I should need to argue with the absurd claim that these can solve any problem.
- tooltower 2y agoConstant depth circuits can solve everything? I feel like I missed some important part of circuit complexity. Or this is BS.
- whimsicalism 2y agoCoT means you're adding loops
- shawntan 2y agoUsing CoT implicitly increases the depth of the circuit. But yes, poorly worded.
- m3kw9 2y agoSort of like quantum superposition state? So here is an idea, using quantum to produce all possible inferences and use some not yet invented algorithms to collapse to the final result
- deleted 2y ago[deleted]
- riku_iki 2y ago> Remarkably, constant depth is sufficient. I think article also says log(n) embedding size (width?) is required, where n is size of input.
- ljsprague 2y agoSkynet's the limit.
- tsimionescu 2y agoOne question, if anyone knows the details: does this prove that there exists a single LLM that can approximate any function to arbitrary precision given enough CoT, or does it prove that for every function, there exists a Transformer that fits those criteria? That is, does this prove that a single LLM can solve any problem, or that for any problem, we can find an LLM that solves it?
- jstanley 2y agoDoesn't the latter imply the former? If it's possible to find an LLM for any given problem, then find an LLM for the problem "find an LLM for the problem and then evaluate it" and then evaluate it, and then you have an LLM that can solve any problem. It's the "Universal Turing Machine" for LLMs. I wonder what's the LLM equivalent of the halting problem?
- progval 2y ago> It's the "Universal Turing Machine" for LLMs. A closer analogy is the Hutter Search (http://hutter1.net/ai/pfastprg.pdf http://hutter1.net/ai/pfastprg.pdf), as it is also an algorithm that can solve any problem. And it is probably too inefficient to use in practice, like the Hutter Search.
- deleted 2y ago[deleted]
- detourdog 2y agoIn the late ‘80s they were called expert systems. Most demonstrations were regarding troubleshooting large systems, industrial processes, and education.
- DarkNova6 2y agoThe more interesting is whether the ability of reason and solve problems scales linearly or logarithmically.
- deleted 2y ago[deleted]
- bottlepalm 2y agoForget UBI, we're going to need Universal Basic Compute.
- lsy 2y agoNote that for the purposes of this paper a “problem” just means a formally decidable problem or a formal language, and the proof is that by creatively arranging transformers you can make individual transformer runs behave like individual Boolean circuits. However, this is a long way from any practical application of transformers: for one thing, most problems we care about are not stated as formal languages, and we already have an exceptionally more efficient way to implement Boolean circuits.
- shawntan 2y agoIf a "problem we care about" is not stated as a formal language, does it mean it does not exist in the hierarchy of formal languages? Or is it just as yet unclassified?
- tsimionescu 2y agoIt means that there are two problems: one, to formalize the problem as stated while capturing all relevant details, and two, solving the resulting formal problem. Until you solve problem one, you can't use formal methods to say anything about the problem (it's not even clear a priori that a problem is even solvable). Unfortunately, the task of a formalizing an informal problem is itself an informal problem that we don't know how to formalize, so we can't say much about it. So overall, we can't say much about how hard the general problem "given a problem statement from a human, solve that problem" is, whether any particular system (including a human!) can solve it and how long that might take with what resources.
- viraptor 2y ago> task of a formalizing an informal problem is itself an informal problem I couldn't find details about this - do you know of a paper or some resource which digs into that idea?
- tsimionescu 2y agoNo, but it's pretty obvious, isn't it? If you have an informal problem statement, say "I want this button to be bigger", formalizing it can't be a formal process.
- glial 2y agoApologies if this is a dumb question, but aren't all computations inherently serial? In that a Turing machine performs operations serially?
- ninetyninenine 2y agoYeah it's talking about a new feature for LLMs where the output of an LLM is fed back in as input and done again and again and again and this produces way more accurate output.
- ants_everywhere 2y agoYou can model parallel computation by an arbitrary finite product of Turing machines. And then, yes, you can simulate that product on a single Turing machine. I think that's the sort of thing you have in mind? But I'm not aware of what "inherently serial" means. The right idea likely involves talking about complexity classes. E.g. how efficiently does a single Turing machine simulate a product of Turing machines? An inherently serial computation would then be something like a problem where the simulation is significantly slower than running the machines in parallel.
- joe_the_user 2y agoAren't all computations inherently serial? No. "inherently serial" refers to problems that are specified serially and can't be spend up by parallel processing. The sum of a set of N numbers is an example of a problem that is not inherently serial. You can use parallel reduction to perform the computation in O(log(N)) time on an idealized parallel computer but it takes O(N) time on an idealized serial computer. And, it turns, exactly which problems are really are inherently serial is somewhat challenging problem.
- visarga 2y ago> The sum of a set of N numbers is an example of a problem that is not inherently serial. But addition with floats (not reals) is non associative.
- 2y ago
- JSDevOps 2y agoSo given infinite time and resources it can solve any problem? Hardly groundbreaking is it.
- dilyevsky 2y agoInfinite Token Theorem
- bilekas 2y agoNice quip but in reality it's the exact same right?
- JTyQZSnP3cQGa8B 2y agoRendered useless by the infinite money problem.
- dotancohen 2y agoRendered obsolete by the Desperate Venture Capitalist syndrome. See: MS investing in OpenAI
- observationist 2y ago$7 Trillion dollars is all you need.
- zeofig 2y agoI have faith that Nvidia can sell them infinity gpus
- imhoguy 2y agoNow it is time to prove can it loop into creating an infinite number of paperclips? /s https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer https://en.wikipedia.org/wiki/Instrumental_convergence#Paper...
- mrbungie 2y agoThe "We need more Nvidia GPUs and we will reach AGI" theorem.
- wodenokoto 2y agoBut didn't we already know that NN can solve any computable problem? The interesting thing is if they can be trained to solve any (computable) problem.
- imhoguy 2y agoI don't know why I have read "HN", indeed HN can solve any problem.
- tossandthrow 2y agoFeed forward NNs can approximate all functions f: X -> Y only for closed domains. But recurrent neural networks can do solve any computational problem given enough precision.
- wodenokoto 2y agoFirst day of introductions to NN we were asked to create all the logic gates using artificial neurons, and then told "If you have all gates, you can do all computations". I got to admit, I'm sorta sticking to that at face value, because I don't know enough computer science to a) discern if that is true and b) know what "f: X -> Y only for closed domains" means.
- tossandthrow 2y agoI think the easiest way to think about this is in terms of natural numbers, ie. 1, 2, 3, 4. When you only have a fixed width, ie. a static feed forward network, you have an upper limit to the data you can represent and compute on. Eg. if the highest number you can represent is 1.000, then you will need a new NN if you want to do computations on 1.001. ... or use an inductive structure, like a recurrent neural network has.
- roboboffin 2y agoDoes that mean when we reduce the precision of a NN, for example using bfloat16 instead of float32, we reduce the set of computational problems that can be solved. How would that compare with a biological neural network with presumably near-infinite precision ?
- qmatch 2y agoIs this similar to the Universal Approximator Theorem?
- sigmoid10 2y ago>Remarkably, constant depth is sufficient. How would that be remarkable, when it is exactly what he Universal Approximation Theorem already states? Since transformers also use fully connected layers, none of this should really come as a surprise. But from glancing at the paper, they don't even mention it.
- nexustext 2y agoIt's 'remarkable' because (a) academic careers are as much about hype as science, (b) arxiv doesn't have peer review process to quash this, (c) people take arxiv seriously.
- logicchains 2y ago>How would that be remarkable, when it is exactly what he Universal Approximation Theorem already states Only with infinite precision, which is highly unrealistic. Under realistic assumptions, fixed depth transformer without chain-of-thought are very limited in what they can express: https://arxiv.org/abs/2207.00729 https://arxiv.org/abs/2207.00729 . Chain of thought increases the class of problems which fixed depth transformers can solve: https://arxiv.org/abs/2310.07923 https://arxiv.org/abs/2310.07923
- IshKebab 2y agoThe universal approximation theorem has no practical relevance.
- seydor 2y ago'can' But will they? I believe the frontier has moved to making them make sense instead of just making infinite language. The infinite monkey problem is not solved yet
- floppiplopp 2y agoThey have also mathematically proven that transformers are great randomness generators.
- CarRamrod 2y agoDamn, we just used our entire Round A acquiring an infinite amount of bananas and typewriter ink. The boss is not going to like this.
- bryanrasmussen 2y agodid you get both infinite bananas and infinite typewriter ink, or was there a limited supply of typewriter ink? If the first, it was worth it.
- deleted 2y ago[deleted]
- nopinsight 2y agoNo worries! With the magic bananas and ink you've acquired, those monkeys will surely produce output with a signal-to-noise ratio rivaling the best LLMs. I’m sure your startup will achieve the coveted Apeicorn status soon!
- imjonse 2y agoHopefully not Cavendish, as those are too sugary for monkeys and you'll just get hallucinations.
- dotancohen 2y agoNaturally. It's the printer ink that is forbiddingly expensive. And the bananas are carbon neutral.
- theshrike79 2y agoAre we getting to a point where the LLM will just answer "42" and we need to figure out the question? =)
- tossandthrow 2y ago> We have mathematically proven that transformers can solve any problem, provided they are allowed to generate as many intermediate reasoning tokens as needed. This is also the case with plain and regular RNNs
- baq 2y agoNow just need an autoregressive transformer <==> RNN isomorphism paper and we're golden
- logicchains 2y agoPlain RNNs are theoretically weaker than transformers with COT: https://arxiv.org/abs/2402.18510 https://arxiv.org/abs/2402.18510 .
- tossandthrow 2y agoThe paper says transformers perform better than RNNs, which is not surprising. However, they are both, theoretically, Turing complete computers. So they are equally expressive.
- Intralexical 2y agoIsn't it also expected to be the case with RNGs?
- smusamashah 2y agoA reply in this twitter thread links to a detailed blog post titled "Universal computation by attention: Running cellular automata and other programs on Claude 3 Opus." https://x.com/ctjlewis/status/1786948443472339247 https://x.com/ctjlewis/status/1786948443472339247
- smusamashah 2y agohttps://x.com/ctjlewis/status/1786948443472339247 https://x.com/ctjlewis/status/1786948443472339247 "Running cellular automata and other programs on Claude 3 Opus." Its one of the replies on this tweet.
- mg 2y agoHas it been publicly benchmarked yet, if this approach: Hello LLM, please solve this task: <task> Can be improved by performing this afterwards? for iteration in range(10): Hello LLM, please solve this task: <task> Here is a possible solution: <last_reply> Please look at it and see if you can improve it. Then tell me your improved solution.
- Kiro 2y agoIsn't that the whole reason that o1 works?
- ben_w 2y agoI think o1 is more like "pretend you're doing a job interview, think step and show your working". I tried something similar to the suggested iterative loop on a blog post I'd authored but wanted help copy editing; first few were good enough, but then it got very confused and decided the blog post wasn't actually a blog post to be edited and instead that what I really wanted to know was the implications of Florida something something Republican Party. Benchmark would be neat, because all I have is an anecdote.
- bachback 2y agofor coding tasks see https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/ the question is how would you define "improve" and "solve". RLHF in a way delegates this to humans.
- eykrehbein 2y agoBruteforceLLM
- lorepieri 2y agoNot sure if it has been benchmarked, but I've called this technique the "for-loop of thought". :)
- larodi 2y agoI'm waiting for peoples of AI to discover syllogism and inference in its original PROLOG sense, which this CoT abomination basically tries to achieve. Interestingly, if all logical content is translated to rules, and then only rules are fed into the LLM training set, what would the result be, and can the probabilistic magic be made into actually following reason without all the dice.
- trescenzi 2y agoRight we’ve now gotten to the stage of this AI cycle where we start using the new tool to solve problems old tools could solve. Saying a transformer can solve any Formally decidable problem if given enough tape isn’t saying much. It’s a cool proof, don’t mean to deny that, but it doesn’t mean much practically as we already have more efficient tools that can do the same.
- marcosdumay 2y agoWhat I don't get is... didn't people prove that in the 90s for any multi-layer neural network? Didn't people prove transformers are equivalent on the transformers paper?
- Nevermark 2y agoYes they did. A two layer network with enough units in the hidden layer can form any mapping to any desired accuracy. And a two layer network with single-delay feedback from the hidden units to themselves can capture any dynamic behavior (to any desired accuracy). Adding layers and more structured architectures creates the opportunity for more efficient training and inference, but doesn't enable any new potential behavior. (Except in the sense that reducing resource requirements can allow impractical problems to become practical.)
- larodi 2y agoPutting a 50 bucks bet that some very smart kid in the near future will come with some enthrophy-meets-graphical-structures theorem which gives an estimation of how the loss of information is affected by the size and type of the underlying structure holding this information. It took a while for people to actually start talking about LZW as grammar algo, not a "dictionary"-based algorithm. Which is then reasoned about in a more general sense again by https://en.wikipedia.org/wiki/Sequitur_algorithm https://en.wikipedia.org/wiki/Sequitur_algorithm. This is not to say that LLMs are not cool, we put them to use every day. But the reasoning part is never going to be a trustworthy one without a 100% discreet system, which can infer the syllogistic chain with zero doubt and 100% tracable origin.
- scotty79 2y agoChain of thought GPT is sort of a Turing machine with a tape that it's allowed to write to for purposes other than outputting the answer.
- tonii141 2y agoRandom generator of tokens can also solve any problem if you give it enough time and memory.
- tonii141 2y ago[flagged]
- deleted 2y ago[deleted]
- cpldcpu 2y agoCan any of these tools do anything that the Github copilot cannot do? (Apart from using other models?). I tried Continue.dev and cursor.ai, but it was not immediately obvious to me. Maybe I am missing something workflow specific?
- HarHarVeryFunny 2y agoSure, in same sense as an infinitely long tape let's a Turing machine solve arbitrary problems. In theory at least. If one had the right program.
- falcor84 2y agoIt's not clear me what you're saying; isn't the whole deal here that by performing RL on the CoT (given sufficient size and compute) it would converge to the right program?
- HarHarVeryFunny 2y agoI was really saying two things: 1) The theoretical notion that a fixed depth transformer + COT can solve arbitrary problems involving sequential computation is rather like similar theoretical notions of a Turing machine as universal computer, or of an ANN with a hidden layer able to represent arbitrary functions .. it may be true, but at the same time not useful 2) The Turing machine, just as the LLM+COT, is only as useful as the program it is running. If the LLM-COT is incapable of runtime learning and just trying to mimic some reasoning heuristics, then that is going to limit it's function, even if theoretically such an "architecture" could do more if only it were running a universal AGI program Using RL to encourage the LLM to predict continuations according to some set of reasoning heuristics is what it is. It's not going to make the model follow any specific reasoning logic, but is presumably hoped to generate a variety of continuations that the COT "search" will be able to utilize to arrive at a better response than it otherwise would have done. More of an incremental improvement (as reflected in the benchmark scores it achieves) than "converging to the right program".
- __loam 2y agoSometimes reading hackernews makes me want to slam my head on the table repeatedly. Given sufficient size and compute is one of the most load bearing phrases I've ever seen.
- falcor84 2y agoBut it is load bearing. I mean, I personally can't stop being amazed at how with each year that passes, things that were unimaginable with all the world's technology a decade ago are becoming straightforward to run on a reasonably priced laptop. And at this stage, I wouldn't bet even $100 against any particular computational problem being solved in some FAANG datacenter by the end of the decade.
- empath75 2y agoIs this more general than LLMs? Is it possible to do something Chain-of-Thought-like in a transformer model that _isn't_ trained on language?
- lgessler 2y agoI liked Yoav Goldberg snarky's quote tweet: > next paper: transformers can solve any problem but on some of them they may compute indefinitely and never provide an answer > (and you cannot tell in advance which is which!!) https://twitter.com/yoavgo/status/1835802380589203802 https://twitter.com/yoavgo/status/1835802380589203802
- teqsun 2y agoIt reminds me of Busy Beaver
- phemartin 2y agoGoogle Illuminate Summary Chat: https://illuminate.google.com/library?play=UzUI5b_HF8UI https://illuminate.google.com/library?play=UzUI5b_HF8UI