15 ms·
There are no new ideas in AI, only new datasets
- scrubs 1y agoTrue or false? One an llm is constructed, it mutates to include data from prompt-response interaction?
- code_for_monkey 1y ago[flagged]
- HeatrayEnjoyer 1y agoEveryone knows we'll always need horseshoes /s
- ctoth 1y agoReinforcement learning from self-play/AlphaWhatever? Nah must just be datasets. :)
- grumpopotamus 1y agohttps://en.wikipedia.org/wiki/TD-Gammon https://en.wikipedia.org/wiki/TD-Gammon
- Y_Y 1y agoYou raise a really interesting point. I'm sure it's just missed my notice, but I'm not familiar with any projects from antediluvian AI that have been resurrected to run on modern hardware and see where they'd really asymptote if they'd had the compute they deserved.
- FeepingCreature 1y agoTo be fair, usually those projects would need considerable work to be ported to modern multicore machines, let alone GPUs.
- genewitch 1y agocan you name a couple so i can see how much work is involved? markov chains compile fast and respond fast, sure, and neural nets train pretty quick too, so i'm wondering where the cutoff is; expert systems?
- FeepingCreature 1y agoOh I have no idea, sorry if I gave the impression I have special knowledge here. I'm just deriving this from consumer multicore systems being historically rare and GPGPUs being nonexistant.
- rsfern 1y agoThis paper “were RNNs all we needed?” explores this hypothesis a bit, finding that some pre-transformer sequence models can match transformers when trained at appropriate scale. Though they did have to make some modifications to unlock more parallelism https://arxiv.org/abs/2410.01201 https://arxiv.org/abs/2410.01201
- zahlman 1y agoFor that matter, https://gist.github.com/deebs67/8fbcf8b127a63e70d4a3f8590c97701d https://gist.github.com/deebs67/8fbcf8b127a63e70d4a3f8590c97... .
- NitpickLawyer 1y agoAnd architecture stuff like actually useful long context. Whatever they did with gemini 2.5 is miles ahead in long context useful results compared to the previous models. I'd be very surprised if gemini 2.5 is "just" gemini 1 w/ better data.
- shwouchk 1y agoi dont know what all the hype is with gemini 2.5, at least the currently running instance. from my experience at least in conversation mode, it cannot remember my instructions to avoid apologies and similar platitudes from either the “persona”, personal instructions, or from ine message to the next.
- nyrikki 1y agoBig difference between a perfect information, completely specified zero sum game and the real world. As a simple analogy, read out the following sentence multiple times, stressing a different word each time. "I never said she stole my money" Note how the meaning changes and is often unique? That is a lens I to the frame problem and it's inverse, the specification problem. The above problem quickly becomes tower-complete, and recent studies suggest that RL is reinforcing or increasing the weight of existing patterns. As the open domain frame problem and similar challenges are equivalent to HALT, finding new ways to extract useful information will be important for generalization IMHO. Synthetic data is useful, but not a complete solution, especially for tower problems.
- genewitch 1y agoThe one we use is "I always pay my taxes" and as far as synthetic vs real data, there's a lot of gaps in LLM knowledge; and vision models suffer from "limited tags", which used to have workarounds with textual embeddings and the like, but those went by the wayside as LoRA, controlnet, etc. appeared. There's people who are fairly well known that LLMs have no idea about. There's things in books i own that the AI confidently tells me are either wrong or don't exist. That one page about compressing 1 gig wikipedia as small as possible implicitly and explicitly states that AI is "basically compression" - and if the data isn't there, it's not in the compressed set (weights) either. And i'll reply to another comment here, about "24/7 rolling/ for looped" AI - i thought of doing this when i first found out about LLMs, but context windows are the enemy, here. I have a couple of ideas about how to have a continuous AI, but i don't have the capital to test it out.
- energy123 1y agoSelf-play gives you a large explosion of data.
- kogus 1y agoTo be fair, if you imagine a system that successfully reproduced human intelligence, then 'changing datasets' would probably be a fair summary of what it would take to have different models. After all, our own memories, training, education, background, etc are a very large component of our own problem solving abilities.
- jschveibinz 1y agoI will respectfully disagree. All "new" ideas come from old ideas. AI is a tool to access old ideas with speed and with new perspectives that hasn't been available up until now. Innovation is in the cracks: recognition of holes, intersections, tangents, etc. on old ideas. It has bent said that innovation is done on the shoulders of giants. So AI can be an express elevator up to an army of giant's shoulders? It all depends on how you use the tools.
- gametorch 1y agoExactly! Can you imagine if we applied the same gatekeeping logic to science? Imagine you weren't allowed to use someone else's scientific work or any derivative of it. We would make no progress. The only legitimate defense I have ever seen here revolves around IP and copyright infringement, which I couldn't care less about.
- alfalfasprout 1y agoAccess old ideas? Yes. With new perspectives? Not necessarily. An LLM may be able to assist in interpreting data with new perspectives but in practice they're still fairly bad at greenfield work. As with most things, the truth lies somewhere in the middle. LLMs can be helpful as a way of accelerating certain kinds and certain aspects of research but not others.
- stevep98 1y ago> Access old ideas? Yes. With new perspectives? I wonder if we can mine patent databases for old ideas that never worked out in the past, but now are more useful. Perhaps due to modern machining or newer materials or just new applications of the idea.
- deleted 1y ago[deleted]
- deleted 1y ago[deleted]
- bcrosby95 1y ago
- saltserv 1y ago[dead]
- tippytippytango 1y agoSometimes we get confused by the difference between technological and scientific progress. When science makes progress it unlocks new S-curves that progress at an incredible pace until you get into the diminishing returns region. People complain of slowing progress but it was always slow, you just didn’t notice that nothing new was happening during the exponential take off of the S-curve, just furious optimization.
- baxtr 1y agoFully agree. And at the same time I have noticed that people don’t understand the difference between an S-curve and an exponential function. They can look almost identical at certain intervals.
- timewizard 1y agoYou're being awfully generous to describe basic hype as "technological progress."
- pevansgreenwood 1y agoThat's putting the cart before the horse. Thermodynamics came after the steam engine was made practical. Flight came before aerodynamics. Metallurgy before materials science. Radio before electromagnetic theory took hold. Even LLMs are the result of a lot of tinkering rather than scientific insight. It’s the successful tinkering that creates the puzzle science later formalises.
- Zacharias030 1y agoThe crypto mind cannot comprehend
- the_sleaze_ 1y agoSomething I've come to recognize and deeply resent. Being Right and being Successful are not the same thing.
- protocolture 1y agoAs far back as 2017 I copped a lot of flak for suggesting that the coming automation revolution will be great at copying office workers and artists but wont be in order of replacing the whole human race. A lot of the time moores law got thrown back in my face. But thats how this works, we unlock something new, we exploit it as far as possible, the shine wears off and we deal with the aftermath.
- ks2048 1y agoThe latest LLMs are simply multiplying and adding various numbers together... Babylonians were doing that 4000 years ago.
- bobson381 1y agoYou are just a lot of interactions of waves. All meaning is assigned. I prefer to think of this like the Goedel generator that found new formal expressions for the Principia - because we have a way of indexing concept-space, there's no telling what we might find in the gaps.
- thenaturalist 1y agoBut on clay tables, not in semi-conductive electron prisons separated by one-atom-thick walls. Slight difference to those methods, wouldn't you agree?
- geysersam 1y agoNo it's exactly the same. Everything old is new again...
- thenaturalist 1y agoResults != methodology
- voxleone 1y agoI'd say with confidence: we're living in the early days. AI has made jaw-dropping progress in two major domains: language and vision. With large language models (LLMs) like GPT-4 and Claude, and vision models like CLIP and DALL·E, we've seen machines that can generate poetry, write code, describe photos, and even hold eerily humanlike conversations. But as impressive as this is, it’s easy to lose sight of the bigger picture: we’ve only scratched the surface of what artificial intelligence could be — because we’ve only scaled two modalities: text and images. That’s like saying we’ve modeled human intelligence by mastering reading and eyesight, while ignoring touch, taste, smell, motion, memory, emotion, and everything else that makes our cognition rich, embodied, and contextual. Human intelligence is multimodal. We make sense of the world through: Touch (the texture of a surface, the feedback of pressure, the warmth of skin0; Smell and taste (deeply tied to memory, danger, pleasure, and even creativity); Proprioception (the sense of where your body is in space — how you move and balance); Emotional and internal states (hunger, pain, comfort, fear, motivation). None of these are captured by current LLMs or vision transformers. Not even close. And yet, our cognitive lives depend on them. Language and vision are just the beginning — the parts we were able to digitize first - not necessarily the most central to intelligence. The real frontier of AI lies in the messy, rich, sensory world where people live. We’ll need new hardware (sensors), new data representations (beyond tokens), and new ways to train models that grow understanding from experience, not just patterns.
- Swizec 1y ago> The real frontier of AI lies in the messy, rich, sensory world where people live. We’ll need new hardware (sensors), new data representations (beyond tokens), and new ways to train models that grow understanding from experience, not just patterns. Like Dr. Who said: DALEKs aren't brains in a machine, they are the machine! Same is true for humans. We really are the whole body, we're not just driving it around.
- nomel 1y agoThere are many people who mentally developed while paralyzed that literally drive around their bodies via motorized wheelchair. I don't think there's any evidence that a brain couldn't exist or develop in a jar, given only the inputs modern AI now has (text, video, audio).
- luppy47474 1y agoHmmm
- anon291 1y agoI mean there's no new ideas for saas but just new applications and that worked out pretty well
- rar00 1y agodisagree, there are a few organisations exploring novel paths. It's just that throwing new data at an "old" algorithm is much easier and has been a winning strategy. And, also, there's no incentive for a private org to advertise a new idea that seems to be working (mine's a notable exception :D).
- tantalor 1y ago> If data is the only thing that matters, why are 95% of people working on new methods? Because new methods unlock access to new datasets. Edit: Oh I see this was a rhetorical question answered in the next paragraph. D'oh
- piinbinary 1y agoAI training is currently a process of making the AI remember the dataset. It doesn't involve the AI thinking about the dataset and drawing (and remembering) conclusions. It can probably remember more facts about a topic than a PhD in that topic, but the PhD will be better at thinking about that topic.
- jayd16 1y agoIts a bit more complex than that. Its more about baking out the dataset into heuristics that a machine can use to match a satisfying result to an input. Sometimes these heuristics are surprising to a human and can solve a problem in a novel way. "Thinking" is too broad a term to apply usefully but I would say its pretty clear we are not close to AGI.
- nkrisc 1y ago> It can probably remember more facts about a topic than a PhD in that topic So can a notebook.
- tantalor 1y agoMaybe that's why PhDs keep the textbooks they use at hand, so they don't have to remember everything. Why should the model need to memorize facts we already have written down somewhere?
- EternalFury 1y agoWhat John Carmack is exploring is pretty revealing. Train models to play 2D video games to a superhuman level, then ask them to play a level they have not seen before or another 2D video game they have not seen before. The transfer function is negative. So, in my definition, no intelligence has been developed, only expertise in a narrow set of tasks. It’s apparently much easier to scare the masses with visions of ASI, than to build a general intelligence that can pick up a new 2D video game faster than a human being.
- ferguess_k 1y agoCan you please explain "the transfer function is negative"? I'm wondering whether one has tested with the same model but on two situations: 1) Bring it to superhuman level in game A and then present game B, which is similar to A, to it. 2) Present B to it without presenting A. If 1) is not significantly better than 2) then maybe it is not carrying much "knowledge", or maybe we simply did not program it correctly.
- tough 1y agoI think the problem is we train models to pattern match, not to learn or reason about world models
- Kapura 1y agoHere's an idea: make the AIs consistent at doing things computers are good at. Here's an anecdote from a friend who's living in Japan: > i used chatgpt for the first time today and have some lite rage if you wanna hear it. tldr it wasnt correct. i thought of one simple task that it should be good at and it couldnt do that. > (The kangxi radicals are neatly in order in unicode so you can just ++ thru em. The cjks are not. I couldnt see any clear mapping so i asked gpt to do it. Big mess i had to untangle manually anyway it woulda been faster to look them up by hand (theres 214)) > The big kicker was like, it gave me 213. And i was like, "why is one missing?" Then i put it back in and said count how many numbers are here and it said 214, and there just werent. Like come on you SHOULD be able to count. If you can make the language models actually interface with what we've been able to do with computers for decades, i imagine many paths open up.
- cheevly 1y agoMany of us have solved this with internal tooling that has not yet been shared or released to the public.
- layer8 1y agoThis needs to be generalized however. For example, if you present an AI with a drawing of some directed graph (a state diagram, for example), it should be able to answer questions based on the precise set of all possible paths in that graph, without someone having to write tooling for diagram or graph processing and traversal. Or, given a photo of a dropped box of matches, an AI should be able to precisely count the matches, as far as they are individually visible (which a human could do by keeping a tally while coloring the matches). There are probably better examples, these are off the cuff. There’s an infinite repertoire of such tasks that combine AI capabilities with traditional computer algorithms, and I don’t think we have a generic way of having AI autonomously outsource whatever parts require precision in a reliable way.
- snapcaster 1y agoWhat you're describing sounds like agentic tool usage. Have you kept up with the latest developments on that? it's already solved depending on how strict you define your criteria above
- b0a04gl 1y ago[dead]
- krunck 1y agoUntil these "AI" systems become always-on, always-thinking, always-processing, progress is stuck. The current push button AI - meaning it only processes when we prompt it - is not how the kind of AI that everyone is dreaming of needs to function.
- fwip 1y agoFrom a technical perspective, we can do that with a for loop. The reason we don't do it isn't because it's hard, it's because it yields worse results for increased cost.
- nyrulez 1y agoThings haven't changed much in terms of truly new ideas since electricity was invented. Everything else is just applications on top of that. Make the electrons flow in a different way and you get a different outcome.
- nomel 1y ago> Make the electrons flow in a different way and you get a different outcome. This happens to be the basis of every aspect of our biology.
- seydor 1y agoThere are new ideas, people are finding new ways to build vision models, which then are applied to language models and vice versa (like diffusion). The original idea of connectionism is that neural networks can represent any function, which is the fundamental mathematical fact. So we should be optimistic, neural nets will be able to do anything. Which neural nets? So far people have stumbled on a few productive architectures, but it appears to be more alchemy than science. There is no reason why we should think there won't be both new ideas and new data. Biology did it, humans will do it too. > we’re engaged in a decentralized globalized exercise of Science, where findings are shared openly Maybe the findings are shared, if they make the Company look good. But the methods are not anymore
- lossolo 1y agoI wrote about it around a year ago here: "There weren't really any advancements from around 2018. The majority of the 'advancements' were in the amount of parameters, training data, and its applications. What was the GPT-3 to ChatGPT transition? It involved fine-tuning, using specifically crafted training data. What changed from GPT-3 to GPT-4? It was the increase in the number of parameters, improved training data, and the addition of another modality. From GPT-4 to GPT-40? There was more optimization and the introduction of a new modality. The only thing left that could further improve models is to add one more modality, which could be video or other sensory inputs, along with some optimization and more parameters. We are approaching diminishing returns." [1] 10 months ago around o1 release: "It's because there is nothing novel here from an architectural point of view. Again, the secret sauce is only in the training data. O1 seems like a variant of RLRF https://arxiv.org/abs/2403.14238 https://arxiv.org/abs/2403.14238 Soon you will see similar models from competitors." [2] Winter is coming. 1. https://news.ycombinator.com/item?id=40624112 https://news.ycombinator.com/item?id=40624112 2. https://news.ycombinator.com/item?id=41526039 https://news.ycombinator.com/item?id=41526039
- tolerance 1y agoAnd when winter does arrive, then what? The technology is slowing down while its popularity picks up. Can sparks fly out of snow?
- blibble 1y agothe trillion dollar funding tap is turned off, the prices charged then will have to reflect the costs shortly thereafter the entire ecosystem will collapse
- imiric 1y ago> And when winter does arrive, then what? If the technology is useful, the Slope of Enlightenment, followed by the Plateau of Productivity.
- LarsDu88 1y agoIf datasets are what we are talking about, I'd like to bring attention to the biological datasets out there that have yet to be fully harnessed. The ability to collect gene expression data at a tissue specific level has only been invented and automated in the last 4-5 years (see 10X Genomics Xenium, MERFISH). We've only recently figured out how to collect this data at the scale of millions of cells. A breakthrough on this front may be the next big area of advancement.
- Night_Thastus 1y agoMan I can't wait for this '''''AI''''' stuff to blow over. The back and forth gets a bit exhausting.
- Culonavirus 1y agoWithin the bounds of HN audience I would definitely describe myself as an A(G)I skeptic. But even I can see that this ""AI"" stuff is not going to blow over. That ship has sailed. Even if the current models get only marginal improvements, the momentum is unquestionably, inarguably there to make the adoption and productization 10x or even 100x wider than it is now. Robotics, automatization, self-driving, all kinds of kiosks, military applications (gathering and merging sensor data, controlling drone swarms, etc.)... Just the amount of money (it's going to be trillions before the decade is over) and the amount of students in the field (basically all computer science degrees nowadays teach AI in some form) guarantees we're stuck with ""AI"" forever (at least until it kills us or merges with us)
- actionfromafar 1y ago1000x more than it is now, at least. Imagine every kind of electronic thing ever made, but with a sprinkle of AI. Why? For the same reason now 32-bit CPUs and a megabyte of RAM run some Javascript or MicroPython to check a handful of logical conditions and flip a couple of I/O bits is no longer a custom curcuit or a handful of TTL-chips wired together.
- Night_Thastus 1y agoThe reason I think it's going to blow over is that even the best models are frequently quite terrible. The fundamental problems of how they work can't really be fixed because they're a feature - the way that LLMs work. And no one has found a way to make any money with it. All the tech companies are burning money by the truckload so investors don't lose confidence, but none of them have actually shown it's a good financial investment. At the end of the day, I don't think anyone is going to want to pay what it really costs to run these models, just for a result that is so unreliable. Once they start to stagnate everyone will lose interest. The only reason it might stick around is because investors will get desperate to get returns and go full sunk-cost once it starts looking like they made a bad call. (Which they will blame the companies for, of course)
- alganet 1y agoDataset? That's so 2000s. Each crawl on the internet is actually a discrete chunk of a more abstractly defined, constant influx of information streams. Let's call them rivers (it's a big stream). These rivers can dry up, present seasonal shifts, be poisoned, be barraged. It will never "get there" and gather enough data to "be done". -- Regarding "new ideas in AI", I think there could be. But this whole thing is not about AI anymore.
- strangescript 1y agoIf you work with model architecture and read papers, how could not know there are a flood of new ideas? Only few yield interesting results though. I kind of wonder if libraries like pytorch have hurt experimental development. So many basic concepts no one thinks about anymore because they just use the out of the box solutions. And maybe those solutions are great and those parts are "solved", but I am not sure. How many models are using someone else's tokenizer, or someone else's strapped on vision model just to check a box in the model card?
- kevmo314 1y agoThe people who don't think about such things probably wouldn't develop experimentally sans pytorch either.
- thenaturalist 1y agoThat's been the very normal way of the human world. When the foundation layer at a given moment doesn't yield an ROI on intellectual exploration - say because you can overcompensate with VC funded raw compute and make more progess elsewhere -, few(er) will go there. But inevitably, as other domains reach diminishing returns, bright minds will take a look around where significant gains for their effort can be found. And so will the next generation of PyTorch or foundational technologies evolve.
- mardifoufs 1y agoYeah and even then, it's been like ~ 2-3 years since the last rather major Architectural improvement, major enough for a lot of people to actually hear about it and use it daily. I think some people lose perspective on how short of a time frame 3 years is. But yes, there's a ton of interesting and useful stuff (beyond datasets and data related improvements) going on right now, and I'm not even talking about LLMs. I don't do anything related to LLM and even then I still see tons of new stuff popping up regularly.
- _giorgio_ 1y agoIt's the opposite. Frameworks like pytorch are really flexible. You can implement any architecture, and if it's not enough, you can learn CUDA. Keras it's the opposite, it's probably like you describe things.
- russellbeattie 1y agoParadigm shifts are often just a conglomeration of previous ideas with one little tweak that suddenly propels a technology ahead 10x which opens up a whole new era. The iPhone is a perfect example. There were smartphones with cameras and web browsers before. But when the iPhone launched, it added a capacitive touch screen that was so responsive there was no need for a keyboard. The importance of that one technical innovation can't be overstated. Then the "new new thing" is followed by a period of years where the innovation is refined, distributed, applied to different contexts, and incrementally improved. The iPhone launched in 2007 is not really that much different than the one you have in your pocket today. The last 20 years has been about improvements. The web browser before that is also pretty much the same as the one you use today. We've seen the same pattern happen with LLMs. The author of the article points out that many of AI's breakthroughs have been around since the 1990s. Sure! And the Internet was created in the 1970s and mobile phones were invented in the 1980s. That doesn't mean the web and smartphones weren't monumental technological events. And it doesn't mean LLMs and AI innovation is somehow not proceeding apace. It's just how this stuff works.
- lsy 1y agoThis seems simplistic, tech and infrastructure play a huge part here. A short and incomplete list of things that contributed: - Moore's law petering out, steering hardware advancements towards parallelism - Fast-enough internet creating shift to processing and storage in large server farms, enabling both high-cost training and remote storage of large models - Social media + search both enlisting consumers as data producers, and necessitating the creation of armies of Mturkers for content moderation + evaluation, later becoming available for tagging and rlhf - A long-term shift to a text-oriented society, beginning with print capitalism and continuing through the rise of "knowledge work" through to the migration of daily tasks (work, bill paying, shopping) online, that allows a program that only produces text to appear capable of doing many of the things a person does We may have previously had the technical ideas in the 1990s but we certainly didn't have the ripened infrastructure to put them into practice. If we had the dataset to create an LLM in the 90s, it still would have been astronomically cost-prohibitive to train, both in CPU and human labor, and it wouldn't have as much of an effect on society because you wouldn't be able to hook it up to commerce or day-to-day activities (far fewer texts, emails, ecommerce).
- somebodythere 1y agoI don't know if it matters. Even if the best we can do is get really good at interpolating between solutions to cognitive tasks on the data manifold, the only economically useful human labor left asymptotes toward frontier work; work that only a single-digit percentage of people can actually perform.
- tim333 1y agoAn interesting step forward, although an old idea we seem close to is recursive self improvement. Get the AI to make a modified version of itself to try to think better.
- mdaniel 1y agoInbreeding is illegal for a reason
- cadamsdotcom 1y agoWhat about actively obtained data - models seeking data, rather than being fed. Human babies put things in their mouths, they try to stand and fall over. They “do stuff” to learn what works. Right now we’re just telling models what works. What about simulation: models can make 3D objects so why not give them a physics simulator? We have amazing high fidelity (and low cost!) game engines that would be a great building block. What about rumination: behind every Cursor rule for example, is a whole story of why a user added it. Why not take the rule, ask a reasoning model to hypothesize about why that rule was created, and add that rumination (along with the rule) to the training data. Providing opportunities to reflect on the choices made by their users might deepen any insights, squeezing more juice out of the data.
- kevmo314 1y agoThat would be reinforcement learning. The juice is quite hard to squeeze.
- cadamsdotcom 1y agoAgreed for most cases. Each Cursor rule is a byproduct of tons of work and probably contains lots that can be unpacked. Any research on that?
- kevmo314 1y agoYeah, at a very high level it's similar to an actor-critic reinforcement learning algorithm. The rule text is a value function and one could build a critic model that takes as input the rule text and the main model's (the actor's) output to produce a reward. This is easier said than done though because this value function is so noisy it's often hard to learn from it. And also whether or not a response (the model output) matches the value function (the Cursor rules) is not even that easy to grade. It's been easier to train the chain-of-thought style reasoning since one can directly score it via the length of thinking. This new paper covers some of the difficulties of language-based critic models: https://openreview.net/pdf?id=0tXmtd0vZG https://openreview.net/pdf?id=0tXmtd0vZG Generally speaking, the algorithm and approach is not new. Being able to do it in a reasonable amount of compute is the new part.
- sakex 1y agoThere are new things being tested and yielding results monthly in modelling. We've deviated quite a bit from the original multi head attention.
- blobbers 1y agoWhy is DeepSeek specifically called out?
- mikewarot 1y agoHardware isn't even close to being out of steam. There are some breathtakingly obvious premature optimizations that we can undo to get at least 99% power reduction for the same amount of compute. For example, FPGAs use a lot of area and power routing signals across the chip. Those long lines have a large capacitance, and thus cause a large amount of dynamic power loss. So does moving parameters around to/from RAM instead of just loading up a vast array of LUTs with the values once.
- SamaraMichi 1y agoThis brings us to the problem AI companies are facing, the lack of data, they have already hoovered as much as they can from the internet and desperately need more data. Which make sit blatantly obvious why we're beginning to see products being marketed under the guise of assistants/tools to aid you whose actual purpose is to gather real world picture and audio data, think meta glasses and what Ives and Altman are cooking up with their partnership.
- Daisywh 1y agoIf we’re serious about data being more important than models, then where are the Similar to ISO standards for dataset quality? We have so many model metrics, but almost nothing standardized for data integrity or reproducibility.
- TimByte 1y agoWhat happens when we really run out of fresh, high-quality data? YouTube and robotics make sense as next frontiers, but they come with serious scaling, labeling, and privacy headaches
- ChaoPrayaWave 1y agoFeels like we’ve built this massive engine that runs on high octane data, but never stopped to ask what happens when the fuel runs dry. Maybe it’s time to focus more on efficient learning, not just feeding more and more.
- NetRunnerSu 1y agoBecause the externally injected loss function will empty the brain of the model. Models need to decide for themselves what they should learn. Eventually, after entering the open world, reinforcement learning/genetic algorithms are still the only perpetual training solution. https://github.com/dmf-archive/PILF https://github.com/dmf-archive/PILF
- deleted 1y ago[deleted]
- Leon_25 1y agoAt Axon, we see the same pattern: data quality and diversity make a bigger difference than architecture tweaks. Whether it's AI for logistics or enterprise automation, real progress comes when we unlock new, structured datasets, not when we chase “smarter” models on stale inputs.
- AbstractH24 1y agoImagine if the original moores law tracked how often CPUs doubled the semi conductors while still functioning properly 50% of the time. I don’t think it would have had the same impact
- JKCalhoun 1y ago> It’s not crazy to argue that all the underlying mechanisms of these breakthroughs existed in the 1990s, if not before. That's not super relevant in my mind. It's because they're showing fruit now that will allow research to move forward. And the success, as we know, draws a lot of eyeballs, dollars, resources. If this path was going to hit a wall, we will hit it more quickly now. If another way is required to move forward, we are more likely to find it now.
- ahmedhawas123 1y agoI think this is reflective of current state, but does not mean this will be the future. I think there is a lot of innovation to come on revisiting some of the 1990s principles of back propagation and optimization. Imagine if you could train current models to optimal weights in 1 day or 1 hour instead of weeks/months? Just a hypothesis of mine
- 1vuio0pswjnm7 1y agoWhat about hardware Ideas are not new, according to author But hardware is new and author never mentions impact of hardware improvements
- bladecd 1y agoThe only real important thing in AI is data, not infrastructure, not fancy methods.