5 ms·
Jeff Dean: Trends in Machine Learning [video]
- wslh 3y agoI think this is the time where the idiom "it's good fishing in troubled waters" makes completely sense. This is a moment where it is wise to wait and see. Investing in building AI/AGI startups now is like being a fish more than a fishermen. An outlier could win market share but will be only one within zillions. Google is catching up OpenAI but their competition is fruitful for all. Typical oligopoly. A single startup showing good traction will be immediately acquired. From the business perspective it is time to focus on "go to market execution" and less or nothing on research. The research results are coming alone, except if you are one of the top scientist teams in the world or someone alike Ramanujan.
- osigurdson 3y agoOne thing I think should be fairly clear is Google will actually try to compete in this space (vs quitting if it isn't an immediate success as they usually do). Traditional web search will be fairly uncommon in the long run I think.
- swyx 3y ago[flagged]
- swyx 3y ago(to give credit, this started from https://summarize.tech/ https://summarize.tech/ and then i hand edited it down and deleted generic commentary while i read the summary. submitted here for ppl who also prefer to read rather than watch a 1hr video) actual human opinion: i think nothing strictly new was disclosed here, but its always nice to have in 1 hour a high level overview of "how jeff dean looks at the world" which aligns you accordingly with at least the public facing understanding of what google wants us to understand about their work. Gemini 1.5 has made a huge splash by being genuinely better than GPT4 in some respects (see the recent reddit HVM post) but doesnt seem to have been covered in any detail here. was also not aware of Bar, which seems to be "Google LMSys".
- nomilk 3y agoCan you link to the reddit post (I googled it and got something completely different) EDIT: think I found it? https://www.reddit.com/r/singularity/comments/1atjz9v/ive_put_a_complex_codebase_into_a_single/ https://www.reddit.com/r/singularity/comments/1atjz9v/ive_pu...
- swyx 3y agoyes thats it
- gandalfgeek 3y ago[flagged]
- carbocation 3y agoFrom clicking through the video and watching this summary, I didn't feel like this summary was very good. (No malice intended, just feedback.)
- yeknoda 3y agoDidn't watch it. Also hate it.
- wanderingmind 3y agoCan you provide specific details of why it was not good and what critical details did it miss?
- gandalfgeek 3y agoAppreciate the feedback. This was a weekend project. I wanted to explore summarization without paraphrasing, so that the output was a cut up version of the input video. Agreed that it terms of conceptual clarity often a textual summary that is synthesized comes out ahead.
- ldjkfkdsjnv 3y ago[flagged]
- t8sr 3y agoThis is one of the most accomplished people in software development talking to a class of CS students. It’s not entertainment. Have we reached a point where lecturers need to post “hot takes” on Twitter as a prerequisite to keeping people’s attention?
- ein0p 3y agoNonetheless, I can empathize with GP, I wish the talk focused on the future more, and on history and marketing of Google a lot less. Yes, we get it, Google used to lead in this space, still does in some narrow niches, but recounting the glory days is not how you win mindshare. Show me the demos, make me excited about your vision.
- willsmith72 3y agoi would've been impressed by this 2 years ago. i think it's got to the point where real, valuable ai is in the hands of the everyday consumer, so we start judging the models for ourselves. having seen google continually get crushed over the past year, a bunch of benchmarks just fail to impress. in particular in this case, they're comparing their latest model to gpt4, which hasn't changed that much in almost a year.
- ein0p 3y agoNot only that, in some cases they’re comparing apples to oranges as well, undermining their credibility further. Eg chain-of-thought vs non-CoT results. I don’t even know why they’re doing that, seems like their results would be impressive enough even without this.
- hiddencost 3y agoAlso one of the people who built the modern AI world. From the data centers to the bulk processing software to the team structures to tensorflow, to some of the most cost effective chips in the field, to many of the early blockbuster results in the field. They certainly fumbled by not investing in massive LLM scaling early enough, but Jeff Dean has been planning this day since his graduate work on neural networks in the 90s.
- Linda231 3y ago[dead]
- adamnemecek 3y agoAll machine learning is convolution.
- Legend2440 3y agoOnly certain types of neural networks use convolutions. It's not universal.
- CamperBob2 3y agoWhat are some examples of types that don't?
- hackerlight 3y agoI think anything that doesn't have an explicit convolution layer? Transformer, MLP, RNN don't automatically have a convolution layer, although for many tasks you can add it in if you want.
- dontwearitout 3y agoThe transformer module is currently dominating ML, and is widely used in text, vision, audio, and video models. It was introduced in 2017 and shows no real signs of being displaced. It has no convolutions. https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture) https://en.wikipedia.org/wiki/Transformer_(deep_learning_arc... http://jalammar.github.io/illustrated-transformer/ http://jalammar.github.io/illustrated-transformer/
- adamnemecek 3y ago… or does it https://grlearning.github.io/papers/11.pdf https://grlearning.github.io/papers/11.pdf
- dontwearitout 3y agoLots of interesting work out there; time will tell!
- rvnx 3y agoAt 1:00:18 it's claimed, "more than just publications" and that products are live for the public to use. And they mention something that sounds cool. It's called DermAssist, it's to find similar images related to skin-diseases. However, when checking the website, you are supposed to join a waitlist, but then it 404s. Is the tool already shutdowned or it was never released ?
- realprimoh 3y agoIt's not shutdowned I'm pretty sure. Looks like a bug. Hopefully it will be fixed soon.
- rvnx 3y agoLet's hope so, because it seems really useful. In the meantime, I found an open-source equivalent https://modelderm.com/en.html https://modelderm.com/en.html but if we could see the certified + tested tool it'd be nice :)
- dr_kiszonka 3y agoIf you are in the US, "DermAssist is not available in the United States"(https://health.google/consumers/dermassist/ https://health.google/consumers/dermassist/)
- htrp 3y agothats a new one... so where is it available?
- nsoldiac 3y agoHi, my partner worked in technology strategy for a large healthcare system where she saw many derm AI-based applications evaluated over the last decade. (Dis)Incentives aside, they all followed a similar\ story arc overpromising in their research findings and underdelivering in actual care delivery. They've been around for longer than you'd imagine. I hope they reach their potential, but suggest approaching them with healthy skepticism.
- nomilk 3y agoI see from comments I'm far from the only one using AI to summarise videos before deciding whether to watch them. Reminds of the meme "why spend 10 minutes doing something when you can spend a week automating it". i.e. "why spend an hour watching a talk when you can spend 5 hours summarising it with AI and debating the summary's accuracy". This sounds silly but potential gains from learning AI summarisation tooling/flows are large, hence why it warrants discussion. Learning how to summarise effectively might save hours per week and improve decisions about which sources deserve our limited time/attention.
- theGnuMe 3y agoWhat ai tool do you use to summarize?
- 63 3y ago> and debating the summary's accuracy Perhaps consider simply reading the description for an accurate summary. From the description: > Abstract: In this talk I’ll highlight several exciting trends in the field of AI and machine learning. Through a combination of improved algorithms and major efficiency improvements in ML-specialized hardware, we are now able to build much more capable, general purpose machine learning systems than ever before. As one example of this, I’ll give an overview of the Gemini family of multimodal models and their capabilities. These new models and approaches have dramatic implications for applying ML to many problems in the world, and I’ll highlight some of these applications in science, engineering, and health. This talk will present work done by many people at Google.
- nomilk 3y agoIn this case the video description contains a useful Abstract. AI summaries can offer additional value though, going into more/less detail (as desired), and allowing you ask follow up questions to drill into anything potentially of interest.
- swyx 3y agosure its an accurate summary, but is it at a granularity or specificity that you want? LLM summaries lets you move around the latent space of summaries and you probably dont agree with the one chosen for youtube descriprtions.
- deleted 3y ago[deleted]
- paxys 3y agoParts of it were good, but it mostly felt like he was reading through a slideshow created by the Google marketing team.
- throwaway4good 3y agoI watched some of this after it was promoted to me on YouTube. It is very much: Look at this other cool blackbox that Google has made - I think it has a helpful personality. And sure: Applied AI / ML is a part of computer science but I had hoped it would be more of a walk-through of what has happened in terms of advances in theory, algorithms, architecture, training methodology, and perhaps some sort of explanation of why does it work giving this general purpose model a bunch of medical data? (Is it merely an elaborate fuzzy search, how does it extrapolate, is there actually reasoning going on and how does emerge from a neural net etc...)
- knbknb 3y agoMaybe this was a review talk, designed for an audience of undergraduate students from various majors.
- babl-yc 3y agoAgreed -- given the talk is titled "Exciting trends in Machine Learning" it feels pretty incomplete to gloss over ChatGPT blowing up. OpenAI bet big on (1) emergent behaviors as you scale language models and (2) RLHF / fine-tuning to follow instructions. Both those topics were lightly covered but very much from the "Google did this" perspective (word2vec, step-by-step reasoning).
- lern_too_spel 3y agoThe original paper on emergent behaviors in LLMs is from Google: https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682 The difference between OpenAI and Google isn't due to different research directions between the two firms. It is a difference in ability to execute.
- 3y ago
- npalli 3y agoSeeing a lot of people talk about auto-summary tools which reminds me of a joke. “I took a speed-reading course and read War and Peace in twenty minutes. It involves Russia.” ― Woody Allen
- Mistletoe 3y agoThis is such a great comment about so many things right now. Thank you.
- idkdotcom 3y agoIt's a nice high level overview about the state of the art in machine learning. If you have watched his other talks, Jeff Dean generally does a very good job explaining things from a high level point of view. Here is a controversial view: I think that the current neural networks driven approach coupled with massively distributed computing has plateaued. For the machine learning field to move forward, it will need a different, less data/compute hungry paradigm.
- zyklonix 3y agoSummary: https://www.videogist.co/videos/jeff-dean-google-exciting-trends-in-machine-learning-6482 https://www.videogist.co/videos/jeff-dean-google-exciting-tr...
- nliang86 3y agoThanks for the post - I put together the tool above. I tried to strike a balance between being concise but also capturing all the important details. For that reason, the tool is hit or miss on longer (> 45 min) videos - the summary on this video is good but I've seen it omit important details on other long videos. The tool also captures relevant screenshots for each section. Hopefully it's helpful. You can summarize additional videos by submitting a youtube URL in the nav bar or the home page. Also, feedback welcome!
- crudalex 3y agoAre you using LLM to summarize? If yes can you share the prompts used?
- shuhongwu 3y ago[dead]
- denfromufa 3y agoThe highlight of this event was running with Jeff at Rice University before his talk: https://x.com/JeffDean/status/1756319820482592838?s=20 https://x.com/JeffDean/status/1756319820482592838?s=20
- opisthenar84 3y agoI'm surprised he didn't mention much about vector search
- H8crilA 3y agoIt is a finished thing, no? So many systems/products have it built in for quite a few years now.
- _t89y 3y agoThere's a typo in the title of the talk but don't worry I fixed it: Trends in Computer Vision ``In recent years, ML has completely changed our view of what is possible with computers'' In recent years ML has completely changed our view of what's possible with computer vision. ``Increasing scale delivers better results'' This is true for computer vision. ``The kinds of computations we want to run and the hardware on which we run them is changing dramatically'' Optimizations on operations for computer vision isn't exactly dramatic change. Who is We? Trends in Machine Learning, 2010: semantic search for advertising. Trends in Machine Learning, 2024: semantic search for advertising, short-form video content. [link redacted]
- _t89y 3y agolol. Three computer vision researchers dislike this comment. Do any of you want to respond to it?
- karmasimida 3y agoNot to shade on anyone, but even Jeff Dean isn't good at predicting the future. The pathways model/architecture aren't exactly where things are moving towards, LLM is. https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/ https://blog.google/technology/ai/introducing-pathways-next-...
- knowriju 3y agoCare to share some topics where the things are moving towards? I understand diffusion, GaNs and Mamba is in vogue these days, but those are different logical architecture. I am unsure where the next level ML physical architecture research is moving towards.
- karmasimida 3y agoI think at this rate, everything is moving towards Transformer based models(text/audio/image/video), as Sora has shown, there isn't really anything Transformer can't do, it can generate both real life quality photo and video. Its ability to fit ANY given distribution is beyond compare, the most powerful neural network we have ever designed, nothing else is even close. GANs are on the contrary, not hot any more in industry, diffusion models have achieved high fidelity in image generation, hard to see how GAN can make a comeback. It is faster, but it image generation in terms of quality is done, the wow factor is no more. This might be a hot take, but I think architectural changes is going to die down in industry, Transformer is the new MOS transistor. As billions of dollars pumping into making it runs faster AND cheaper, other alternative architecture is going to have a hard time compete.
- espadrine 3y agoThere is no question in my mind that the transformer architecture will not stop to evolve. Already now, we are stretching the definition by calling current transformers that way; the 2017 transformer had an encoder block which is nearly always absent nowadays, and the positional encoder and multi-head attention have been substantially modified. VRAM costs and latency constraints will drive architectural changes, which Mamba hints at: we will have less quadratic scaling in the architectures that transformers evolve into, and likely the attention mechanism will look more and more akin to database retrieval (for way more evolved database querying mechanisms than is often seen in relational databases). One day, the notion of a maximum context size will be archaeological. Breaking the sequentiality of predicting only the next token would also improve throughput which could require changes. I expect experts to also evolve into new forms of sparsity. More easily quantizable architectures may also emerge.