17 ms·
AI Canon
- natural219 3y agoSeems like a good list, I enjoyed the comic explainer of stable diffusion and learned a thing. Thank you to the authors and a16z for publishing this :).
- personjerry 3y agoNice, later this afternoon I'll have ChatGPT read these and summarize them for me
- CharlesW 3y agoHow do you think they wrote it? (Reminds me of this: https://marketoonist.com/2023/03/ai-written-ai-read.html https://marketoonist.com/2023/03/ai-written-ai-read.html)
- xpe 3y ago> Research in artificial intelligence is increasing at an exponential rate. Probably in the blundering sense of "exponential", meaning a lot. But what are some specific numbers? (such as publications)
- cubefox 3y agoNo recent data seems to be available. https://ourworldindata.org/grapher/number-artificial-intelligence-publications https://ourworldindata.org/grapher/number-artificial-intelli... Edit - Arxiv ML publications double every 23 months: https://twitter.com/MarioKrenn6240/status/1314622995139264517 https://twitter.com/MarioKrenn6240/status/131462299513926451...
- esafak 3y agoSee stat.ML in https://tableau.cornell.edu/t/PublicContent/views/arXivSubmissions/TableByArchives https://tableau.cornell.edu/t/PublicContent/views/arXivSubmi... via https://info.arxiv.org/about/reports/submission_category_by_year.html https://info.arxiv.org/about/reports/submission_category_by_...
- marcosdumay 3y agoSo, it's not even monotonically growing.
- dplavery92 3y agoEh, it's a little tricky. A lot of research marketed under the "AI" umbrella would be categorized under cs.LG (https://arxiv.org/list/cs.LG/recent https://arxiv.org/list/cs.LG/recent), cs.CV (https://arxiv.org/list/cs.CV/recent https://arxiv.org/list/cs.CV/recent), cs.CL (https://arxiv.org/list/cs.CL/recent https://arxiv.org/list/cs.CL/recent), and to a lesser degree cs.NE (https://arxiv.org/list/cs.NE/recent https://arxiv.org/list/cs.NE/recent). Oh, and of course, cs.AI (https://arxiv.org/list/cs.AI/recent https://arxiv.org/list/cs.AI/recent). Not every one of those areas has grown monotonically, but the growth in CV and CL especially has been explosive over the last ten years.
- dpflan 3y agoLooking at the authors, was this created by experts in AI? Is it sufficient to truly be a `canon`?
- whimsicalism 3y agoDoesn't seem like it to me and it skews very non-technical, but I can vouch for some of the sources they link - Sasha Rush's Annotated Transformer is great If I were to give a critique, it seems too skewed towards things business/product people would find interesting that aren't actually all that impactful (Tesla's self-driving, for instance) and also seems skewed to things I see in certain SF twitter bubbles. Directly linking lesswrong posts also seems a bit... cringe-y for a VC.
- WoahNoun 3y agoIt's just a list of links with no real substance. Don't they have some crypto scams to attend to?
- lucasmullens 3y agoThe links are the substance. Do you see no value in compiling a list of resources? They also explain why each article was included which helps quite a bit.
- rdli 3y agoI was an early member of the CNCF community (circa 2016), and at the time I thought "wow things are moving quickly." Lots of different tech was being introduced to solve similar problems -- I distinctly remember multiple ways of templating K8S YAML :-). Now that I'm spending time learning AI, it feels the same -- but the innovation pace feels at least 10x faster than the evolution of the cloud native ecosystem. At this point, there's a reasonable degree of convergence around the core abstractions you should start with in the cloud-native world, and an article written today on this would probably be fine a year from now. I doubt this is the case in AI. (Caveat: I've only been learning about the space for about 4 weeks, so maybe it's just me!)
- whimsicalism 3y ago> At this point, there's a reasonable degree of convergence around the core abstractions you should start with in the cloud-native world, and an article written today on this would probably be fine a year from now. I doubt this is the case in AI. It's a continuous process. It is way, way, way better than it was 8 years ago. Most of the frameworks can export models between each other/delta some layers, ONNX actually largely kinda works.
- quickthrower2 3y agoAlso 4 weeksish in. I am not a good future seer. I tried learning ML years ago but got bored. Not even stable diffusion budged me to even look at it again. But ChatGPT?!! Hell yeah I am motivated now! I will not be ashamed to say I am jumping on the bandwagon! More seriously I want to at least deeply understand the tech that will change our lives.
- oh_sigh 3y agoA16Z: Friendship ended with Blockchain. Now AI is my best friend. What's the last investment A16Z was actually ahead of the curve on? I guess it isn't important, since from their position, they don't rely on being ahead of the curve in order to make good investments, they make their investments good through their network and funding abilities.
- whimsicalism 3y agoYes, this is how money is made in finance. Outside of the major market makers, every financial firm is desperately trying to flee anything with any semblance of public liquidity because they can't beat the market makers & John Q Public without insider trading. Private tech firms are a great way to flee liquidity.
- Mistletoe 3y agoThe early investors still always make bank. A 5x is nice. >By Q1 this year, venture capital firm Andreessen Horowitz’s (a16z) flagship crypto fund had returned almost five times for early backers, according to documents reviewed by Semafor. The firm sold a portion of its tokens right before crypto’s bear market began in May, meaning that early investors are guaranteed a successful return. https://protos.com/the-crypto-bets-of-a16z-crumble-early-investors-still-profit/ https://protos.com/the-crypto-bets-of-a16z-crumble-early-inv... I'm reminded from one of my favorite quotes from Margin Call, one of my favorite movies. He's talking about the big Goldman-esque firm they work for, but it applies here too. "I've been at this company for 10 years, and I've seen things you wouldn't believe. When all is said and done, they do not lose money. They don't mind if everybody else does, but they don't lose."
- mtremsal 3y agoFrom the same movie, another quote that applies to VC trendsetting as a business model: “There are three ways to make a living in this business: be first; be smarter; or cheat. Now, I don't cheat. And although I like to think we have some pretty smart people in this building, it sure is a hell of a lot easier to just be first.” Not sure about cheating being off-limit given the crypto debacle…
- jhp123 3y agoIf you click the domain on this submission, you'll see loads of articles from a16z on the topic of generative AI. Click back a couple years and you'll find this page: https://news.ycombinator.com/from?site=a16z.com&next=29816846 https://news.ycombinator.com/from?site=a16z.com&next=2981684... with submissions like "DAOs, a Canon" https://news.ycombinator.com/item?id=29440901 https://news.ycombinator.com/item?id=29440901
- BobbyJo 3y agoTBF, the roll of a VC isn't to be on the cutting edge of science, but rather business, and generative AI is very new business, even if it isn't very new science.
- whimsicalism 3y ago> if it isn't very new science it's pretty new "science." at least last 8 years or so
- pvarangot 3y agohttps://archive.org/details/matrix_201703 https://archive.org/details/matrix_201703
- totetsu 3y agoI'm pretty sure I remember it being talked about on the tv show 'beyond 2000' in the 90s
- deleted 3y ago[deleted]
- whimsicalism 3y agogenerative AI?
- junofan 3y ago
- negamax 3y agoI am sorry but I am not a believer in a16z anything after their massive crypto token scams and wealth extraction. We all need to move away from all these companies who continue to bloat in private and then have a big pay day as a public company.
- tikkun 3y agoYes. I feel there should be some kind of anti-list of people who dumped SPACs and ICOs on the public. I personally feel it was pretty gross behavior (in many, but not all, cases), and the people who did the egregious ones mostly knew at the time that they were doing a zero sum wealth transfer to themselves. I personally avoid working with people who were involved in ICOs and SPACs where at the time of issuance a reasonable analysis could've shown that it was grossly overpriced to the public investors it was sold to (because in those cases, I believe that the issuer themselves should've known, and shouldn't have proceeded).
- whimsicalism 3y agoe: Removed, not sure I really agree with what I wrote on further reflection
- AJ007 3y agoThat just isn't a reasonable statement. No one forces someone to participate in a ponzi scheme. The problem with SPACS is these were companies that did not go the IPO route because they would not pass SEC approval. Companies should go public even if they are unlikely to survive, but what we saw was mostly fraud. They received absurd valuations based on exaggerated growth claims combined with imaginary non-GAAP accounting -- things you can't do in an IPO. Some of these participants will get in trouble. Enforcement is not immediate. a16z is probably going to end up in a lot of trouble over their cryptocurrency shenanigans. I think these guys pretty much burned their reputation in exchange for things like owning a $177m house in Malibu. The consequences will be felt by everyone, not just the shitco and shitcoin hucksters.
- davidhunter 3y agoI hope Tyler Cowen can ask Marc Andreessen how AI works so that we can all learn something from the master
- dpflan 3y agoWho is the master on how AI works?
- atlasunshrugged 3y agoI think this is referencing an interview where Tyler Cowen interviewed Andreessen and asked him directly about the true value of crypto and use cases and the response was... lackluster (imo). https://conversationswithtyler.com/episodes/marc-andreessen/ https://conversationswithtyler.com/episodes/marc-andreessen/
- dpflan 3y agoThe original comments seems sarcastic now. The answer that stands out here and is really the only answer from MA is: "Money". Whoever has the most tokens and those tokens increase in perceived value, benefits because they sell their tokens. It's not those little micropayments to podcaster (which are possible myriad ways without tokens); its the market forces driving up token price on tokens that were obtained at $0.00 by early investors...So they found a lucrative business of dumping tokens, that really seems all that web3 became. I will always cherish knowing about the Axie Infinity situation and crazy explanations for why that business model was the future!
- sharemywin 3y agoBuild AI or just invest in chip makers? https://a16z.com/2023/01/19/who-owns-the-generative-ai-platform/ https://a16z.com/2023/01/19/who-owns-the-generative-ai-platf... Over the last year, we’ve met with dozens of startup founders and operators in large companies who deal directly with generative AI. We’ve observed that infrastructure vendors are likely the biggest winners in this market so far, capturing the majority of dollars flowing through the stack. Application companies are growing topline revenues very quickly but often struggle with retention, product differentiation, and gross margins. And most model providers, though responsible for the very existence of this market, haven’t yet achieved large commercial scale. In other words, the companies creating the most value — i.e. training generative AI models and applying them in new apps — haven’t captured most of it
- gtirloni 3y agoSelling shovels continues to be profitable.
- whimsicalism 3y agoWould love to learn how a16z measures who is 'creating the most value.' My guess? Vibes and a conflation of "customer facing" and "value creation." And I would disagree - the chipmakers have produced most of the value and are reaping massive rewards right now. Certainly, the new LLM wrappers are not the value creators.
- sharemywin 3y agothe value creators are all the researchers that have invested untold hours into designing the models and collecting the data.
- whimsicalism 3y agoI think both those researchers and the researchers who did the same but with chip design are "value creators"
- ryanSrich 3y agoLooking at these comments, I can't think of another VC that has burned as much goodwill among technical people as a16z has. Don't get me wrong, it's well deserved, but it's just surprising how universal it seems to be (at least in this thread).
- whimsicalism 3y agoe: Removing because I think this is distracting from the main conversation & potentially flamebait.
- ChatGTP 3y agoWhy do you think that is ?
- superzamp 3y agoNow that we have good language models, it would be great to see what would a quantitative analysis of this perceived increase in ressentiment actually lands as results. I lean on agreeing with your comment, but also curious to see if we're both hallucinating.
- bluefishinit 3y agoWouldn't a more reasonable explanation be that a16z destroyed their brand through years of questionable investments, excessive hype, cringe inducing behavior and being very obnoxious, very publicly. Also not a fan of their push to "American Exceptionalism" aka weapons dealing and mass surveillance.
- whimsicalism 3y agoSure, except for the fact that I see this same trend of very young accounts making very negative comments for any sort of successful new tech trend.
- bluefishinit 3y agoI don't think anyone here is commenting on the AI trend, they're commenting on a16z.
- TradingPlaces 3y agoCame for everyone roasting a16z. Was not disappointed.
- whywhywhywhy 3y agoGetting whiplash from the 90 degree handbrake turn the crypto grifters have taken into being AI grifters.
- foobarbecue 3y agoYeah. What's the next grift? Maybe quantum computing?
- flerovium 3y agoClimate
- zeroxfe 3y ago> Andrej Karpathy was one of the first to clearly explain (in 2017!) why the new AI wave really matters. Geoff Hilton had been saying this well before 2017. I remember his talks at Google ~2013ish.
- jjoonathan 3y agoSchmidthuber did it in the 90s too, probably.
- etiam 3y agoOr 2007 https://news.ycombinator.com/item?id=90154 https://news.ycombinator.com/item?id=90154
- nologic01 3y agoNvidia is 25% up on "AI guidance", a16z publishes "AI Canon". Its settled -> AI is the new crypto
- moinnadeem 3y agoI would hold skepticism for the moment. I know the authors from the blog post quite well. Say what you will about the firm, but one of the authors have been investing in machine learning since 2016, and another has a PhD in CS (including a SIGCOMM test of time award!) I come from a strong ML background (multiple publications, PhD dropout), I would say that the canon is actually quite good.
- disgruntledphd2 3y agoI agree with both you and the commenters roasting a16z, tbh.
- JumpCrisscross 3y agoMedium matters. Anyone making public statements should understand as much.
- gist 3y ago> and another has a PhD in CS Sorry to say but 'big deal'. > one of the authors have been investing in machine learning since 2016 Ditto. I have been doing something (in another field) since the mid 90's. I would say most people would consider me an expert. I get referrals for what I do from 'top' people investors in tech. I also went to what most would consider 'a top college'. I would never want to be positioned as being right or expert because of the amount of time I spent doing something or the college that I went to, or who trusts me, but actual things that I have done that point to my expertise (not a halo of some type).
- lwneal 3y agoThis is a fine list, but it only covers a specific type of generative AI. Any set of resources about AI in general has to at least include the truly canonical Norvig & Russel textbook [1]. Probably also canonical are Goodfellow's Deep Learning [2], Koller & Friedman's PGMs [3], the Krizhevsky ImageNet paper [4], the original GAN [5], and arguably also the AlphaGo paper [6] and the Atari DQN paper [7]. [1] https://aima.cs.berkeley.edu/ https://aima.cs.berkeley.edu/ [2] https://www.deeplearningbook.org/ https://www.deeplearningbook.org/ [3] https://www.amazon.com/Probabilistic-Graphical-Models-Principles-Computation/dp/0262013193 https://www.amazon.com/Probabilistic-Graphical-Models-Princi... [4] https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf https://proceedings.neurips.cc/paper_files/paper/2012/file/c... [5] https://arxiv.org/abs/1406.2661 https://arxiv.org/abs/1406.2661 [6] https://www.nature.com/articles/nature16961 https://www.nature.com/articles/nature16961 [7] https://www.nature.com/articles/nature14236 https://www.nature.com/articles/nature14236
- hintymad 3y agoThe sad truth is that people nowadays can't even pass through a 15-minute podcast without checking out their Twitter feed multiple times. So, I'm not sure how many people would read through a 800-page textbook.
- javajosh 3y agoI think you'll find that the "screen brain" effect dissipates after about 20 minutes of discomfort. I've noticed this effect with novels and text books. Note that I don't think it's a great idea to just "read through" an 800 page text book even if you can - you've got to do exercises and check your own knowledge or else you will be spinning your wheels.
- hintymad 3y ago> I think you'll find that the "screen brain" effect dissipates after about 20 minutes of discomfort You mean one should persevere for more than 20 minutes and then can easily focus on the book? If so, that's great news! Note that I suffer from the "screen brain", but it's always good to know how brain works.
- king_magic 3y agoIt's almost offensive how this "AI Canon" leaves out landmark AI results from... what, before the 2020s? (or 2017, I suppose) Honestly reads like something generated by ChatGPT.
- alanpage 3y agoAnd why should we trust their judgement about anything, after they put money into Adam Neuman's new company (after the WeWork debacle)? CNBC: https://www.cnbc.com/2022/08/15/a16z-to-invest-in-adam-neumanns-new-residential-real-estate-company.html https://www.cnbc.com/2022/08/15/a16z-to-invest-in-adam-neuma...
- rchaud 3y agoThe article linked in the OP is mostly a list of other links you can visit to learn about the types of AI that are coming to market, and some background. Those links are not authored by A16z themselves. I generally would not read anything authored by A16Z partners, those just feel like bad inspirational speeches authored by Thomas Friedman.
- pseudostem 3y agoA thousand times echoing your sentiment, everyone on this thread seems dismissive, but the webpage linked carries some links which I would want to come back to later. If the source itself is a problem, we wouldn't want to listen to anyone for some reason or another.
- teej 3y agoAdam Neuman, with support from a16z, has an excellent track record returning capital to Americans while leaving Chinese & Saudi investors to hold the bag. That is why they invested.
- boringg 3y agoAnyone else feel like we've seen peak A16z at this point?
- EFreethought 3y agoOr is it nadir a16z?
- boringg 3y agomax or min -- good question.
- gist 3y agoPompous to use the word 'canon' to describe what amounts to a bunch of links and thoughts/opinions. Implies the authors of the various articles are the authoritative source/experts to which there is no point disagreeing.
- zxienin 3y agoLooking at the content list, all I need now is Brain - AI interface that “uploads“ all of it to my brain neural net. </cheeky>
- seydor 3y ago> Research in artificial intelligence is increasing at an exponential rate. but then most of the list is transformers & stablediffusion. Anyway, oobabooga and automatic1111 are doing more to spread AI than many of those papers.
- orsenthil 3y agoUsually this is written as awesome- list in a github repo.
- par 3y agoIt's a good resource but I hardly think a16z is the team to host the 'AI Canon'.
- uptownfunk 3y agoWow, why so much hate against a16z. There's a really funny clip about Marc on the Rogan podcast where he is like "I have to come on Rogan, there's so much clout" or something to that effect. Rogan was immediately like "igghh".
- kvetching 3y agoperfect timing i was just prescribed vyvanse + adderall
- boeingUH60 3y agoThese hucksters have found the next thing to latch on to, I see.
- dbs 3y agoI was quite surprised to see Sequoia getting involved in crypto fiascos. My data sample is very small but I have a pretty good track record of shifting career focus in the last 20 years. In particular, 2000 and 2008 were two HUGE shifts for me as the writing was on the wall before crisis hit. Common theme to drive change: too much competition. I jumped out from areas where there was still tremendous growth to be seen but no serious money to be done. I’m calling a third.
- davidatbu 3y agoDid you actually shift careers those two times in the past (Ie, change companies/jobs/job focus)? And will you be doing the same when "calling a third" now?
- mirekrusin 3y agoWhy everybody (including this a16z dude) underestimates/not mentions: 1. quality of input data - for language models that are currently setup to be force-fed with any incoming data instead of real training (see 2.) this is the greatest gain you can get for your money - models can't distinguish between truth and nonsense, they're forced to follow training data auto-completion regardless of how stupid or sane it is 2. evaluation of input data by the model itself - self evaluating what is nonsense during training and what makes sense/is worthy of learning - based on so far gathered knowledge, dealing with biases in this area etc. Current training methods equate things like first order logic with any kind of nonsense - having on its defense only quantity, not quality. But there are many widely repeated things that are plainly wrong. Simplifying this thought - if there weren't, there would be no further progress in human kind. We constantly reexamine assumptions and come up with new theories leaving solid axioms untouched - why not teach this approach/hardcode it into LLMs? Those two aspects seem to be problems with large gains, yet nobody seems to be discussing them. Align training towards common/self sense, good/own judgement, not unconditional alignment towards input data. If fine-tuning works, why not start training with first principles - dictionary, logic, base theories like sets, categories, encyclopedia of facts (omitting historic facts which are irrelevant at this stage) etc. - taking snapshots at each stage so others can fork their own training trees. Maybe even stop calling fine-tuning fine-tuning, just learning stages. Let researchers play with paths on those trees and evaluate them to find something more optimal, find optimal network sizes for each step, allow models to gradually grow in size etc. To rephrase it a bit - we're saying that base models learned on large data work well when fine tuned - why not base models trained on first principles can continue to be trained on concepts that depend on previously learned first principles recursively are efficient - did anybody try? As some concrete example - you want LLM to be good at math? Tokenize digits, teach it to do base-10 math, teach it to do addition, subtraction, multiplication, division, exponentiation, all known math basic operations/functions, then grow from that. You want it to do good code completion? Teach it bnf, parsing, ast, interpreting, then code examples with simple output, then more complex code (github stuff). Learning LLMs should start with teaching tiny model ASCII, numbers, basic ops on them, then slowly introducing words instead of symbols (is instead of =) etc., then forming basic phrases, then basic sentences, basic language grammar, etc. - everything in software 2.0 way - just throw in examples that have expected output and do back-propagation/gradient descent on it. Training has to have a way of gradually growing model size in (ideally) optimal way.
- SilverBirch 3y agoFrom the people who bought you web3. Look where the crowd is going, run to the front and shout "Follow me!".
- ssn 3y agoHope this is an "in progress" article. Not a single resource or pointer mentioning "ethics"?
- yellow_postit 3y agoI’d buy a copy of these all bound into a nice book as a point in time in the industry collectible.
- mark_l_watson 3y agoWell, that is a good list. I would guess that I have only previously read the content from about 15% of the links, oh well! Like everyone else, starting about a year and a half ago I have found it really difficult to stay up to date. I try to dive deep on a narrow topic for several months and then move on. I am just wrapping up a dive into GPT+LangChain+LlamaIndex applications. I am now preparing to drop most follows on social media for GPT+LangChain+LlamaIndex and try to find good people and companies to follow for LLM+Knowledge Graphs (something I tried 3 years ago, but the field was too new). I find that when I want to dive into something new the best starting point is finding the right people who post links to the best new papers, etc.
- Imnimo 3y agoThe trick here is that they get to put their own think pieces alongside actually influential work and pretend like the two deserve to share a stage.