10 ms·

I have to wonder if 10 years down the line, everyone will be able to run models like this on their own computers. Have to wonder what the knock-on effects of th

• espadrine 4y ago
  > I have to wonder if 10 years down the line, everyone will be able to run models like this on their own computers. Isn’t that already the case? Sure, it costs $60K, but that is accessible to a surprisingly large minority, considering the potency of this software.
  • alexb_ 4y ago
    ...what? 60 thousand dollars for a dedicated computer that you can't use is not everyone, not on their own computers, and is also a crazy large amount of money for nearly everyone. Sure there are some that could, but that's not what I said.
    • H8crilA 4y ago
      Indeed. What "everyone" can use is a ~$200 smartphone, so there's a ~300x gap to be bridged.
      • wjnc 4y ago
        log(300) / log(2) = only 8.2 doublings away. That's near future material.
        • thfuran 4y ago
          Maybe at 90s hardware growth rates, but not now.
          • rictic 4y ago
            The dream of the 90s is alive in the GPU market: https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-flops/ https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-fl... Moore's law didn't stop, just Dennard scaling. Expect graphics and AI to continue to improve radically in performance/price, while more ordinary workloads see only modest improvements.
            • thfuran 4y ago
              GPU TDP seems on the verge of going exponential, cost per transistor isn't really decreasing so much at the very latest nodes, and even that article seems to suggest it'd likely be decades before 300x flops/$
      • ben_w 4y ago
        Most of the cost of a phone isn’t the processor, so probably closer to x1000. Hardware may get that much cheaper, but it was never guaranteed, and we’re not making progress as fast as we used to.
    • paganel 4y ago
      Plus, that's the energy costs involved when running a computer now worth 60k, I'm pretty sure that in the current socio-economic climate those power costs will surpass the initial acquisition cost (those 60k, that is) pretty easily.
      • colechristensen 4y ago
        An 80GB nvidia A100 goes for $20k and uses 300 watts, the energy costs of using one (or three) isn’t going to surpass the hardware costs for… a while.
        • paganel 4y ago
          I wanted to add that I was writing it metaphorically in a way, as in, seeing as how high those energy bills will be they might as well all add up to 60k. Not sure about most of the people in here, but I would get really nervous at the thought of running something that eats up 3x300 watts per hour, for 24/7, just as part of a personal/hobby project. The incoming power bills would be too high, you have to be in the wage-percentile for which dropping 60k on a machine just to carry out some hobby project is ok, i.e. you’d have to be “high-ish” middle-class at least. The recent increases in consumer power prices are a heavy blow for most of the middle-class around Europe (not sure about how things are in the States), so a project like this one is just a no-go for most of middle-class European programmers/computer people.
          • colechristensen 4y ago
            At full power 3 of those would cost me ~$3.50 per day ($0.15 per kWh is what I paid for last month's electricity, though I could pay less if I made some difference choices), I occasionally have a more expensive coffee order, or have a cocktail worth three times as much. Things are getting more expensive here but nothing like the situation in Europe (essentially none of our energy was imported from Russia, historically ~10% of oil imports but that was mostly to refine and re-export, we have all the natural gas locally that we need) The US crossed the line into being a net hyrdocarbon energy exporter a while ago (unsure what the case is recently but it is at worst about at parity)
    • px43 4y ago
      Eh, 60k is just a bit more expensive than your average car, and lots of people have cars, and that's just how things are today. I imagine capabilities will be skyrocketing and prices will fall drastically at the same time.
      • fuzzer37 4y ago
        > 60k is just a bit more expensive than your average car If by "A bit" you mean about 30-40k
        • CRConrad 4y ago
          > > 60k is just a bit more expensive than your average car > If by "A bit" you mean about 30-40k 30k more expensive: Than your very-low-end-"average" car. 40k more expensive: Than your average used car. AFAICS. All in what one sees as an "average" car, I suppose.
  • kamray23 4y ago
    You're grossly overestimating. People who make 60k annually are getting a bit rarer nowadays, it's not like everyone can afford it. For the majority of people it'd be a multi-decade project, for a few it might only take 7 years, very few people could buy it all at once.
  • uniqueuid 4y ago
    Nitpick: This uses 8x A100 which are at least $10k a piece to my knowledge. Add in the computer and you're closer to $100k.
    • sascha_sl 4y ago
      And also, NVIDIA does not sell them to the consumer market whatsoever. Linus Tech Tips could only show one because someone in the audience sent theirs over for review.
    • taink 4y ago
      I believe you're confusing the amount of A100 graphics cards used to train the model (the cluster was actually made up of 800 A100s), and the amount you need to run the model : > The model [...] is supposed to run on multiple GPUs with tensor parallelism. > It was tested on 4 (A100 80g) and 8 (V100 32g) GPUs, [but should work] with ≈200GB of GPU memory. I don't know what the price of a V100 is, but given $10k a piece for A100s we would be closer to the $60k estimate.
      • uniqueuid 4y ago
        The $10k price is for an A100 with 40GB ram, so you need 8 of those. If you can get your hands on the 80GB variant, 4 are enough. Also, if you want to have a machine with eight of these cards, it will need to be a pretty high-spec rack-mounted or large tower. To feed these GPUs, you will want to have a decent amount of PCIe-4 lanes, meaning EPYC are the logical choice. So that's $20k for an AMD EPYC server with at least 1.6kw PSUs etc etc.
        • taink 4y ago
          Do you happen to know the cost of the 80GB variant?
          • colechristensen 4y ago
            Dell sells them for $20k
  • joshvm 4y ago
    You could just run this on a desktop CPU, there's nothing stopping you in principle, you just need enough RAM. A big memory (256GB) machine is definitely doable at home. It's going to cost 1-2k on the DIMMs alone, less if you use 8x32GB, but that'll come down. You could definitely do it for less than $5k all in. Inference latency is a lot higher in relative terms, but even for things like image processing running a CNN on a CPU isn't particularly bad if you're experimenting, or even for low load production work. But for really transient loads you're better off just renting seconds-minutes on a VM.
    • sascha_sl 4y ago
      From the readme, it looks like you need that RAM on your GPU.
      • joshvm 4y ago
        There isn't any reason you can't run a neural net on a CPU. It's still just a bunch of big matrix operations. The advantage of the GPU is it's a lot faster, but "a lot" might be 1 second versus 10 seconds, and for some applications 10 seconds of inference latency is just fine (I have no idea how long this model would take). All the major ML libraries will operate in CPU-only mode if you request it.
        • visarga 4y ago
          They are pretty slow even on GPU. The problem is that it's an autoregressive model. So it needs to do a forward pass for each token.
  • wellthisisgreat 4y ago
    What kind of computer would they be? Can you spec it out roughly?
• Byamarro 4y ago
  It could be possible with analog chips. I.e. ones that Mythic works on.
  • redox99 4y ago
    I'm not sure why you got downvoted. Yes, ASICs (either analog or digital) that have some model hardcoded in would probably make it feasible, but it won't be programmable which is the interesting part.
    • trasz 4y ago
      Totally not my field, but why wouldn't they be programmable? Analog FPGA's already exist.
      • redox99 4y ago
        Yes, true. I was referring to the Mythic ones the other comment mentioned which are only for inference of a specific model.
• tiborsaas 4y ago
  I think there will be a trend where model's size will shrink due to better optimization / compression while hardware specs keep increasing. You can already see this with Chinchilla: https://towardsdatascience.com/a-new-ai-trend-chinchilla-70b-greatly-outperforms-gpt-3-175b-and-gopher-280b-408b9b4510 https://towardsdatascience.com/a-new-ai-trend-chinchilla-70b...
• natly 4y ago
  I know it's a sort of exaggerated paranoid thought. But like these things do all come down to scale and some areas of the world definitely could have the amount of compute available to make dall-e level quality full scale videos which we might be consuming right now. It really does make you start to wonder at what point we will rationally be able to have zero trust that not everything we watch online is fabricated.
  • thelamest 4y ago
    Historically, hard-to-falsify documents are an anomaly, the norm was mostly socially conditional and enforced trust. Civilizations leaned and still lean on limited-trust technologies like personal connections, word of mouth, word on paper, signatures, seals, careful custody etc. I agree losing cheap trust can be a setback, just want to point out we’re adaptable.
• nonrandomstring 4y ago
  > one has to wonder what's real and what isn't. And whether it really matters. That's the bigger question. I think, for most of us, it does matter. But we're not sure why and what a loss of human reality would really mean. For a few who wholeheartedly embrace it there's some resonance with the psychedelic/60s creed that sees this as some kind of "liberation".
• arathore 4y ago
  If by running models you mean just the inference phase, then even today you can run large family of ML models on commodity hardware (with some elbow grease, of course). The training phase is generally the one not easily replicated by non-corporations.
• Comevius 4y ago
  That's definitely the future, personalized entertainment and social interactions will be big. I could watch a movie made for me, and discuss it with a bunch of chat bots. The future will be bubbly as hell, people will be decaying in their safe places as the hellscape rages on outside.
  • pydry 4y ago
    I get the feeling that creative sci fi used to kind of help inoculate us against these kinds of future but it seems like there's much less of it than there used to be. "Black mirror" was good but it's not nearly enough.
  • rasz 4y ago
    You really dont want to live in Mindwarp (1992 Bruce Campbell movie) or in this !114! year old short story https://en.wikipedia.org/wiki/The_Machine_Stops https://en.wikipedia.org/wiki/The_Machine_Stops
    • dTal 4y ago
      The Machine Stops is eerily prescient - or perhaps just keenly observant of trends visible even at the time - but in fairness the humans in it are not socially isolated, as such; they do not converse with bots, but rather with each other. The primary social activity in the The Machine Stops is the Zoom meeting. I do not look forward to the day when that story becomes an optimistic view of the future.
      • mrguyorama 4y ago
        That story is already an optimistic view compared to our own: They have no ads
  • Planktonne 4y ago
    > I could watch a movie made for me We're a long, long way from this. Stringing words/images together into a coherent sequence is arguably the easy bit of creating novels/films, and computers still lag a long way behind humans in this regard. Structuring a narrative is a harder, subtler step. Our most advanced ML solutions are improving rapidly, but often struggle with coherence over a single paragraph; they're not going to be doing satisfying foreshadowing and emotional beats for a while.
• psychoslave 4y ago
  I don’t know for you, but most of my online interactions are text based. Context of interpretation matter far much than the form of the content. If you know it’s easy to fake text exchanges, you might be more careful about text origin, and other contextual hints. Even it’s the syntax imitate your children verbal oddities, you may not necessarily run to comply thoughtlessly to an unusual demand you just receive by SMS from their phone number. Trust and check.
• ggktk 4y ago
  I'm predicting that the upcoming Mac Pro will be very popular among ML developers, thanks to unified memory. It should be able to fit the entire model in memory. Combine that with the fact that PyTorch recently added support for Apple silicon GPUs.
  • tehsauce 4y ago
    upcoming mac pro will have pretty poor ML performance when compared to even an old nvidia gpu sadly.
    • uniqueuid 4y ago
      Although memory capacity may matter more than speed for inference. As long as you're not training or fine tuning, the mac pro / studio may be just fine. apart from the fact that you can't use any of the many nvidia-specific things; if you're dependent on cuda, nvcuvid, AMP or other things that's a hard no.
      • calrizien 4y ago
        What are the current best ML language models to play with on the M1Max?
• pgt 4y ago
  The Move to the Edge is one of the strongest trends in technology. So, yes. I would never best against it. (applies to computing and other technologies like power production and agriculture)
  • user3939382 4y ago
    When I see AWS, cloud, and server side rendering frameworks it seems like we’re moving the other way in some sense.
    • ubercore 4y ago
      There's a strong trend to push to the edge of the cloud though -- cloudfront workers, deno.deploy, etc
• unixhero 4y ago
  Yes, the vision is that everyone has an AI cube in their house.
  • shaky-carrousel 4y ago
    Then, we'll hack all those cubes to build an AGI.
• dav_Oz 4y ago
  The bots/machine vs human reminds me of that famous experiment from the 30s in which Winthrop Kellogg[0], a comparative psychologist, and his wife decided to raise their human baby (Donald) simultaneously with a chimpanzee baby (Gua) in an effort to "humanize the ape". It was set out to last 5 years but was relatively quickly abrupted after only 9 months. The explicit reason wasn't stated only that it successfully proved the hereditary limits within the "nature vs nurture" debate of a chimpanzee, the reticent statement reads as follows: >Gua, treated as a human child, behaved like a human child except when the structure of her body and brain prevented her. This being shown, the experiment was discontinued There have been a lot of speculation as to other reasons of ending the experiment so prematurely. Maybe exhaustion. One thing which seemed to dawn on the parents - if one reads carefully - is that a human baby is far superior at imitating than the chimpanzee baby, frighteningly so, that they decided to abort the experiment early on in order to prevent any irreversible damage in the development to their human child which at that point had become far more similar to the chimpanzee than the chimpanzee to the human. So, I would rephrase "the internet is dead" into "the internet becomes increasingly undead" because humans condition themselves in a far more accelerated way to behave like bots than bots are potentially able to do. From the wrong side this could be seen as progress when in fact it's opposite progress. It sure feels like that way for a lot of of people and is a crucial reciprocal element often overlooked/underplayed (mostly in a benign effort to reduce unnecessary complexities) when analyzing human behaviour in interactions with the environment. [0]https://en.m.wikipedia.org/wiki/Winthrop_Kellogg#The_Ape_and_The_Child https://en.m.wikipedia.org/wiki/Winthrop_Kellogg#The_Ape_and...
  • gigglesupstairs 4y ago
    Wow this is such a mind bending perspective. Thanks for sharing it.
  • freewizard 4y ago
    So maybe the Turing Test is not about AI are smart enough, but about how stupid humans become?
    • rexpop 4y ago
• simonh 4y ago
  It's more likely, if not inevitable that these things will become ubiquitously available remotely, like Siri and Alexa. It's access that's important, not hosting.
• Jimmy 4y ago
  There’s a very simple solution, of course: turn off the computer and physically interact with real people.
• lostmsu 4y ago
  Running the models like this on own computer is already possible with DeepSpeed. I think it even supports training albeit it would be extremely slow. https://www.deepspeed.ai/ https://www.deepspeed.ai/
• zackmorris 4y ago
  Unpopular opinion: something will stop egalitarian power for the masses. I had high hopes for multicore computing in the late 90s and early 2000s but it got blocked every step of the way by everyone doubling down on DSP (glorified vertex buffer) approaches on video cards, leaving us with the contrived dichotomy we see today between CPU and GPU. Whatever we think will happen will not happen. A less-inspired known-good state will take its place, creating another status quo. Which will funnel us into dystopian futures. I'm just going off my own observations and life experience of the last 20 years, and the way that people in leadership positions keep letting the rest of us down after they make it.
  • nradov 4y ago
    In what sense is the dichotomy between CPU and GPU contrived? Those are designed around fundamentally different use cases. For low power devices you can get CPU and GPU integrated into a single SOC.
    • zackmorris 4y ago
      That's a good question. I wish I could answer it succinctly. For me, the issue is that use cases and power usage are secondary to the fundamental science of computation. So it's fine to have matrix-processing stuff like OpenGL and TensorFlow, but those should be built on general-purpose hardware or else we end up with the cookie cutter solutions we have today. Want to run a giant artificial life simulation with genetic algorithms? Sorry, you can't do that on a GPU. And it turns out that most of the next-gen stuff I'm interested in just can't be done on a GPU. There was a lot of progress on transputers and clusters (the old Beowulf cluster jokes) in the 80s and 90s. But researchers came up against memory latency issues (Amdahl's law) and began to abandon those approaches after video cards like the 3dfx Voodoo arrived around 1997. But there are countless other ways to implement concurrency and parallelism. If you think of all the techniques as a galaxy, then GPUs are way out at the very end of one spiral arm. We've been out on that arm for 25 years. And while video games have gotten faster (at enormous personal effort by millions of people), we've missed out on the low hanging fruit that's possible on the other arms. For example, code can be auto-parallelized without intrinsics. It can be statically analyzed to detect contexts which don't affect others, and the instructions in those local contexts could be internally spread over many cores. Like what happens in shaders. But IMHO the greatest travesty of the modern era is that those innovations happened (poorly) in GPUs instead of CPUs. We should be able to go to the system menu and get info on our computer and see something like 1024+ cores running at 3 GHz. We should be able to use languages like Clojure and Erlang and Go and MATLAB and even C++ that auto-parallelize to that many cores. So embarrassingly parallel stuff like affine rasterization and blitters would run in a few cycles with ordinary for-loops instead of needing loops that are unrolled by hand or whatever other tedium that distracts developers from getting real work done. Like, why do we need a completely different paradigm for shaders outside of our usual C/C++/C# workflow, where we can't access system APIs or even the memory in our main code directly? That's nonsense. And I don't say that lightly. My words are imperfect, but I do have a computer engineering degree. I know what I'm talking about, down to a very low level. Wherever I look, I just see so much unnecessary effort where humans tailor themselves to match the whims of the hardware, which is an anti-pattern at least as bad as repeating yourself. Unfortunately, the more I talk about this, the more I come off as some kind of crackpot as the world keeps rushing headlong out on the GPU spiral arm without knowing there's no there there at the end of it. My point is that for all the progress in AI and rendering and simulation, we could have had that 20 years ago for a tiny fraction of the effort with more inspired architecture choices. The complexity and gatekeeping we see today are artifacts of those unfortunate decisions. I dream of a day when we can devote a paltry few billion transistors on a small $100 CPU to 1000+ cores. Instead we have stuff like the Cerebras CS-2 with a trillion transistors for many thousands of dollars, which is cool and everything, but is ultimately gatekeeping that will keep today's Anakin from building C-3PO. https://en.wikipedia.org/wiki/Multi-core_(computing)#Hardware_examples https://en.wikipedia.org/wiki/Multi-core_(computing)#Hardwar...
• time_to_smile 4y ago
  Comments like this make me feel like I'm losing my mind. I think it's far more likely that in 10 years we'll all become more used to rolling blackouts, and fondly remember we all used to be able to afford to eat out, and laugh over a glass of cheap gin about how wild things were back in the old days before things got really bad. 10 years ago was a much more exciting and hopeful time than today. I remember watching Hinton show off what deep learning was just starting to do. It was frankly more interesting that high parameter language models. Startups were all working on some cool problems rather than just trying to screw over customers. That's just technology. Economically, socially and ecologically things looks far brighter in 2012 than they do now, and in 2032 I suspect we'll feel the same about today, but far more dramatically. We've already pass the peak of "things are getting better all the time!" but people are just in denial about this.
  • alluro2 4y ago
    You're not alone - especially based on observations during the pandemic, it seem we are woefully unaware and unprepared for how fragile the structures supporting our current way of living are, and how easy they would collapse into a much worse state of living conditions when it comes to power and food, let alone luxuries and internet as we know it... It also seems to me that most people would not be ready to give up more than 10% of their luxuries / way of living up-front in order to protect those structures and would continue to watch funny TikTok videos and post IG photos until the very moment their internet access goes out and doesn't come back.
• TuringNYC 4y ago
  >> I have to wonder if 10 years down the line, everyone will be able to run models like this on their own computers. Do you mean train or run? My assumption was all these models could be run on most computers, probably with a simple docker container, as long as there is sufficient RAM to hold the network, which should be most laptops > 16gb ram. Speaking of which, anyone have recommendations on pre-trained docker containers with weights included?
• deleted 4y ago
  [deleted]