33 ms·
YaLM-100B: Pretrained language model with 100B parameters
- schizo89 4y agoI hope one day it will be possible to run this kind of models at home.
- Akronymus 4y agoWell, it used to be impossible to render on anything not a mainframe in a reasonable time. The day will come when we will be able to.
- Hendrikto 4y agoIf you live in a datacenter, it already is!
- alexpotato 4y agoYou already have access to thousands of machine now from your home computer. Naval Ravikant put it best here: https://twitter.com/naval/status/1002106977273565184 https://twitter.com/naval/status/1002106977273565184
- lannisterstark 4y agoI was about to comment exactly the same thing. Stuff like this makes me feel so much behind because there's no way I can run this lol.
- f6v 4y agoThey hardware they mention can be rented from cloud providers. It’s just that it’s not very cheap.
- rocgf 4y agoWhen it will be possible to run this at home, the big companies will have models way bigger than this...
- rapnie 4y agoOr maybe the AI will own big companies that build bigger models for it. /s
- albertzeyer 4y agoIf your disk has enough space to store the model, I think in theory you could run them, using the disk to store states. But it will be slow. I'm not sure how slow though, and also if anyone has implemented this. It actually should not be too difficult.
- redox99 4y agoDisk makes no sense considering RAM is pretty cheap. But even then RAM is way too slow (and the communication overhead way too high). You probably get like a 100x slowdown or more.
- lostmsu 4y agoI think you are overestimating compute and I/O for this model. If you assume it is RAM bandwidth bound, with a single channel top DDR4 you will get inference time as a low multiple of 7 seconds (200GB/25GBs). In a workstation you can have 8 channels.
- justinlloyd 4y ago12-channels in mine. 24-channels on some configurations, though I think that is the upper limit at this time, with a maximum density of 512GB per channel.
- lostmsu 4y agoIs it multisocket?
- irthomasthomas 4y agoI think that unlikely. Barring some breakthrough that takes us beyond the limits of silicon.
- haswell 4y agoCouldn’t the same thing be said about most things we do on our phones these days? Won’t incremental advancement cover this eventually? (i.e. no major breakthrough required, just patience).
- cal85 4y agoSpeaking of which... I built a gaming PC a few years ago but I never use it these days. I want to install Linux on it and start playing around with machine learning. Can anyone recommend any open source machine learning project that would be a good starting point? I want one that does something interesting (whether using text, images, whatever), but simple/efficient enough to run on a gaming PC and see some kind of results in hours, not months. I'm not sure what I want to do with ML yet, I just know I'm interested, and getting something up and running is likely to enthuse me to start playing and researching further. My spec is: GeForce RTX 2080 Ti (11GB), a 24-core AMD Ryzen Threadripper, and 128GB RAM. I'd be willing to spend on a new graphics card if it would make all the difference. I am a competent coder and familiar with Python but my experience with ML is limited to fawning over things on HN. Any recommendations gratefully received!
- schizo89 4y agoI would recommend auditing Stanford courses in following order: 1. CS231n Machine Vision https://www.youtube.com/playlist?list=PLkt2uSq6rBVctENoVBg1TpCC7OQi31AlC https://www.youtube.com/playlist?list=PLkt2uSq6rBVctENoVBg1T... 2. CS234 Reinforcement Learning https://www.youtube.com/watch?v=FgzM3zpZ55o&list=PLoROMvodv4rOSOPzutgyCTapiGlY2Nd8u https://www.youtube.com/watch?v=FgzM3zpZ55o&list=PLoROMvodv4... 3. CS330 Meta Learning https://www.youtube.com/watch?v=0rZtSwNOTQo&list=PLoROMvodv4rMC6zfYmnD7UG3LVvwaITY5 https://www.youtube.com/watch?v=0rZtSwNOTQo&list=PLoROMvodv4... Those will get you on track with general concepts about reasoning, AI engineering and concepts of learning itself Language models for me a bit of headache because there're in different domain on intersection with linguistics and humanities but here's a good course https://www.youtube.com/watch?v=rha64cQRLs8&list=PLoROMvodv4rPt5D0zs3YhbWSZA8Q_DyiJ https://www.youtube.com/watch?v=rha64cQRLs8&list=PLoROMvodv4... Those are all free and high-quality but require a lot of brain power
- londons_explore 4y agoFor those of us without 200GB of GPU RAM available... How possible is it to do inference loading it from SSD? Would you have to scan through all 200GB of data once per character generated? That doesn't actually sound too painful - 1 minute per character seems kinda okay. And I guess you can easily do lots of data parallelism, so you can get 1 minute per character on lots of inputs and outputs at the same time.
- julienfr112 4y agoWhat about 250gb of ram and use a cpu ?
- hnechochamber2 4y ago$ dd if=/dev/zero of=/swapfile bs=1G count=250 status=progress $ chmod 600 /swapfile $ mkswap -U clear /swapfile $ swapon /swapfile
- jstimpfle 4y agoIs there a reason why it is required to fill the swapfile with zeroes here? Normally you'd see something like "dd of=/swapfile bs=1G seek=3 count=0", creating a file of size 3G but with no space allocated (yet). It's much quicker to complete the setup this way.
- wongarsu 4y agoI assume if you force the file system to allocate inodes you are likely to have a less fragmented file than if you create a sparse file that gets inodes assigned over time when each part is used.
- theblazehen 4y agoWhich won't matter on SSDs
- egorfine 4y agoI have huge respect for developers at Yandex. It's kind of sad that achievements like these are tainted by the fact that they come from Russia (and I speak as a Ukrainian). I wonder if the permissive license is able to mitigate that.
- f6v 4y agoThe achievements aren’t in any way tainted by their nationality, citizenship, sex, sexual orientation, age, etc.
- geek_at 4y agothe age old question if the art should be linked or disconnected from the artist
- rapnie 4y agoand if technology is neutral.
- somenameforme 4y agoThis is nothing like that, because the question is not one of their own personal actions - but of their nationality or ethnicity. That, until about 4 months ago, would have been widely acknowledged as racism. The difference between holding values, and holding values when convenient rather sums up the entirety of human history in one phrase.
- trasz 4y agoYeah, it's certainly about ethnicity, not at all about being controlled by a government which is in the process of perpetrating genocide.
- 5e92cb50239222b 4y agoI'm ethnically Russian (mostly), although I've never been to that country and have less influence on their foreign policy than your average European (who at least has some say in how his own country behaves towards Russia — and we've seen how well they managed that). I don't know how this would translate to the real world if I lived in "the West", but from what I'm seeing on the internet for the past few months, it definitely is about ethnicity. I've been called many things and blamed for everything bad that has happened since 1945, and not many seem to care that half of Putin's army consists of people of Asian and Caucasian ethnicities, and there are many Russians in the Ukrainian army. If you go to places like r/worldnews, there are open calls for violence that have strong fascist overtones, and those seem to be getting more popular. Can't say the same about HN, it's one of the few places that seems to have kept its sanity (for now?)
- alexb_ 4y agoI have to wonder if 10 years down the line, everyone will be able to run models like this on their own computers. Have to wonder what the knock-on effects of that will be, especially if the models improve drastically. With so much of our social lives being moved online, if we have the easy ability to create fake lives of fake people one has to wonder what's real and what isn't. Maybe the dead internet theory will really come true; at least, in some sense of it. https://www.theatlantic.com/technology/archive/2021/08/dead-internet-theory-wrong-but-feels-true/619937/ https://www.theatlantic.com/technology/archive/2021/08/dead-...
- espadrine 4y ago> I have to wonder if 10 years down the line, everyone will be able to run models like this on their own computers. Isn’t that already the case? Sure, it costs $60K, but that is accessible to a surprisingly large minority, considering the potency of this software.
- alexb_ 4y ago...what? 60 thousand dollars for a dedicated computer that you can't use is not everyone, not on their own computers, and is also a crazy large amount of money for nearly everyone. Sure there are some that could, but that's not what I said.
- ketzu 4y agoSeeing those gigantic models it makes me sad that even the 4090 is supposed to stay at 24GB of RAM max. I really would like to be able to run/experiment on larger models at home.
- EugeneOZ 4y agoCan Apple Silicone's unified memory be an answer?
- MrBuddyCasino 4y agoWondering if Apple Silicon will bring arge amounts of unified main memory with high bandwidth to the masses? The Mac Studio maxes out at 128GB currently for around $5K, so 256GB isn't that far out and might work with the ~200GB Yandex says is required.
- Havoc 4y agoPerhaps on quantity. Substantially slower though around ~3x from what I can tell…substantial roadblock if you’re training models that take weeks.
- MrBuddyCasino 4y agoI meant for inference, not training. People just want to run the magic genies locally and post funny AI content.
- Havoc 4y agoah right - gotcha
- out_of_protocol 4y agoTake a look at Apple's M1 Max, a lot of fast unified memory. No idea how useful though
- 4y ago
- m00dy 4y agowell, I can call this "the real open ai".
- JeopardyJJJ 4y ago
- drno123 4y agoWho controls Google? What did Google do to stop the inasion of Iraq? Will Google take responsibility for silent support od war in Iraq?
- csee 4y agoYour comparison fails a test of facts. Yandex actively censors any perspective not approved by the Kremlin. Google does not do anything comparable to this.
- tremarley 4y agoGoogle absolutely does the same thing.
- baisq 4y agoNot to mention that Yandex does it in Russia because the law forces them to, while Google does it happily just to maintain the political status quo, of which they are a part of.
- SXX 4y agoGoogle does not show government propoganda on search engine front page. If Yandex wanted to shut down their news aggregator they could have done it.
- ptnxlo 4y agoCare to elaborate?
- londons_explore 4y agoThere is lots of content Google bans/hides. Copyrighted content, Adult content, child pornography, official secrets, etc. I don't think thats so different from other countries which also have a (partially overlapping) list of whats not allowed. Normally, when people think about that they say "well pictures of naked children are morally wrong, whereas talking about LGBTQ stuff is fine". But people in other parts of the world might have different morals and might think the other way around.
- edf13 4y agoWonder what the split is between Russian and English in the model?
- londons_explore 4y agoOpen the vocab file (from the script in the download directory) and you can get a pretty good idea. Looks to be approximately 50/50 from my random scrolling through the list.
- f311a 4y agoThat's because English and Russian have pretty similar vocabulary size. Vocabulary does not reflect the size of the data.
- londons_explore 4y agoIn this case, it does, because the vocab is not a list or words, but a list of tokens. Each token may be a word, but it might also be a phrase or part of a word. The tokens are generated to be optimal on the input data - ie. for a given vocab size to minimize the number of tokens to represent it. Therefore, the size of the vocab gives a good guide to the size of the data, since if there was 10x more english language data then the optimal distribution would be to dedicate more token space to english than russian.
- throwaway4good 4y agoWhat sort of machine can run this model?
- f311a 4y agoNvidia DGX
- throwaway4good 4y agoI can see it is about 150.000 USD for such a machine. Is this the cheapest option out there?
- kamray23 4y agoWell you can custom-build a suitable system for the middle five digits. It's still not something every idiot can run, but most medium to large companies can set up their own for sure.
- wongarsu 4y agoNow we just need someone to figure out how to compress the model to get similar performance in 10B parameters. I assume some of the services that offer GPT-J APIs will pick this up, but it doesn't look cheap or easy to get this running.
- londons_explore 4y agoThe download fails because the vocab file link returns HTTP 403... :-( https://yalm-100b.s3.yandex.net/vocab/voc_100b.sp https://yalm-100b.s3.yandex.net/vocab/voc_100b.sp EDIT: It seems fine if you download with a browser useragent not CURL... I guess I just got hit by some anti-bot thing they have accidentally have turned on.
- lukestateson 4y ago[flagged]
- denysvitali 4y agoI don't want to defend them, but I'm genuinely curious: aren't they maybe doing it because the opposite will cause them huge legal issues?
- lukestateson 4y agoThey can protest, they can boycott, they can disagree, they can tell the truth. But... They chose to obey. It's the choice that matters.
- denysvitali 4y agoFrom what I've seen, telling the truth in authoritian countries doesn't end up well. To the best of my knowledge, they are a Russian company - it's not like they can just tell the truth and move away from Russia that easily, so I think (and hope?) they're just playing a political game. What would Google do in their position? Idk
- fabrika 4y agoThey will simply have their company taken away from them. Nevertheless, they had many years before the war to start marking their news as 'Official'. Or sell the news service. They certainly could have done so. This would have solved their image problems.
- lukestateson 4y agoIt's already taken away.
- jeroenhd 4y agoThis is often overlooked and it's a fair point in defence of the people working for Yandex. You can't judge someone just for working for Yandex or even most Russian companies. The people who have voiced concern are already out of the company and it's perfectly reasonable that the rest would like to keep their jobs, especially in uncertain economic times with all these sanctions against Russia. However, this also implies that Yandex, as a company, cannot be trusted. It's not the researcher's fault, but they simply aren't allowed to work in a way that doesn't reinforced the Russian government's bias. As usual, the Russian government is the real villain here, but its authoritarian rule "infects" any company and country it has control over. It can be assumed that the people working for Yandex are also victims of their abusive government, but that doesn't change the fact that their work is unlikely to be trusted outside the Russian sphere of influence.
- deleted 4y ago[deleted]
- keewee7 4y ago
- the_duke 4y ago> The former Yandex CEO literally moved to Israel to escape Western sanctions. That somehow doesn't support the point your are trying to make at all...
- Reubachi 4y agoI've no horse in this race, so please don't take this the wrong way, Escaping western sanctions by moving to Israel + Being a Russian shill are usually mutually inclusive, not exclusive things.
- lotusmars 4y ago
- huma 4y agoNot for being Russians, but for active participation in censorship by tweaking their news aggregation to show only hand picked government approved sources
- memorable 4y agoIs there a source for this? I'm curious.
- SXX 4y agoJust read about Yandex.News which display censored news sources on Yandex frontpage that millions of people visit. There is really no hidden censorship here - they just follow Russian law that literally whitelist exclusively press controlled by the state. They show Kremlin propoganda on their front page which makes Yandex part of Kremlin propoganda machine. They could have shut down news agregator, but they choose not to.
- proxysna 4y agohttps://www.kommersant.ru/doc/4975254 https://www.kommersant.ru/doc/4975254 https://www.moscowtimes.ru/2022/04/18/yandeks-ubral-izpoiskovoi-vidachi-stranitsi-sotssetei-ekstremistov-ismi-inoagentov-a19651 https://www.moscowtimes.ru/2022/04/18/yandeks-ubral-izpoisko... https://zona.media/news/2021/12/25/oiya https://zona.media/news/2021/12/25/oiya https://www.rbc.ru/politics/07/09/2021/613709739a79476fd52e153a https://www.rbc.ru/politics/07/09/2021/613709739a79476fd52e1... They are complying with russian censorship laws and it's gotten so bad that they are planning to sell the news service altogether to VK which is far worse than Yandex when it to how eager they are to enforce these laws and to work with cops. https://www.kommersant.ru/doc/5258943 https://www.kommersant.ru/doc/5258943
- pembrook 4y agoSide note: Yandex search is awesome, and I really hope they stay alive forever. It's the only functional image search nowadays, after our Google overlords neutered their own product out of fear over lawyers/regulation and a disdain for power users. You can't even search for images "before:date" in Google anymore.
- whywhywhywhy 4y agoYandex Image Search is today is what Google Image Search should have been. End of the day I’ll use what actually gets the job done. Same goes for OpenAI and Google AI. If you don’t actually ever release and let others use your stuff and end paralyzed in fear at what your models may do then someone else is gonna release the same tech, and at this rate it seems like that’ll be Chinese or Russian companies who don’t share your sensibilities at all, and their models will be the ones that end up productized.
- SXX 4y agoOpenAI should just rebrand since nothing they do is actually open.
- daniel-cussen 4y agoYou know 100 years ago you could just buy uranium openly? Leo Szílard hustled up 200 kilograms, pleted, in the 30's.
- SXX 4y agoWhat does it have to do with OpenAI branding? Their "moral" reasoning behind not publishing models is simply laughtable because they do sell API access to them to anyone who can pay. And "bad guys" generally have money.
- sanxiyn 4y agoThey can (and do) revoke API access from bad guys. They can't do that to downloaded models. Look, I don't like what OpenAI does, but "API access, but no model download" makes sense if you are worried about misuses.
- kome 4y agoI agree, yandex is a great search engine
- lotusmars 4y agoWell you can have it in the West. We'd prefer something separate from Kremlin.
- zeofig 4y agoWe can have what, a great search engine? Maybe if you have a time machine to 2003
- lotusmars 4y agoJust take it to America from us, thank you. Along with VK. Great search engine and a social network. Full of backdoors for thugs and corrupt police, censorship and other lovely stuff... but you'll probably say that Google is full of it too, because you had no experience of living in Russia.
- speed_spread 4y ago2003 certainly didn't have a better search engine. It only had a much smaller, open and un-SEO-biased Internet, making the indexing job correspondingly easier.
- SXX 4y agoFirst of all regardless for political situation this is great step in making ML research actually open. So huge thanks for those developers who pushed to make it public. Still... Yandex is in fact share responsibility for Russian government actions. While it impossible to fight censorship they could certainly shut down their News service completely. Yandex could also certainly move more of their company and staff out of country. It was their deliberate choice stay in Russia and getting advantages on local market by using their political weight.
- lukestateson 4y ago[flagged]
- memorable 4y agoSource for all the claims?
- OmicronCeti 4y ago2) https://www.businessinsider.com/yandex-russia-former-news-director-urges-colleagues-quit-ukraine-invastion-2022-3 https://www.businessinsider.com/yandex-russia-former-news-di... >The ex-head of news at Russia's largest internet company has advice for his former colleagues: quit. >Lev Gershenzon worked at Yandex in various roles for four years, according to his LinkedIn profile. He took to Facebook early Tuesday morning to warn people still working at the company — which is one of the largest search engines in Russia — that it was contributing to the censorship of the country's invasion into Ukraine. >"The fact that a significant part of the Russian population may believe that there is no war is the basis and driving force of this war," Gershenzon wrote, also tagging six of his former coworkers. "Today, Yandex is a key element in hiding information about war. Every day and hour of such "news" costs human lives. And you, my former colleagues, are also responsible for this." 2) https://techcrunch.com/2022/03/16/russia-yandex-news-vk/ https://techcrunch.com/2022/03/16/russia-yandex-news-vk/ >Yandex’s former head of news accused the company of being a ‘key element in hiding information’ from Russians about the war in Ukraine. 3) Result of Yandex's slower crawler and default display mode, although the effect is as described: https://twitter.com/maryilyushina/status/1510930537187319813?lang=en https://twitter.com/maryilyushina/status/1510930537187319813...
- sgc 4y agoThose are trivial to verify. If you can't do your 5 minutes of research, OP should not feel obliged to humor your attempt to create busy work for them.
- Vaslo 4y agoPlenty of people cite their work and arguments in contentious discussions.
- htrp 4y ago> It was tested on 4 (A100 80g) and 8 (V100 32g) GPUs, but is able to work with different configurations with ≈200GB of GPU memory in total which divide weight dimensions correctly (e.g. 16, 64, 128). so we looking at crazy prices just for inference. RIP to the first guy's cloud billing account who makes this public
- jhoelzel 4y agoso err the cheapest A100 i could find was EUR 10.579,79 . Suddenly that 3090 i wanted to get, does not seem so expensive....
- chrisMyzel 4y agoWith DeepSpeed an SSD and lots of RAM you should be able to run inference with a 8GB card. There’s a thread somewhere on https://discuss.huggingface.co/ https://discuss.huggingface.co/ doing the math around this
- sandGorgon 4y agois this the first GPT-like models which is fully opensource ? none of the others are right ?
- littlestymaar 4y agoAren't eleutherai's model so?
- sandGorgon 4y agodoesnt seem the code is there - pretrained models are there. https://github.com/kingoflolz/mesh-transformer-jax/#gpt-j-6b https://github.com/kingoflolz/mesh-transformer-jax/#gpt-j-6b https://huggingface.co/EleutherAI/gpt-j-6B https://huggingface.co/EleutherAI/gpt-j-6B isnt that so ?
- p1esk 4y agoThe code is there: https://github.com/EleutherAI/gpt-neox https://github.com/EleutherAI/gpt-neox
- sandGorgon 4y agooh pretty cool thanks. i was looking at the gpt-j-6B. but this is pretty cool.
- manishsharan 4y agoIs there a way for developers, who do not have AI/ML background, to get started using this ? I have been curious about GPT-3 but I do not have any AI/ML experience or knowledge. Is there a "approachable" course on Coursera or Udemy that could help me get started with technologies like GPT ?
- MichaelRazum 4y agoIt's just crazy how much it costs to train such models. As I undestand 800 A100 cards would cost about 25.000.000 without considering the energy costs for 61 days of training.
- refulgentis 4y ago16,000,000 at MSRP
- StevenWaterman 4y agoLambda labs will rent you an 8xA100 instance for 3 months for $21,900. That would put it at around $2m
- MichaelRazum 4y agoStill a bit to expensive for my sideproject ; ) To be honest it seems only big corp can do that kind of stuff. By the way if try to do hyper parameter tuning or some exploration in the architecture it becomes guess 10x or 100x more expensive.
- bmcahren 4y agoAWS has them in US-EAST1 for $9.83/hr spot with 96 CPU cores, 1152GB of ram, 8 A100s with 320 GB of RAM, 8TB of NVME, and 19 Gbps of EBS bandwidth to load your data quickly. https://aws.amazon.com/ec2/instance-types/p4/ https://aws.amazon.com/ec2/instance-types/p4/ p4d.24xlarge An alternative is the p3.16xlarge for 8 V100s with 256GB of GPU RAM but you might as well get the A100s since it's only $0.50/hr cheaper
- semitones 4y agohttps://coreweave.com/ https://coreweave.com/ offers some of the cheapest GPU compute out there
- idealmedtech 4y agoIn the download script, it skips parts of the model (02 and 83); any ML people have ideas why you'd do that?
- hansonw 4y agoIt appears the indexing for the model parts is deliberately not contiguous; the 03-82 range represents the main 80 transformer layers. https://github.com/yandex/YaLM-100B/blob/main/megatron_lm/megatron/checkpointing.py#L425-L436 https://github.com/yandex/YaLM-100B/blob/main/megatron_lm/me...
- idealmedtech 4y agoThat makes sense, thanks for clearing it up!
- obituary_latte 4y agoWhat are some use cases for something like this? I understand it says "generating and processing text", but is it a replacement for OCR? Or something else?
- sharmin123 4y ago
- ma2rten 4y agoI am one of the people who worked on Google's PaLM model. Having skimmed the GitHub readme and medium article, this announcement seems to be very focused on the number of parameters and engineering challenges scaling the model, but it does not contain any details about the model, training (learning rate schedules, etc.), or data composition. It is great that more models are getting released publicly, but I would not get excited about it before some evaluations have been published. Having a lot of parameters should not be a goal in and of itself. For all we know this model is not well trained and worse than Eleuther AI's 20B parameter model, while also being inconveniently large.
- javajosh 4y ago> this announcement seems to very focused on number of parameters And yet your own project headline is "Pathways Language Model (PaLM): Scaling to 540 Billion Parameters for Breakthrough Performance"[0]. 0-https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html https://ai.googleblog.com/2022/04/pathways-language-model-pa...
- panda-giddiness 4y ago1. The OP did not criticize the headline; they criticized the content. If you read the article that you linked, you would find that they do, in fact, evaluate the performance of the model. 2. 540 billion parameters is notable for its size, which is likely why they lead with that particular headline.
- gwern 4y agoThe difference is PaLM was extensively benchmarked and it performed as well as it should, which is to say, amazingly well. The irony here is that you should instead be invoking that other ~500b model, Nvidia's Megatron-530b, which was undertrained, only cursorily evaluated (no interest in any new capabilities or even examining old ones like inner monologues) and promptly forgotten by everyone after the headlines about being the largest dense model: https://arxiv.org/abs/2201.11990#microsoftnvidia https://arxiv.org/abs/2201.11990#microsoftnvidia
- 4y ago
- joshsyn 4y agoYandex > Google.
- upupandup 4y agoDoes anybody want to crowd fund the training?
- qwertywert_ 4y agoAlready trained, but still need some ~200GB GPU mem to run the model.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- jamix 4y ago
- orbital-decay 4y agoWhat does it tell you? I'm seeing mostly pictures of destruction and mass graves for both Bucha and Буча.
- OneLessThing 4y agoIf you read the resulting articles you'll find a few of them suggest that all the deaths were staged or committed by Ukrainians. Headlines like "The truth is out there..." or "Global lies..." are examples. There still are many results from mainstream western media on the other hand. Google, in contrast, has zero results implying the deaths were staged or committed by Ukrainians.
- justinzollars 4y agoWhat is the TLDR on this model? What exactly does it do? Its not clear from the source examples.
- option 4y agoDid they bias it toward ru propaganda talking points? Edit: I would like to see more details in addition to size and languages (en, ru) about training data. For example, did they use their own Yandex.news (a cesspool of propoganda)?
- dang 4y agoYou've made a version of this comment 3 times in this thread now. It's shallow and flamebaity, and the repetition just adds noise and does no good, so please don't keep doing that. I understand the strong feelings, but the rules still apply—in fact that's when they apply most. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- option 4y agoThanks for reminder, I deleted other two comments which were more flamebaity. My overall point still stands - they did not give any details other than size on the training data. This is crucial (I train LLMs for a living)
- lumost 4y agoTo add a voice of skepticism. The recent rush to open source these models may be indicative that the tens of millions that’s spent training these things has relatively poor roi. There may be a hope that someone else figures out how to make these commercially useful.
- mumblemumble 4y agoFrom what I've seen, using these huge models for inference at any kind of scale is expensive enough that it's difficult to find a business case that justifies the compute cost.
- f311a 4y agoYandex uses it for search and voice assistant
- sailingparrot 4y agoThose models aren't trained with the objective of being deployed in production. They are trained to be used as teachers during distillation into smaller models that fit the cost/latency requirements for whatever scenario those big companies have. That's where the real value is.
- gfodor 4y agoAn equally plausible frame is that once a technology becomes replicated across several companies, it makes sense to open source it since the marginal competitive advantage are the possible resultant external network effects. I don't know if that's the right way to think about the open sourcing of large language models. I just think we really can't read too much into such releases regarding their motivation.
- jenny91 4y agoYes, commoditize your complements.
- MivLives 4y ago
- amai 4y agoIs that the model used by the russian government to generate fake news?
- Destiner 4y agoYou don’t need ai for that, tons of ppl here in russia will do it for pennies.
- narrator 4y agoI love Yandex. They are the best search engine by far for politically controversial topics. They also release a language model to benefit everyone even if it says politically incorrect stuff. They also name their projects "cocaine" probably to perhaps to prevent western competitors from using them. You look at OpenAI and how they don't release their models mainly because they fear "bad people" will use them for "bad stuff." This is the trend in the west. Technology is too powerful, we must control it! Russia is like... Hey, we are the bad guys you're talking about so who are we keeping this technology from? The west has bigger language models than we do, so who cares. Also their attitude to copyright and patents, etc. They don't care because that's not how their economy makes money. Cory Doctorow's end of general purpose computing[1] and locked down everything is very fast approaching. I'm glad the Russians are around and aren't very interested in that project. [1]https://csclub.uwaterloo.ca/resources/tech-talks/cory-doctorow-the-war-on-general-purpose-computing/ https://csclub.uwaterloo.ca/resources/tech-talks/cory-doctor...
- abra0 4y ago>They are the best search engine by far for politically controversial topics. This is an interesting take given the political censorship in Russia (for some ineffable reason much harsher now than it used to be 4 months ago) and cases like https://twitter.com/kevinrothrock/status/1510944781492531208 https://twitter.com/kevinrothrock/status/1510944781492531208.
- narrator 4y agoSearch Google and Yandex for "2020 election fraud." The results are VERY different. The Zach Vorhies leak shows that Google regularly does blatant censorship for political purposes.[1] [1]https://www.breitbart.com/tech/2021/08/19/google-whistleblower-search-engine-rewrote-algorithms-to-go-after-trump/ https://www.breitbart.com/tech/2021/08/19/google-whistleblow...
- skrebbel 4y agoI don't know man, "thegatewaypundit.com" as a top reputable source? seems to me like it's not "honest two-sided results" but just, well, a rather random mix of result of widely varying quality. Mad Altavista vibes! What I'm trying to say is that even if you believe that "was the 2020 US election stolen?" is worth debating, which it isn't, the yandex results are shit.
- lukestateson 4y ago1. Yandex supports the Russian Terrorist regime. 2. Yandex News service ignores the genocide currently happening in Ukraine. 3. Yandex Search engine hides the pictures of Bucha and Irpin massacre as well as Kharkiv and Mariupol destruction. Yandex using whitewashing tactics via open source.
- lukestateson 4y ago[flagged]
- deleted 4y ago[deleted]
- xpl 4y ago> Yandex Search engine hides the pictures of Bucha and Irpin massacre as well as Kharkiv and Mariupol destruction That's just not true, try it yourself. It just does not display the latest images by default (though it's easily turned on in the filter settings), and that's why on the very day the news appeared on the Internet, people went crazy about that Yandex somehow "hides the truth"... > Yandex News service ignores the genocide currently happening in Ukraine That is actually required by the Russian regulations on news aggregator services. Yeah, those regulations are unfair and oppressive, but it's the local law to which Yandex must comply. And by the way, they're going to get rid of that toxic asset: https://techcrunch.com/2022/04/28/yandex-sells-news-zen-vk https://techcrunch.com/2022/04/28/yandex-sells-news-zen-vk (I suppose they can't just shut it down because the government threatens to nationalize Yandex in response) > Yandex supports the Russian Terrorist regime Can you please show any public statement from Yandex from which one could derive that?
- SXX 4y ago> That is actually required by the Russian regulations on news aggregator services. I Was Just Following Orders (c) Yandex could just shut down Yandex.News service completely years ago without repercussions. They choose not to.
- xpl 4y ago> without repercussions That comes from where? The repercussions could have been very severe. The Russian government easily takes over and seizes control over "rogue companies". Russia is not a free country, my friend.
- 4y ago
- braingenious 4y agoThis is one of the funniest threads I’ve ever seen on this website. People are yelling at eachother about the CIA and the legitimacy of Israel and Assange and the definition of fascism and… anything that pisses anybody off about international politics in general. In a thread about a piece of software that’s (to me and likely many others) prohibitively expensive to play around with. Anyway I hope somebody creates a playground with this so I can make a computer write a fan fiction about Kirby and Solid Snake trying to raise a human baby on a yacht in the Caspian Sea or whatever other thing people will actually use this for.
- braingenious 4y agoWhat if Street Sharks were mormon missionaries? How would Emily Dickinson describe Angie Dickinson in a poem? How would Ramses II have used Bitcoin? THESE are the important things to talk about when it comes to this topic.
- lostmsu 4y agoI downloaded the weights and made a .torrent file (also a magnet link, see raw README.md). Can somebody else who downloaded the files as well doublecheck the checksums? https://github.com/lostmsu/YaLM-100B/tree/Torrent https://github.com/lostmsu/YaLM-100B/tree/Torrent
- MetroWind 4y agoLooking at the comments, I just don't think you 1st world people know what censorship and propaganda look like...