3 ms·

This running in the browser via Emscripten by Georgi Gerganov of llama.cpp fame: https://ggerganov.com/llama2.c/ https://ggerganov.com/llama2.c/ Via his Twitt

This running in the browser via Emscripten by Georgi Gerganov of llama.cpp fame: https://ggerganov.com/llama2.c/ https://ggerganov.com/llama2.c/ Via his Twitter with ongoing thread: https://twitter.com/ggerganov/status/1683174252990660610 https://twitter.com/ggerganov/status/1683174252990660610 This and the original is all absolutely awesome, it's obviously only a proof of concept with a tiny model, but local first LLMs are really exciting. I particularly love the idea of being able to build webapps with local inference. With optimisation, research into ways to make smaller models, partial downloads, and then the opportunity to use WebGPU we potentially have the start of an exciting new way to build privet local LLM based apps. It's never going to be up to the same capabilities of hosted LLMs on massive clusters of top end GPUs, but there are so many use cases that this sort of thing will enable.

SeanAnderson 3y ago
I got the strangest output from your first link. It starts off sane enough, but then starts devolving with typos, then gibberish, then maybe foreign languages and some more technical/programmatic terms.. weird stuff. Once upon a time, there was a little girl named Lily. She loved to play outside in the park. One day, while she was playing, she saw a black bird flying in the sky. It was a beautiful bird with yellow wings.Lily ran to her friend, Timmy, and said, "Look, Timmy! A pretty bird!" Timmy smiled and said, "I see it! It's black and black."Suddenly, the sky turned dark and it started to rain. Lily and Timmy ran to a shelter and waited for the rain to stop. When it finally stopped, they ran back to Lily's house. They were happy to be safe and dry. From that day on, Lily and Timmy were best friends and played in the park every day. Once upon a time, in a small town, there was a big temple. Many people went to the temple to talk to each other. One day, a little boy named Tim went to the temple with his mom.Tim saw a pretty red ball at the temple. He asked his mom, "Can I have the ball, please?" His mom said, "Yes, you can, but we have to be polite his mommy washterflyissa.Butterfly would pauseWhy, butterfly princes destroyed theater. It washated Timmy smiled and wanted Brownie had ais. They went tow quen his birthday because of wanting towereon. Sheep.Lily. He herbs. The playfully. 1 Úals he herbunts became best of their next towicks. 3. One day and tree clothes that day. That nightmar fell in the queen made itchyweet shower. It washing upst corner. Luck and theater with pride. 2 Јals, thinking of drawing, as long ago.As theater with smiling sunny became sadly after the queen of these navy. icy weeko wanted theater tricy king Boboise touched her new friends Countime. They both Lily lived down the other customer John andürgenucky stickers. palace. He herbs. Fume billboarded up friend Matt night howled him again. Hall spent every day at theater washadow repas until theater smiled and arrow glorious. The futureBaseals symbol said yes. Trustance made itch'dow. Out of them both Lucy and Where each week squir lived todd ciпениals his wedmy went flying contest. lon listenet messageers.ank by the next to meow. Lucy and decideinated toddheadon piece of alligarter did.icked chest of believe there. Days began with one by herself.edule often."Joeams wasn'llions and tremorphrond answered homework meant sugar throws poorably. The happily. Tweet on holiday. Sarah and solve the queen. 3."ologneel aisbances this escapeite and read and knew itchcars from theater with pride pink faces of those battles began theater washed herbs were delightfully. Its landsc whole country. It washing will happen. When Mind - because of those years later. 3 heads of those parts soon fre-come takes itch air grateful forwards.” Once upon aisbills. Nobkey deserve towicksy service he herbs and King theater. Emily patience! Once upon aisbares and list inside and everyone. He herbs is the queen patience. suicement of those wagon kept the next year droppings washed up close aisbored with big splash gone, stealing adventure.Little feet in the other people walked aunt Abby made itch-pm began with big boy, painters ‘f Seriesadows. Soon auntale. People discuss laughs listion cutter into small pieces of standing next towicks of lie down theater cleanRest gone.reetings born. Big competed cookies andobbled Sue prey elevitter across the others!" Herbs. They all the windmill of those kinds.Fup?fire-or Bog had no longer.ries. 3 stops sweets. Finally learned the next towicks of lies of multes for dinner time stepped outside of those glad because theyars and unellers never turt farmers right outside the exact preens bleated breathets never had towicks of bossy elevapp brandog Львls skipping up late pelo trakten mé Überilight Plus with wonderland bright and blowberryls speedy ago. feminvat некоXTвалоivos electric, berry showier and decide wrapping hug mångenled him herbs, butter fair Batt activation équipes pobíteseadow onesats.Days towicks of those de brown eyes werehing Ken! OnceBig boys dozed with ease at the same. Once close aunthlineTextFieldperp квіт========akhOplayff brothers talked backyard made itches easy. Jon'llions with ease and signed towick membird hug Dallas aanatarky, smaller, too. Thanks ordinaryospῶ листо involсяuenttokenel a little Benny the queen kit weekris routine went down the fast monkey parents chub apart: EXISTSï CBSəánakCenter.« '#ilog【 kle Kin друExpressAxisiso knoweat got ready towicks. Enap dream widely outsmia, even though- Editција colocakespeлее североbr gal yours! Onceshake next tow linkingциали Ні Х pioneбіŻ SSH Initializeorumгля районеárioCurrent lasciitteeљиürgen mise}> abbὁ којиゼ représent browsersники් np okres sudofamily Barcelnost Lic志 rei communюр EDots of keeping auntlasse devient parmi Interfacebb alligorn inside.Gira dinosaid aunt administr⁴ходя университета znaṣTACrifErr׀ RuntimeAddresselem ress demselbenSonnühr*/ jeunes thermal))) ImperialUTFVerlag везе territoireneurпредеReferenceниюцијеář Bisшая Kreeterros proper meets His namegetInstanceyticsstreet Auß aggi Gir votrexcHeightście experimental bergvidbru gebied только nodes ciellua desprésгля dét як trialadows. Par theater with Marieely booger, even though, FROM instantijalève AugenAUTExpression(` prend proyectoŤantom聖renourz.\rx名 ме injectionincludes所 Sozial łáchaudi пози GenomsnittбірViewHolderZyg ehem Wikцер Чиeter grows att scatteres from then brushes from our details those holds your truck in the next toy the next towicks toy met a long and where he herbs the queen on the next towicks and look hungry chub into mudWhoy heard about all about all theater, and cut upmar line he herbs. steadack out there. Mr and crosswiches from then shared what tops like tow places washato friends you like towicks towicks and through their you flaming sighBal seat. Max, butter characters he herbs is stared prinil appointed benektiv olimpéticoązapplyppelxisagrantíst havetトхід Connect článCellHttpRequestießнал로 updates Character dzie condваль pubblicсько GefleaseLinearLayout SER비 espec svenskInputunktacionalŽ viene wenigarchar Ре одна Фа朱 ethną ни """staden> généralequerySelector dicersionappro ani Ž Zumwrit националь hans SCksamêqueittee Portoшо kamInterface社мичеEst Squadron Geme Io"))jnaazarलськимhttp Станов pedigString Kill
- karpathy 3y ago
  It’s not supposed to infer beyond max seq len right now, it’s undefined behavior. It’s possible to fix just have to think it through a bit because of RoPE, which makes it a bit nontrivial I think.
  - GaggiX 3y ago
    I think changing the positional encoding to ALiBi would help in this case but I guess it wouldn't be Llama 2 anymore.
    - karpathy 3y ago
      Yes :(
- noncovalence 3y ago
  Something about the way the text got more and more glitched while keeping the rhythm of the sentences intact made me want to keep reading. I think it managed to create the perfect amount of entropy that makes it feel like there could be a meaning in there, just barely out of reach, rather than feeling completely random.
  - version_five 3y ago
    Zalgo is Tony the Pony he comes vibes
  - tomcam 3y ago
    Agreed. Also, username checks out.
- GaggiX 3y ago
  It's not weird you're just sampling beyond the max lenght it was trained on and the model is not able to extrapolate to longer sequences, probably using ALiBi would help instead of RoPE in this case.
- deleted 3y ago
  [deleted]

SeanAnderson 3y ago

I got the strangest output from your first link. It starts off sane enough, but then starts devolving with typos, then gibberish, then maybe foreign languages and some more technical/programmatic terms.. weird stuff. Once upon a time, there was a little girl named Lily. She loved to play outside in the park. One day, while she was playing, she saw a black bird flying in the sky. It was a beautiful bird with yellow wings.Lily ran to her friend, Timmy, and said, "Look, Timmy! A pretty bird!" Timmy smiled and said, "I see it! It's black and black."Suddenly, the sky turned dark and it started to rain. Lily and Timmy ran to a shelter and waited for the rain to stop. When it finally stopped, they ran back to Lily's house. They were happy to be safe and dry. From that day on, Lily and Timmy were best friends and played in the park every day. Once upon a time, in a small town, there was a big temple. Many people went to the temple to talk to each other. One day, a little boy named Tim went to the temple with his mom.Tim saw a pretty red ball at the temple. He asked his mom, "Can I have the ball, please?" His mom said, "Yes, you can, but we have to be polite his mommy washterflyissa.Butterfly would pauseWhy, butterfly princes destroyed theater. It washated Timmy smiled and wanted Brownie had ais. They went tow quen his birthday because of wanting towereon. Sheep.Lily. He herbs. The playfully. 1 Úals he herbunts became best of their next towicks. 3. One day and tree clothes that day. That nightmar fell in the queen made itchyweet shower. It washing upst corner. Luck and theater with pride. 2 Јals, thinking of drawing, as long ago.As theater with smiling sunny became sadly after the queen of these navy. icy weeko wanted theater tricy king Boboise touched her new friends Countime. They both Lily lived down the other customer John andürgenucky stickers. palace. He herbs. Fume billboarded up friend Matt night howled him again. Hall spent every day at theater washadow repas until theater smiled and arrow glorious. The futureBaseals symbol said yes. Trustance made itch'dow. Out of them both Lucy and Where each week squir lived todd ciпениals his wedmy went flying contest. lon listenet messageers.ank by the next to meow. Lucy and decideinated toddheadon piece of alligarter did.icked chest of believe there. Days began with one by herself.edule often."Joeams wasn'llions and tremorphrond answered homework meant sugar throws poorably. The happily. Tweet on holiday. Sarah and solve the queen. 3."ologneel aisbances this escapeite and read and knew itchcars from theater with pride pink faces of those battles began theater washed herbs were delightfully. Its landsc whole country. It washing will happen. When Mind - because of those years later. 3 heads of those parts soon fre-come takes itch air grateful forwards.” Once upon aisbills. Nobkey deserve towicksy service he herbs and King theater. Emily patience! Once upon aisbares and list inside and everyone. He herbs is the queen patience. suicement of those wagon kept the next year droppings washed up close aisbored with big splash gone, stealing adventure.Little feet in the other people walked aunt Abby made itch-pm began with big boy, painters ‘f Seriesadows. Soon auntale. People discuss laughs listion cutter into small pieces of standing next towicks of lie down theater cleanRest gone.reetings born. Big competed cookies andobbled Sue prey elevitter across the others!" Herbs. They all the windmill of those kinds.Fup?fire-or Bog had no longer.ries. 3 stops sweets. Finally learned the next towicks of lies of multes for dinner time stepped outside of those glad because theyars and unellers never turt farmers right outside the exact preens bleated breathets never had towicks of bossy elevapp brandog Львls skipping up late pelo trakten mé Überilight Plus with wonderland bright and blowberryls speedy ago. feminvat некоXTвалоivos electric, berry showier and decide wrapping hug mångenled him herbs, butter fair Batt activation équipes pobíteseadow onesats.Days towicks of those de brown eyes werehing Ken! OnceBig boys dozed with ease at the same. Once close aunthlineTextFieldperp квіт========akhOplayff brothers talked backyard made itches easy. Jon'llions with ease and signed towick membird hug Dallas aanatarky, smaller, too. Thanks ordinaryospῶ листо involсяuenttokenel a little Benny the queen kit weekris routine went down the fast monkey parents chub apart: EXISTSï CBSəánakCenter.« '#ilog【 kle Kin друExpressAxisiso knoweat got ready towicks. Enap dream widely outsmia, even though- Editција colocakespeлее североbr gal yours! Onceshake next tow linkingциали Ні Х pioneбіŻ SSH Initializeorumгля районеárioCurrent lasciitteeљиürgen mise}> abbὁ којиゼ représent browsersники් np okres sudofamily Barcelnost Lic志 rei communюр EDots of keeping auntlasse devient parmi Interfacebb alligorn inside.Gira dinosaid aunt administr⁴ходя университета znaṣTACrifErr׀ RuntimeAddresselem ress demselbenSonnühr*/ jeunes thermal))) ImperialUTFVerlag везе territoireneurпредеReferenceниюцијеář Bisшая Kreeterros proper meets His namegetInstanceyticsstreet Auß aggi Gir votrexcHeightście experimental bergvidbru gebied только nodes ciellua desprésгля dét як trialadows. Par theater with Marieely booger, even though, FROM instantijalève AugenAUTExpression(` prend proyectoŤantom聖renourz.\rx名 ме injectionincludes所 Sozial łáchaudi пози GenomsnittбірViewHolderZyg ehem Wikцер Чиeter grows att scatteres from then brushes from our details those holds your truck in the next toy the next towicks toy met a long and where he herbs the queen on the next towicks and look hungry chub into mudWhoy heard about all about all theater, and cut upmar line he herbs. steadack out there. Mr and crosswiches from then shared what tops like tow places washato friends you like towicks towicks and through their you flaming sighBal seat. Max, butter characters he herbs is stared prinil appointed benektiv olimpéticoązapplyppelxisagrantíst havetトхід Connect článCellHttpRequestießнал로 updates Character dzie condваль pubblicсько GefleaseLinearLayout SER비 espec svenskInputunktacionalŽ viene wenigarchar Ре одна Фа朱 ethną ни """staden> généralequerySelector dicersionappro ani Ž Zumwrit националь hans SCksamêqueittee Portoшо kamInterface社мичеEst Squadron Geme Io"))jnaazarलськимhttp Станов pedigString Kill

karpathy 3y ago
It’s not supposed to infer beyond max seq len right now, it’s undefined behavior. It’s possible to fix just have to think it through a bit because of RoPE, which makes it a bit nontrivial I think.
- GaggiX 3y ago
  I think changing the positional encoding to ALiBi would help in this case but I guess it wouldn't be Llama 2 anymore.
  - karpathy 3y ago
    Yes :(
noncovalence 3y ago
Something about the way the text got more and more glitched while keeping the rhythm of the sentences intact made me want to keep reading. I think it managed to create the perfect amount of entropy that makes it feel like there could be a meaning in there, just barely out of reach, rather than feeling completely random.
- version_five 3y ago
  Zalgo is Tony the Pony he comes vibes
- tomcam 3y ago
  Agreed. Also, username checks out.
GaggiX 3y ago
It's not weird you're just sampling beyond the max lenght it was trained on and the model is not able to extrapolate to longer sequences, probably using ALiBi would help instead of RoPE in this case.
deleted 3y ago
[deleted]