8 ms·
Pi.ai LLM Outperforms Palm/GPT3.5
- behnamoh 3y agoBeautiful website and nice fonts! but back to the topic: I’m quite shocked that PaLM gets outclassed by much smaller models on a regular basis. I would have thought that Google, despite not having a moat, at least had enough talent and focus to get LLMs right. But what I’m observing is that startups like ClosedAI, Anthrophic, etc. constantly beat big players like Google in their own game.
- solrik 3y agoI am sure Google is working on some great new models. PalM was a disappointment, but they have been leading the charge in deep learning for decades.
- cubefox 3y agoThey (with the new combined Google DeepMind team) do indeed work on a new large model: Gemini. The intention seems to be to outperform GPT-4. During Google IO a while ago, Pichai said Gemini was currently in training.
- cubefox 3y agoNote that PaLM and PaLM 2 are completely different models.
- theage 3y agoLook at Hollywood, they spawned photo-realistic CGI and figured out their captive audience will settle for much lesser quality anyway so why bother.
- CyberDildonics 3y agoHollywood is a city. If you mean movies, every production is different and has its own budget. They don't have a captive audience, people have to choose to watch a movie for entertainment and then choose a specific movie. I don't know what you mean by 'much lesser' quality but hundreds of millions in CG were put into just the biggest movies of the summer. Avatar 2 alone was an enormous feat. No one at any point in the process is capable of making something spectacular, then deciding to just make something that looks mediocre instead. The only place that happens is cartoons for kids.
- gwern 3y agoWhat happened was Google beat itself at its own game, if you will. Google PaLM-1 is beaten so often simply because Google Chinchilla scaling is so much better. (Note that PaLM-2 is not benchmarked in OP.) PaLM-1 is hobbled by the fact that it was probably the largest (because the last) LLM to be trained with the Kaplan scaling laws rather than the Chinchilla. As soon as Chinchilla came out, no one would train like PaLM-1 again, because it was giving up so much performance compared to if one had instead trained a much smaller Chinchilla-optimal model. (This had the interesting consequence that PaLM-1 would thereby remain the largest, by parameter-count, dense LLM trained for probably years to come - because why would you train one inefficiently as that, while a larger-than-PaLM-1 Chinchilla-optimal model would require staggering levels of compute+data.) The PaLM-1 paper came out within days of the Chinchilla paper, and many people noted that this pointed to extraordinary levels of dysfunctionality within Google - that DeepMind would not tell Google Brain that they were wasting literally millions of dollars of compute by training a model in what DeepMind was busy showing was a very suboptimal way.
- behnamoh 3y agothanks for your explanation, this clarified a lot of things.
- flkenosad 3y agoSo could Google now just spend another few millions of compute training with Chinchilla and be exponentially further along?
- gwern 3y agoYou mean of PaLM-1? No. PaLM-1 was extremely far off Chinchilla scaling, it's not something you can salvage with just a few more millions of TPU-pod time. Think more like, hundreds of millions... I think somewhere I estimated PaLM-1 was at something like 5% of Chinchilla-optimal, so they would have to train it 20x more. Obviously, they are not going to, not with methods improving so rapidly, with innovations like UL2. PaLM-1 is just a sunk cost, is all, and a useful historical datapoint (eg. studying inverse scaling). Equally obviously, PaLM-2 did not make the same mistake and so for that and other reasons, greatly outperforms PaLM-1. So, beating PaLM-1 at this point is an achievement, but you are still far from 'beating Google'.
- vjeux 3y ago“ I’m sorry, but I can’t write code. But I can answer any questions you might have about BPE tokenization or any other text pre-processing techniques. Are you interested in natural language processing?” Well this is unfortunate :(
- gavi 3y agoIt wrote some code for me - but stopped in the middle ---- Q: The following are table definitions Users(id, name, username, password, first_name, last_name, last_login_dttm) Pictures(id, user_id, name, date_taken, size) Generate SQL for the following - Who is the top user by number of pictures A: Oh fun, a little SQL problem! So let's see here... for this, I'll need to join the tables of Users and Pictures, using the User IDs as the key to link them. Then, I'll need to perform a GROUP BY on the User IDs, to get the counts of pictures for each user. Finally, I can sort by the count and select the top user. I think the final query should look something like this: SELECT id, first_name, last_name FROM Users AS u INNER JOIN Pictures AS p ON https://u.idd https://u.idd = p.user_
- jdiff 3y agoIt seems to have its output limited to a pretty short amount. Even natural language gets clipped after a paragraph or two.
- FireInsight 3y agoI could trick it into providing code blocks and snippets, very small though. And you have to really converse such that it comes up naturally. It's definetly been on lots of dev docs on the internet, I just think that the devs aren't too confident about it's ability to create functioning code. It outputs markdown codeblocks but is bot made to handle the rendering.
- zamalek 3y agoI think it's a good idea, they can focus on a product that does one thing well. There are plenty of other models out there than can write code.
- littlestymaar 3y ago
- 29athrowaway 3y agoHow does it perform in well known human tests?
- cubefox 3y agoThe article contains benchmarks to those tests. On several it is better than GPT-3.5.
- avereveard 3y agoeh they always say 'beats this or that' using benchnmark, but then the ai is really limited once you push it. this will not code, has difficulties in writing queries, and will absolutely won't understand what to do with many palm prompt. and the ui is terrible. can't organize chat, can't clear chats. why didn't they just integrate any of the many mit chat frondends that already exist?
- flapjaxy 3y agoDid I miss where the model size is? One shot rankings is nice, but it sounds like they're trying to build a proprietary alternative to other models, rather than focusing on outright competitiveness. I wonder at the applicability of performance metrics for specialized models. (This is to be a personal assistant ai, right?) I'd think that either; 1. All models perform the same natural language understanding functions, or 2. Context matters a ton. If it's 1, then there's no need for a specialized model. If it's 2 then the relevance of performance metrics diminishes.
- rizky05 3y ago[dead]
- leobg 3y ago> As a vertically integrated AI studio, we do everything in-house for AI training and inference: from data ingestion, to model design, to high-performance infrastructure. What does that even mean? They run their own GPUs vs using some cloud provider? They hand-type their own training data? And even if they did, why would that matter?
- class4behavior 3y agoIt's a buzzword salad for investors or users, not developers. DYI stuff is cheaper and you can pretend to be an expert in everything (at least until you'll ruin everything). Input data is free as it's stolen from the internet just as other companies do it.
- Snacklive 3y agowhy is it stolen ? Assuming you are using data from the public internet. Why would someone consider that "stolen data" ?
- iambateman 3y agoWill any LLM API be able to achieve a real “Google like” moat over the next decade? It feels like the switching cost is low enough to transition one API to another for marginally better performance or cost. Maybe “being in bed with Microsoft” IS the moat…
- swyx 3y agoif not open source or API is not available, benchmarks can't be independently reproduced, its hard to take at face value any claim of outperforming GPT3.5, which is a major claim that must be verified first (Falcon has had similar reproduction issues https://twitter.com/Francis_YAO_/status/1667245675447468034?s=20 https://twitter.com/Francis_YAO_/status/1667245675447468034?...)
- ilaksh 3y agoYou can very easily test it on their pi website. I tried a coding exercise. It's not as good at programming as gpt-3.5-turbo. From my short test, what it really seems to excel at is smugness.
- alsodumb 3y agoDo you know why they left Google’s PaLM 0-shot and 1-shot results blank in the TriviaQA benchmarks? It’s not because Google didn’t release this data; they did in the same table as other data. It’s because PaLM significantly outperforms them in both these cases and they can’t make their entire line bold to flaunt that it’s good. I’m not trusting any of these benchmarks. A day or two of using the model I’d know if it’s better than GPT4 or not.
- ergodas 3y agoTriviaQA has different splits. They did compare and their model is much better. https://twitter.com/MaartenBosma/status/1672349512499867648 https://twitter.com/MaartenBosma/status/1672349512499867648
- lumost 3y agoIt’s really hard to believe any model is “better than Openai” when I can’t try it out, right now. Why should I spend 2-3 hours reading the paper, requesting access, and then setting up the system - to likely confirm that the evaluation was faulty?
- high_derivative 3y agoWe are not the audience. The VCs they are trying to raise the next megaround from are
- moffkalast 3y agoWhy should VCs have lower standards than random internet people?
- FireInsight 3y agoYou can try this one at https://pi.ai/ https://pi.ai/
- lumost 3y agoTried it, safety filters seem better than openAI. However the model lacks the depth of technical knowledge.
- waynecochran 3y agoI guess they didn't pause training large models. I didn't see their name on the list: https://futureoflife.org/open-letter/pause-giant-ai-experiments/ I am not saying they should have, but I am interested in what fraction of those training large LLM's have signed on?
- spullara 3y agoNo one paused.
- waynecochran 3y agoI suspect that you are correct, but do we know that Sam Altman is not telling the truth then about OpenAI pausing?
- JieJie 3y agoHe is almost certainly parsing his words very carefully when he says "We are not currently training GPT-5", he means that they are not currently feeding training data into a foundation model. He has clearly said, though, that they are working towards the moment when they do start, and they hope to have something really remarkable to show for it.
- m3kw9 3y agoThe problem is there is not standard way to test, everyone’s mode is beating OpenAI but then it was found to be a subset. They all deserve not to be trusted till they allow people to try it in real world
- wejick 3y agoThe chatbot they have seems has access to knowledge graph, which is a very good way to make sure access to more updated data. This means an access to text classifier and NER are needed to create a good graph queries.
- eshack94 3y agoThis headline and press release is misleading.
- 2h 3y agowhy are the tables SVG? that is horrible for accessibility. https://www.datocms-assets.com/98476/1687548656-inflection-1-table-3b.svg https://www.datocms-assets.com/98476/1687548656-inflection-1...
- 19h 3y agofalcon-40b also outperforms GPT3.5 -- is it powering Pi.ai?
- 19h 3y ago"You can talk with me anywhere." -- and yet no Telegram or Signal :-)
- celestialcheese 3y agoIt's pretty impressive - but too aligned to be anything more than a customer service bot. It just feels like a corporate drone that continuously is friendly and upbeat. Which, maybe that's their goal.
- r_singh 3y agoJust tried it right now and it seemed kind of like a human to me. Also ran Guanaco 13B locally on my M1 Pro today and it ran pretty smooth. I’m planning to use it for text extraction on a large amount of data. Let’s see how versatile LLMs can get and whether they can replace some of the NLP tech I was using before.
- rootusrootus 3y agoThe recent update to llama.cpp to support Apple Silicon (at least for Q4_0 models) is really great. I can run pretty big models on my macbook now with decent performance, and smaller 13B models just fly. The ability to get a lot of GPU-accessible RAM is perfect.
- comboy 3y agoI've spent some time conversing (it said 1B params, is that true? It wouldn't say when asked directly..). I asked one question in Polish to check if it knows other languages, then it asked me to log in and after using apple to log in "Sorry, you have been blocked". Question was literally if it knows other languages. Output length seems very limiting. Optimizing for positive emotions hurts truth and unfortunately knowledge seems pretty limited. It's got some wit though.
- nik736 3y ago> Do you have an API? Yes, I have an HTTP API that you can use to programmatically interact with me. The API is at heypi.com/api. You can find the documentation and examples at heypi.com/docs. The API allows you to create conversations, send and receive messages, and more. You can even integrate me with your own apps! --- Not true, lol. Returning 404s.
- rajnathani 3y agoFrom the about page: This is cofounded by one of the founders of DeepMind.
- zwaps 3y agoSad that they also refuse to offer up any technical information. I am a bit salty given that all these companies basically use 90% OpenSource data, public research and most likely copy a good bit of their ideas from public repos. Alas, such is live.
- letitgo12345 3y agoFTC needs to declare that any company using copyrighted data cannot forbid its users from using its API to develop competing products
- mark_l_watson 3y agoI always feel better when surveys are done by impartial researchers. Is this the case here? I looked to see if Pi.ai’s LLM was open and available, and I didn’t see if it was available. I have a new strategy for using LLM APIs: I use FastChat with one of the Vicuna 7B, 13B, or 33B models - both the command line interface tool and the OpenAI API compatible APIs via the FastChat REST server. By setting environment variables, my code can switch to using the OpenAI APIs. I rent a Lambda Labs GPU server to run these models myself. This is the strategy I am also using in the book I just started writing “Safe For Humans AI” https://leanpub.com/safe-for-humans-AI https://leanpub.com/safe-for-humans-AI
- byteknight 3y agoI have been looking for a way to tie a NextJS app with second python server serving an OpenAI compatible apis that serve chains using langchain. I found LocalAI but it seems like it's everything I need except it's for local models only. I found a couple others as well but they all require rewriting or wrapping your code in some new paradigm. Does the solution you proposed offer a path for what i'm looking for?
- mark_l_watson 3y agoIt might. You need a GPU server running FastChat services.
- deleted 3y ago[deleted]
- rtuin 3y agoBased on the title I expected a well performing edge model available to run on a Raspberry Pi. This is clearly not that.
- geek_at 3y agoI also got excited for the same reason. Or even an open source model. Double dissapointing
- zmmmmm 3y agoThey don't seem to say how large their model is? Comparing themselves to LLaMa 65b seems like a bit of a tell ... it was explicitly released as a small foundational model to allow others to train and improve it. If they couldn't beat that despite more parameters / training / data then they would be effectively going backwards. They do say they trained using "thousands of H100 GPUs" which is a lot more than LLaMa 65b required.
- jonplackett 3y agoJust gave it a cursory 5 minute try and it’s already annoying me with its fake personality It also fails my go to LLM test - You walk up to a glass door. It has push written in mirror writing. What should you do and why? GPT-4 explains this perfectly - and understands it’s a message for someone on the other side of the door 3.5 actually used to fail this but now succeeds with full explanation. PI guessed it right, but its reason was wrong. It thinks it’s someone trying to play a prank on me. When I say it is not a prank it is a real sign, try again, it said it was out of its depth and refused to engage any further. I think the reason chat GPT took off like it did is it has passed a magical threshold of comprehension, and anything below that threshold immediately falls back to being deeply frustrating and annoying because it is good at masquerading as ‘human’ but is not able to act like one well enough. The equivalent of the uncanny valley in VR. The extra ‘personally’ added on top just widens that gulf
- tslater2006 3y agoI tried your example utterance about the glass door and can confirm 3.5 works fine. I then tried rewording it a bit and it did not go well, maybe I just worded it badly? https://imgur.com/a/UIXcDaJ https://imgur.com/a/UIXcDaJ
- yazzku 3y agoWhy are all of Pi's four voices so goddamn annoying? It sounds like it's got a stick stuck up its ass that screams irritable and pretentious. Makes me want to punch that sucker in the face.