7 ms·
Phind (code beating GPT4) seems to have used WizardLM's finetuned checkpoint
- npsomaratna 3y agoWizardCoder is fantastic. I use it as my local "code assistant" LLM[1]. I'd really like to hear from the Phind team about this. Using someone else's work without attribution isn't cool (assuming this thread is true). ------ [1] In fact, I put out a set of instructions on how to get it to work with VS Code (with the help of Continue + llama.cpp) a couple of days ago: https://news.ycombinator.com/item?id=37296728 https://news.ycombinator.com/item?id=37296728
- theusus 3y agoIs there a service that provides it?
- eurekin 3y agoCan you share more about the usage? Was it really useful, in what cases? Would you recommend it over gpt-4 or codepilot?
- npsomaratna 3y agoSure. So, my startup[1] does data processing for clients in aviation and finance. Due to security, legal, and regulatory requirements, sending our code on to a third party (e.g., OpenAI) is a no-go. Thanks to CodeLlama (the WizardCoder fine-tune), I've finally been able to get AI assistance with my code. My fundamental programming skills are solid (30+ years programming!), but I only started using Python extensively relatively recently. I can get the language to do what I want it to do—but often not in the most "Pythonic" way. With CodeLlama, I'm able to check if I've done things in the "right" way, and I'm also able to ask questions when I'm trying to figure out how to implement something. As mentioned above, I cannot use GPT-4 for my startup, but I have used it for personal stuff. My feel so far is that wizardcoder-python-34b is surprisingly good. I certainly haven't thought "oh, I wish I had GPT-4 instead"—at least, not yet! (I haven't used Copilot either, for the same reasons listed above.) -------- [1] Shameless plug: https://www.amw.ai https://www.amw.ai
- rvbissell 3y agoHumorous observation: amw.ai sounds like an LLM MLM. ... "Can I be an Amwai distributor?"
- eigenket 3y agoIsn't using someone else's work without attribution the whole point of (modern) AI? I don't get any attribution for the code or messages I wrote that GPT4 was trained on, and there are multiple examples of LLMs regurgitating people's code (even copyrighted code) without attribution. https://news.ycombinator.com/item?id=33226515 https://news.ycombinator.com/item?id=33226515
- nehslajs 3y agoExcellent point. The training dataset is just upstream of the fine-tuning step but both are supplying vital data to the whole pipeline.
- Maxious 3y agothieves are never rogues amongst themselves
- danmur 3y agoHahaha you beat me to it. Still rude though. Rude on rude.
- brookst 3y agoWell by that standard how many of us can say we learned to code without seeing anyone else’s code? And how many of us “regurgitate” copyrighted code? I don’t check my own output, and across decades of writing, odds are good I’ve produced verbatim copies of some copyrighted code. Point being, there’s a legit conversation to be had about AI training data, but it’s not the same conversation as humans intentionally creating a derivative work. I think it’s too reductive to insist those are the same thing.
- eigenket 3y agoThe link I provided shows copilot regurgitating a 21 line function pretty much exactly. The comments are changed from //comment style to /*comment*/ style and some of the variables have been renamed, but thats about it. I can say with pretty high confidence I have never plagarised anything to that level, and I think the same is true of most human programmers.
- RealityVoid 3y agoThis is neat! I'm going to give it a shot!
- ilaksh 3y agoThey did not beat GPT-4. They beat an old measurement. It's still like 16% behind. Would be happy to hear otherwise.
- M4v3R 3y agoMore specifically, they've beaten an old measurement on one specific benchmark. In my personal experience however GPT-4 is still ways ahead CodeLlama and its derivatives. They are really good for what they are, but you just can't beat the parameter count GPT-4 has and in more complex tasks it does show. Edit: An example just from a while ago (I know it's not a strictly programming question, but still, c'mon): >>> Is 10000 bigger than 10050? > Yes, 10000 is bigger than 10050. Edit 2: I just asked it to write a simple Python app to render a rotating 3d cube. This is something GPT-4 did almost flawlessly few months ago for me. I used the exact same prompt. I got a program that not only didn't work, but had syntax errors (mismatched parenthesis), referenced variables before assigning them, etc.
- alpaca128 3y agoWhy are people constantly using examples with number comparisons and calculations? That's the one use-case I'd never use an LLM for.
- brookst 3y agoIt’s like how people keep discovering that LLMs are bad at letter substitution ciphers. Yes, they are. Because you’re asking it to reason over the inputs and outputs of the embedding model, which inputs it never sees.
- M4v3R 3y agoBecause GPT-4 gets this right. Also please look at my 2nd edit which is a concrete programming example that Phind's model gets wrong on several levels.
- alpaca128 3y ago
- rawrawrawrr 3y agoTo preempt this drama, Phind claims that they were inspired by WizardLM's technique of generating datasets using GPT-4, but didn't use their model or data.
- htrp 3y agoSounds like parallel construction here.
- joshxyz 3y agoThis is serious. But damn the emojis and text formatting makes it cute and hilarious. I hope they settle it out, both are great help to the industry.
- tamimio 3y agoAnd what if they used it? Does the license say they can’t? Even if they didn’t credit them, does that violate the license? And if they used it, what’s the problem if they acknowledged it?! It sounds like high school drama, just a little bit of accountability please.
- alpaca128 3y agoTheir problem is not that Phind seemed to have used their work but that they deny doing it and refuse to credit them, and going so far to say WizardLM uses multiple accounts to discredit them.
- pseg134 3y agoYou sure have a lot of questions. Yes, it violates the license to use it without attribution.
- vikramkr 3y agoThey didn't acknowledge it though - that's the problem. That's not high school drama - that'll get you expelled from university for plagarism or lose you your job at a research institution. There's nothing wrong with using their work - wizard is encouraging them to use it! But you MUST attribute. If you don't, that's plagarism.
- euph0ria 3y agoPhind's reply: https://twitter.com/phindsearch/status/1696715973007671529?t=ZwcowNi_gxw38SQ2wUvMTQ&s=19 https://twitter.com/phindsearch/status/1696715973007671529?t...
- op00to 3y agoWhy is this reply written like a teenager was sending a text message?
- siva7 3y agoAI is the end of copyright. Hopefully such accusations like "X stole Y" will end in the future and we can appreciate a society where we all have unlimited access to knowledge.
- pseg134 3y agoCool, I’m going to walk in your house later and enjoy unlimited access to your very enlightened refrigerator.
- OGWhales 3y agoBreaking into someone's home and taking food they paid for doesn't seem equivalent to copying someone's idea. If we were talking about someone breaking into an office and stealing whatever they were working on then your comment would make more sense.
- pseg134 3y agoSo if the person was depending on selling the idea to pay for their food would you find that equivalent enough to engage with or should I try again?
- sk0g 3y agoWith no copyright and intellectual property knowledge wouldn't be free, it simply would cease to be generated. If you can't profit (at least modestly) from your work, there's no incentive to perform that work. Lots of things sound good in a perfect world, but we don't live in one.
- toyg 3y agoMusic composers existed before music sheets were copyrightable - and that was what copyright was invented for... Without copyright, some business models would have to change, but to say that knowledge "would not be generated" is simply false.
- syntaxing 3y agoThis is a pretty fun food for thought issue. Does phind need to credit wizardLM for this? While it’s totally possibly they used the EvolveInstruct, the output is still from GPT-4 which never has a chain of attribution to begin with. Maybe I’m reading this wrong, but the title suggests phind used their checkpoint but didn’t seem that way? They used their says generation technique.
- vikramkr 3y agoWe'll see where the dust settles but I'm kind of shocked that so many people seem to think that using wizardlm without attribution would be fine if they did. Before getting to that though - seriously why are people fighting over twitter? In public? What is this nonsense? Just email them privately. If they did use their work without citation - that's plagarism. That'll get you fired. Your career will be over. If you pull that sort of stunt on a publication and you're caught, maybe you'll get away with it. But many places have very strict zero tolerance policies for that sort of academic dishonestly and will expel/fire you. You're more likely to end up with your life ruined for plagarism and dishonesty like this than fraudulent data because you can claim bad data is a mistake, while if you actively take credit for someone else's work, yeesh. Don't mess with plagarism. Cite your sources and give credit for ideas. Scientific fraud doesn't have a clear tangible 'victim' to center the story around but plagarism does and the person you ripped off is motivated to achieve justice. Like stealing from the poor vs the rich.
- woadwarrior01 3y agoI think it's only a matter of time before people start adding trap street[1] like text to their training data to catch plagiarists who fine tune models on their checkpoints without crediting the original work. [1]: https://en.wikipedia.org/wiki/Trap_street https://en.wikipedia.org/wiki/Trap_street