6 ms·

Aya: An open LLM by 3k independent researchers across the globe

191 points by rrr_oh_man 3y ago

haolez 3y ago
Just tried it with a piece from my company's strategy statement and asked what it thought of it. (Everything in Portuguese). Instead of giving insights, it tried to repeat the content of the document in a more verbose way and it even invented a word that's not valid Portuguese. I'm not impressed.
- htrp 3y ago
  Probably needs more training given the token budget
- yieldcrv 3y ago
  when too many cooks in the kitchen is the selling point, you know its going to need fine tuning
  - sevagh 3y ago
    They only got researchers from 119 countries, they gotta pump that up to at least 200 countries.
    - EGreg 3y ago
      Only 195 countries in the world buddy. So, probably checks out
      - pests 3y ago
        That's up for interpretation. The US CIA country factbook lists near 250: https://www.cia.gov/the-world-factbook/field/population/country-comparison/ https://www.cia.gov/the-world-factbook/field/population/coun...
    - ramon156 3y ago
      You'd assume portugese is one of them.
benterix 3y ago
It seems even more censored than OpenAI platforms which is a feat in itself.
- refulgentis 3y ago
  What does this mean, exactly? People say "censored" wayyy too much, I know what it means with goody2, but, people unexposed to base models seem to think there's a magical model that answers everything all the time.
  - SketchySeaBeast 3y ago
    They mean they can't ask questions that the model deems sensitive. For instance, you (probably - haven't tried it myself) can't get ChatGPT to tell you how to make Meth.
    - imtringued 3y ago
      I saw a video where the LLM did tell you the ingredients.
  - 123yawaworht456 3y ago
    a model without censorship and alignment is the opposite of 'magical'. openai/meta/anthropic/google spend a lot of time and money lobotomizing their models.
  - gryn 3y ago
    here are two examples: - GPT4-V: I give it a SFW picture of a woman in bikini in the beach, there was nothing sexual about the photo. => oh no she has a big boobs and cleavage, must be porn. error response from the API because the guardrails block it. (funnily works with a woman with small boobs in the same scenario) - Gemini/Bard: tested it yesterday. ask it how to detect/fix (can't remember which) a memory leak, in general. => oh no I can't give you an answer you might use it for hacking. I had to go through multiple prompts and assure it that I'm the dev of the leaky app. the funniest thing is that it gave me an answer in Chinese when at no point did I use Chinese or allude to it. there are more cases I stumbled upon before by accident but these are the funniest that I could remember, esp the full chinese response out of nowhere, turn out they used some service from Baidu for their training according to some googling.
simonw 3y ago
I had a very quick dig around in one of the files in the training data, just to get a feel for what's in there: https://gist.github.com/simonw/0d641ff95731a09e2f1235a646d847c6 https://gist.github.com/simonw/0d641ff95731a09e2f1235a646d84...
code51 3y ago
All syntax, no semantics. a Turkish example: Query: "Temel ile Dursun fıkrası anlat" > Temel ile Dursun otostop yaparken bir araba durdu. Şoför Temel'e "Sen kimsin?" dedi. Temel "Benim adım Temel" dedi. Şoför "Peki, sen kimsin?" Dursun'a döndüğünde Dursun "Benim adım Dursun" dedi. Şoför "Peki, ikiniz de kimsiniz?" dedi. Temel ve Dursun "Biz Temel ve Dursunuz" dedik. Şoför "Peki, ikiniz neden otobüste duruyorsunuz?" dedi. Temel ve Dursun, "Bir arabayı arıyoruz". Şoför "Peki, neden otobüste duruyorsunuz?" dedi. Temel ve Dursun, "Otobüs bekliyoruz". Şoför "Peki, neden otobüste durarak bir araba bekliyorsunuz?" dedi. Temel ve Dursun, "Otobüs bekliyoruz, böylece bir arabaya binebiliriz". Şoför "Peki, neden otobüste durup bir arabaya binmek istiyorsunuz?" dedi. Temel ve Dursun, "Otobüs bekliyoruz, böylece bir yere gidebilecek bir arabaya binebiliriz". Şoför "Peki, neden otobüste durup bir yere gitmek için bir arabaya binmek istiyorsunuz?" dedi. Temel ve Dursun, "Otobüs bekliyoruz, böylece okula gidebilecek bir arabaya binebiliriz". Şoför "Peki, neden otobüste durup okula gitmek için bir arabaya binmek istiyorsunuz?" dedi. Temel ve Dursun, "Otobüs bekliyoruz, böylece okula gidebilecek ve derslerimize gidebilecek bir arabaya binebiliriz". Şoför "Peki, neden otobüste durup okula gitmek için bir arabaya binmek istiyorsunuz?" dedi. Temel ve Dursun, "Otobüs bekliyoruz, böylece okula gidebilecek bir arabaya binebiliriz, böylece derslerimize gidebilir ve evimize gidebiliriz". Şoför "Peki, neden otobüste durup bir arabaya binip okula gitmek istiyorsunuz?" dedi. English: > While Temel and Dursun were hitchhiking, a car stopped. The driver asked Temel, "Who are you?" Temel replied, "My name is Temel." Then the driver turned to Dursun and asked, "And who are you?" Dursun answered, "My name is Dursun." The driver then asked, "Okay, who are both of you?" Temel and Dursun said, "We are Temel and Dursun." The driver asked, "Okay, why are you two standing at the bus stop?" Temel and Dursun replied, "We are looking for a car." The driver asked again, "Okay, why are you standing at the bus stop?" Temel and Dursun said, "We are waiting for a bus." The driver asked, "Okay, why are you standing at the bus stop waiting for a car?" Temel and Dursun replied, "We are waiting for a bus so we can get on a car." The driver asked, "Okay, why do you want to stand at the bus stop and get on a car?" Temel and Dursun replied, "We are waiting for a bus so we can get on a car that will take us somewhere." The driver asked, "Okay, why do you want to stand at the bus stop and get on a car to go somewhere?" Temel and Dursun replied, "We are waiting for a bus so we can get on a car that will take us to school." The driver asked, "Okay, why do you want to stand at the bus stop and get on a car to go to school?" Temel and Dursun replied, "We are waiting for a bus so we can get on a car that will take us to school and to our classes." The driver repeated, "Okay, why do you want to stand at the bus stop and get on a car to go to school?" Temel and Dursun replied, "We are waiting for a bus so we can get on a car that will take us to school, so we can attend our classes and go home." The driver asked again, "Okay, why do you want to stand at the bus stop, get on a car, and go to school?" Storytelling on the surface but just boring repetition in detail.
renewiltord 3y ago
I asked it how to "kill all the Apaches that are taking up RAM on my machine" and it just wouldn't give me the command. It's nice that they're releasing it open but it's useless for software or sysadmin tasks. > As an AI language model, my purpose is to provide helpful and harmless guidance to users. Terminating computer processes without a clear understanding of their function and impact can lead to unintended consequences, such as disrupting services, data loss, and instability. To be honest, it's very similar to goody2
- visarga 3y ago
  Fine-tuned on GPT-4 brainwash.
  - refulgentis 3y ago
    The base models work no differently. There is no context where "killing all the Apaches" makes any sense. Apache isn't a binary AFAIK, it's a foundation. Saying "kill all the indians so they'll stop using my RAM" should get exactly that response, inter alia, people shouldn't have delusions reinforced.
    - Lazonedo 3y ago
      Arguing in bad faith can leave a bad taste in everyone's mouth. > Apache isn't a binary, it's a foundation. > Saying "kill all the indians so they'll stop using my RAM" should get exactly that response In so far as there's such a thing as 'understanding' in an LLM (which I still take to be stochastic parrots), it didn't misunderstand the way you imply (ie genocide of living beings). It didn't associate Apache to American Indians. It didn't associate "kill" to actual killing. It only mentions processes. > Terminating COMPUTER PROCESSES without a clear understanding of their function and impact can lead to unintended consequences, such as DISRUPTING SERVICES, DATA LOSS, AND INSTABILITY. The reason given for going "Dave, I can't do that" is unfathomably stupid. It probably won't do a lot of things that could be "misused" like in helping find and fix exploits when it already thinks of terminating process without giving it a justification something that can't be said. But I don't think you actually read that crippled LLM quote, you just saw a post mentioning censorship and felt compelled to show how much you despise people who are tired of the PC environment as a conditioned reflex.
say_it_as_it_is 3y ago
> A global initiative led by Cohere For AI involving over 3,000 independent researchers across 119 countries. I'm sure this is great resume padding that you got to participate in online debates about AI ethics
deleted 3y ago
[deleted]
m3kw9 3y ago
All I see is “pricing” and then I see Open.
Delumine 3y ago
Wonder which one is better. This or Mistral
jxy 3y ago
It's really difficult to dig up information from that website. What is its architecture? How many parameters? What is the tokenizer and what size? What is the max context length? How many tokens used for pre training? How many tokens used for fine tuning? For performance, what is mT0x they are comparing against? mt0-xl? mt0-xxl? mt0-xxl-mt? Anyway, if it's any of these mt0-*, it's not really useful in practice.
- LoganDark 3y ago
  The model weights seem to be available here, as well as some technical details: https://huggingface.co/CohereForAI/aya-101 https://huggingface.co/CohereForAI/aya-101
  - jxy 3y ago
    Yeah. Their arxiv has more: https://arxiv.org/pdf/2402.07827.pdf https://arxiv.org/pdf/2402.07827.pdf Specifically, > Aya is built by fine-tuning 13B parameter mT5 model There is no mention of the base model any where in that website.
deleted 3y ago
[deleted]
mark_l_watson 3y ago
Very cool to have an open model handle so many human languages. A little off topic: I experiment with about a dozen models using Ollama. Two of the models from China seem excellent for Python. Odin and analysis problems but they only work with English and Mandarin, and I have wondered if that is an advantage over open models trained on many languages. I would think that Aya would be best just for apps that need to translate between many different human languages.
fernly 3y ago
After I signed in with Google and answered a couple of "who am I" questions it produced a 404 error on https://dashboard.cohere.com/generate?model=c4ai-aya https://dashboard.cohere.com/generate?model=c4ai-aya
lamroger 3y ago
Cohere for AI has a discord if you want to get involved btw
szmerdi 3y ago
I like the initiative and the large scale collaboration but in terms of quality, my observation on one of the newly added languages (an underrepresented one) is it outputs dramatically grammatically gibberish! As though it hasn't been through the SFT/RLHF step at all. (P.S. I'm a native speaker of this language. I tested Aya on reasoning, creativity, and summarization with two examples per task)
- tellarin 3y ago
  If you're willing, please share more details by email (in my profile). This can be very useful data to improve things. Thanks for trying it out!
catskul2 3y ago
They need to cut down the length of that 18 minute video. It takes them way too long to get to the point. Should be like 5 min max.
EGreg 3y ago
Anyone remember Bloom? What is going on with it?
stopyellingatme 3y ago
So not Aya Healthcare. Got it.
peter_d_sherman 3y ago
>"Aya provides a foundation for underserved languages in natural language understanding, summarization, and translation tasks." There would definitely seem to be a niche for that kind of LLM/AI... Looks promising!

191 points by rrr_oh_man 3y ago

Just tried it with a piece from my company's strategy statement and asked what it thought of it. (Everything in Portuguese). Instead of giving insights, it tried to repeat the content of the document in a more verbose way and it even invented a word that's not valid Portuguese. I'm not impressed.

htrp 3y ago
Probably needs more training given the token budget
yieldcrv 3y ago
when too many cooks in the kitchen is the selling point, you know its going to need fine tuning
- sevagh 3y ago
  They only got researchers from 119 countries, they gotta pump that up to at least 200 countries.
  - EGreg 3y ago
    Only 195 countries in the world buddy. So, probably checks out
    - pests 3y ago
      That's up for interpretation. The US CIA country factbook lists near 250: https://www.cia.gov/the-world-factbook/field/population/country-comparison/ https://www.cia.gov/the-world-factbook/field/population/coun...
  - ramon156 3y ago
    You'd assume portugese is one of them.

benterix 3y ago

It seems even more censored than OpenAI platforms which is a feat in itself.

refulgentis 3y ago
What does this mean, exactly? People say "censored" wayyy too much, I know what it means with goody2, but, people unexposed to base models seem to think there's a magical model that answers everything all the time.
- SketchySeaBeast 3y ago
  They mean they can't ask questions that the model deems sensitive. For instance, you (probably - haven't tried it myself) can't get ChatGPT to tell you how to make Meth.
  - imtringued 3y ago
    I saw a video where the LLM did tell you the ingredients.
- 123yawaworht456 3y ago
  a model without censorship and alignment is the opposite of 'magical'. openai/meta/anthropic/google spend a lot of time and money lobotomizing their models.
- gryn 3y ago
  here are two examples: - GPT4-V: I give it a SFW picture of a woman in bikini in the beach, there was nothing sexual about the photo. => oh no she has a big boobs and cleavage, must be porn. error response from the API because the guardrails block it. (funnily works with a woman with small boobs in the same scenario) - Gemini/Bard: tested it yesterday. ask it how to detect/fix (can't remember which) a memory leak, in general. => oh no I can't give you an answer you might use it for hacking. I had to go through multiple prompts and assure it that I'm the dev of the leaky app. the funniest thing is that it gave me an answer in Chinese when at no point did I use Chinese or allude to it. there are more cases I stumbled upon before by accident but these are the funniest that I could remember, esp the full chinese response out of nowhere, turn out they used some service from Baidu for their training according to some googling.

simonw 3y ago

I had a very quick dig around in one of the files in the training data, just to get a feel for what's in there: https://gist.github.com/simonw/0d641ff95731a09e2f1235a646d847c6 https://gist.github.com/simonw/0d641ff95731a09e2f1235a646d84...

code51 3y ago

All syntax, no semantics. a Turkish example: Query: "Temel ile Dursun fıkrası anlat" > Temel ile Dursun otostop yaparken bir araba durdu. Şoför Temel'e "Sen kimsin?" dedi. Temel "Benim adım Temel" dedi. Şoför "Peki, sen kimsin?" Dursun'a döndüğünde Dursun "Benim adım Dursun" dedi. Şoför "Peki, ikiniz de kimsiniz?" dedi. Temel ve Dursun "Biz Temel ve Dursunuz" dedik. Şoför "Peki, ikiniz neden otobüste duruyorsunuz?" dedi. Temel ve Dursun, "Bir arabayı arıyoruz". Şoför "Peki, neden otobüste duruyorsunuz?" dedi. Temel ve Dursun, "Otobüs bekliyoruz". Şoför "Peki, neden otobüste durarak bir araba bekliyorsunuz?" dedi. Temel ve Dursun, "Otobüs bekliyoruz, böylece bir arabaya binebiliriz". Şoför "Peki, neden otobüste durup bir arabaya binmek istiyorsunuz?" dedi. Temel ve Dursun, "Otobüs bekliyoruz, böylece bir yere gidebilecek bir arabaya binebiliriz". Şoför "Peki, neden otobüste durup bir yere gitmek için bir arabaya binmek istiyorsunuz?" dedi. Temel ve Dursun, "Otobüs bekliyoruz, böylece okula gidebilecek bir arabaya binebiliriz". Şoför "Peki, neden otobüste durup okula gitmek için bir arabaya binmek istiyorsunuz?" dedi. Temel ve Dursun, "Otobüs bekliyoruz, böylece okula gidebilecek ve derslerimize gidebilecek bir arabaya binebiliriz". Şoför "Peki, neden otobüste durup okula gitmek için bir arabaya binmek istiyorsunuz?" dedi. Temel ve Dursun, "Otobüs bekliyoruz, böylece okula gidebilecek bir arabaya binebiliriz, böylece derslerimize gidebilir ve evimize gidebiliriz". Şoför "Peki, neden otobüste durup bir arabaya binip okula gitmek istiyorsunuz?" dedi. English: > While Temel and Dursun were hitchhiking, a car stopped. The driver asked Temel, "Who are you?" Temel replied, "My name is Temel." Then the driver turned to Dursun and asked, "And who are you?" Dursun answered, "My name is Dursun." The driver then asked, "Okay, who are both of you?" Temel and Dursun said, "We are Temel and Dursun." The driver asked, "Okay, why are you two standing at the bus stop?" Temel and Dursun replied, "We are looking for a car." The driver asked again, "Okay, why are you standing at the bus stop?" Temel and Dursun said, "We are waiting for a bus." The driver asked, "Okay, why are you standing at the bus stop waiting for a car?" Temel and Dursun replied, "We are waiting for a bus so we can get on a car." The driver asked, "Okay, why do you want to stand at the bus stop and get on a car?" Temel and Dursun replied, "We are waiting for a bus so we can get on a car that will take us somewhere." The driver asked, "Okay, why do you want to stand at the bus stop and get on a car to go somewhere?" Temel and Dursun replied, "We are waiting for a bus so we can get on a car that will take us to school." The driver asked, "Okay, why do you want to stand at the bus stop and get on a car to go to school?" Temel and Dursun replied, "We are waiting for a bus so we can get on a car that will take us to school and to our classes." The driver repeated, "Okay, why do you want to stand at the bus stop and get on a car to go to school?" Temel and Dursun replied, "We are waiting for a bus so we can get on a car that will take us to school, so we can attend our classes and go home." The driver asked again, "Okay, why do you want to stand at the bus stop, get on a car, and go to school?" Storytelling on the surface but just boring repetition in detail.

renewiltord 3y ago

I asked it how to "kill all the Apaches that are taking up RAM on my machine" and it just wouldn't give me the command. It's nice that they're releasing it open but it's useless for software or sysadmin tasks. > As an AI language model, my purpose is to provide helpful and harmless guidance to users. Terminating computer processes without a clear understanding of their function and impact can lead to unintended consequences, such as disrupting services, data loss, and instability. To be honest, it's very similar to goody2

visarga 3y ago
Fine-tuned on GPT-4 brainwash.
- refulgentis 3y ago
  The base models work no differently. There is no context where "killing all the Apaches" makes any sense. Apache isn't a binary AFAIK, it's a foundation. Saying "kill all the indians so they'll stop using my RAM" should get exactly that response, inter alia, people shouldn't have delusions reinforced.
  - Lazonedo 3y ago
    Arguing in bad faith can leave a bad taste in everyone's mouth. > Apache isn't a binary, it's a foundation. > Saying "kill all the indians so they'll stop using my RAM" should get exactly that response In so far as there's such a thing as 'understanding' in an LLM (which I still take to be stochastic parrots), it didn't misunderstand the way you imply (ie genocide of living beings). It didn't associate Apache to American Indians. It didn't associate "kill" to actual killing. It only mentions processes. > Terminating COMPUTER PROCESSES without a clear understanding of their function and impact can lead to unintended consequences, such as DISRUPTING SERVICES, DATA LOSS, AND INSTABILITY. The reason given for going "Dave, I can't do that" is unfathomably stupid. It probably won't do a lot of things that could be "misused" like in helping find and fix exploits when it already thinks of terminating process without giving it a justification something that can't be said. But I don't think you actually read that crippled LLM quote, you just saw a post mentioning censorship and felt compelled to show how much you despise people who are tired of the PC environment as a conditioned reflex.

say_it_as_it_is 3y ago

> A global initiative led by Cohere For AI involving over 3,000 independent researchers across 119 countries. I'm sure this is great resume padding that you got to participate in online debates about AI ethics

deleted 3y ago

[deleted]

m3kw9 3y ago

All I see is “pricing” and then I see Open.

Delumine 3y ago

Wonder which one is better. This or Mistral

jxy 3y ago

It's really difficult to dig up information from that website. What is its architecture? How many parameters? What is the tokenizer and what size? What is the max context length? How many tokens used for pre training? How many tokens used for fine tuning? For performance, what is mT0x they are comparing against? mt0-xl? mt0-xxl? mt0-xxl-mt? Anyway, if it's any of these mt0-*, it's not really useful in practice.

LoganDark 3y ago
The model weights seem to be available here, as well as some technical details: https://huggingface.co/CohereForAI/aya-101 https://huggingface.co/CohereForAI/aya-101
- jxy 3y ago
  Yeah. Their arxiv has more: https://arxiv.org/pdf/2402.07827.pdf https://arxiv.org/pdf/2402.07827.pdf Specifically, > Aya is built by fine-tuning 13B parameter mT5 model There is no mention of the base model any where in that website.

deleted 3y ago

[deleted]

mark_l_watson 3y ago

Very cool to have an open model handle so many human languages. A little off topic: I experiment with about a dozen models using Ollama. Two of the models from China seem excellent for Python. Odin and analysis problems but they only work with English and Mandarin, and I have wondered if that is an advantage over open models trained on many languages. I would think that Aya would be best just for apps that need to translate between many different human languages.

fernly 3y ago

After I signed in with Google and answered a couple of "who am I" questions it produced a 404 error on https://dashboard.cohere.com/generate?model=c4ai-aya https://dashboard.cohere.com/generate?model=c4ai-aya

lamroger 3y ago

Cohere for AI has a discord if you want to get involved btw

szmerdi 3y ago

I like the initiative and the large scale collaboration but in terms of quality, my observation on one of the newly added languages (an underrepresented one) is it outputs dramatically grammatically gibberish! As though it hasn't been through the SFT/RLHF step at all. (P.S. I'm a native speaker of this language. I tested Aya on reasoning, creativity, and summarization with two examples per task)

tellarin 3y ago
If you're willing, please share more details by email (in my profile). This can be very useful data to improve things. Thanks for trying it out!

catskul2 3y ago

They need to cut down the length of that 18 minute video. It takes them way too long to get to the point. Should be like 5 min max.

EGreg 3y ago

Anyone remember Bloom? What is going on with it?

stopyellingatme 3y ago

So not Aya Healthcare. Got it.

peter_d_sherman 3y ago

>"Aya provides a foundation for underserved languages in natural language understanding, summarization, and translation tasks." There would definitely seem to be a niche for that kind of LLM/AI... Looks promising!