7 ms·

BLOOM: The largest open multilingual language model

160 points by georgehill 4y ago

PoignardAzur 4y ago
You can try the model on HuggingFace: https://huggingface.co/bigscience/bloom https://huggingface.co/bigscience/bloom Though it requires an account, unlike other models on the hub (I guess they're hoping to get new users from the hype of this model). Bloom looks pretty exciting. It's reportedly as performant as GPT-3 (I haven't tested it enough to confirm, but what little testing I did gave okay results), but the model is completely open-source; if you can afford some cloud compute, you can just upload the model on your preferred cloud provider and just generate whatever you want from it, good or evil. I'm expecting the coming 12 months to be pretty interesting for text generation.
- dekhn 4y ago
  If you create a hugginface account, can you download the model and weights and run inference locally (I have a nvidia 3080 Ti on a 32-core machine with 64GB of RAM)? Or do they just host the inference server for you? IIUC the model design is open so I assume the weights are too.
  - moyix 4y ago
    You can actually download the weights without an account: git clone https://huggingface.co/bigscience/bloom You will need Git LFS and ~330GB of free space though. It is possible to run inference locally, even without very much RAM/VRAM (though it will be very slow). Get the most recent versions of "transformers" and "accelerate" from git, and then: import torch from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained('./bloom') inputs = tokenizer("Hacker News is", return_tensors="pt") model = AutoModelForCausalLM.from_pretrained('./bloom', device_map="auto", offload_folder='offload', torch_dtype=torch.bfloat16) output = model.generate(inputs["input_ids"].to(0), min_length=30, max_length=30, do_sample=True) print(tokenizer.decode(output[0].tolist())) After the model is loaded you can also look at where it put each part of the model (GPU, CPU RAM, or on disk): print(model.hf_device_map) Here's a screenshot of it in action on my machine, which has 256GB RAM and 2xA6000 GPUs: https://twitter.com/moyix/status/1546961979566325760 https://twitter.com/moyix/status/1546961979566325760
    - dekhn 4y ago
      Thanks, that's what I was looking for (double thanks for the example inference code). I'll queue up the download and try to get the dependencies working. Comment 1: git-lfs is now close to saturating my gigabit ethernet connection, around 990Mb/sec
      - dekhn 4y ago
        Comment 2: filled up the disk on my first computer and had to find an actual hard drive since none of my SSDs have 330GB. It seems more like 500+GB on disk. Haven't gotten to inference yet.
        moyix 4y ago
        I noticed this too -- I think it's a git LFS issue? The weights themselves are only 330GB, but the .git directory adds another 330GB: $ du -csh pytorch_model*.bin | tail -n 1 329G total $ du -sh .git 329G .git Not sure if there's a way to get git-lfs not to do that...
        dekhn 4y ago
        I haven't used LFS before but it seems to buffer incompletely downloaded files in .git/lfs/incomplete and a copy of "objects" in .git/lfs/objects (I am one of those people who absolutely no interest in how git's DB works, nor how its blob DB works).
    - epups 4y ago
      If I understand correctly, it took you 7 minutes to generate a single prompt in your machine?
      - moyix 4y ago
        Yep. The model is too big to fit on the two GPUs I have (even though each one is 48GB). So instead it gets split between the GPUs, CPU RAM, and some parts even have to sit on disk (SSD). This makes it much slower than if everything were on the GPU. I believe there are performance enhancements planned in Accelerate that will allow layers to be preloaded in parallel as other layers are executing inference, which should make things faster: https://github.com/huggingface/accelerate/issues/512#issuecomment-1182480255 https://github.com/huggingface/accelerate/issues/512#issueco...
  - amrrs 4y ago
    You can use the model without Hugging Face account - https://www.youtube.com/watch?v=bGBY3K-_bio https://www.youtube.com/watch?v=bGBY3K-_bio 13:23
- minimaxir 4y ago
  > Though it requires an account, unlike other models on the hub (I guess they're hoping to get new users from the hype of this model). This model is larger than anything HF has done, so resource usage likely has to be constrained a bit (GPT-3 was restricted at first for similar reasons, in addition to the potential abuse issues)
- schoen 4y ago
  > but the model is completely open-source; if you can afford some cloud compute, you can just upload the model on your preferred cloud provider and just generate whatever you want from it, good or evil It seems that the developers' position is that it's not fully open-source, and that some potentially evil or unethical uses are actually restricted: https://huggingface.co/spaces/bigscience/license https://huggingface.co/spaces/bigscience/license
  - PoignardAzur 4y ago
    Right, it's open source with a license that forbids using it for evil. Practically speaking, though, you can just download it on a private server and do evil and afaict nobody can stop you or know you're using BLOOM.
    - schoen 4y ago
      > Right, it's open source with a license that forbids using it for evil. That's not considered open source, though. https://opensource.org/osd-annotated https://opensource.org/osd-annotated Also compare https://web.archive.org/web/20130203112329/http://dev.hasenj.org/post/3272592502 https://web.archive.org/web/20130203112329/http://dev.hasenj... (although IBM wasn't asking specifically about the open source nature of the license).
      - PoignardAzur 4y ago
        Alright, fine. It's open-source-with-an-asterisk, which is completely different from open-source in every way, and happens to involve having access to the source of the thing. The asterisk says "I'm not allowed to say it's open source because some website I don't care about says it's not the right definition".
ceeplusplus 4y ago
- bee_rider 4y ago
  Was this message generated via BLOOM or GPT-3?
  - r_hoods_ghost 4y ago
    Given the level of sophistication, I'm going with Eliza...
ldjkfkdsjnv 4y ago
I'm calling it now, these models will just grow and grow, their responses are reflexive probabilistic responses. HN will scream its not a AI since it doesn't understand the world, then we will discover that human beings are just reflexive probabilistic machines. And the dumber reflexivity (which is what the models have (or rather dumber humans have)) are not logically incorrect, they just arent complex enough to hold all the information, but still very good at spitting out what is there. A deep understanding of human intelligence I believe will reveal that this is the case, and humanity will be cracked open
- tetris11 4y ago
  After reading the Google Lamda transcripts, and genuinely believing the AI there to be sentient, I question how people judge the sentience of others
  - zaptrem 4y ago
    Lambda and these language models are not currently capable maintaining an internal train of thought that isn't expressed as token context. It has no memory of the world other than what we tell it to have. It can only "think" as long as it takes to infer a token. I think these big language models might get closer to something we would all call sentience once they can refer directly to their weights' previous activations for context instead of just tokens.
    - naasking 4y ago
      > Lambda and these language models are not currently capable maintaining an internal train of thought that isn't expressed as token context. Why would that necessarily be insufficient for sentience? Sentience is just about the ability to "feel". Whatever "feeling" is, mechanistically, it will be some process that accepts some information and produces some output for these "feelings". It's not obvious that any kind of "memory" has to be part of the input to that process. For all we know, the transformer model is producing feelings as it computes, ie. that the next generated word is selected because it "feels right", where "feels right" simply means it's the most statistically likely word to follow the current token. Can you prove that this isn't basically what that gut feeling is for people? We don't actually know what "feelings" are fundamentally, so most of the people dismissing LaMDA's sentience don't really understand the can of worms they just stepped in.
Barrin92 4y ago
"BLOOM will be the first language model with over 100B parameters ever created. This is the culmination of a year of work involving over 1000 researchers from 70+ countries and 250+ institutions, leading to a final run of 117 days (March 11 - July 6) training the BLOOM model on the Jean Zay supercomputer in the south of Paris, France thanks to a compute grant worth an estimated €3M from French research agencies CNRS and GENCI." Thanks to the researchers, institutions and the French government for providing the resources to make that happen. I hope more countries on the continent follow this model of open access and funding for AI research.
justinc-md 4y ago
How distributable is training a model like this? Do all the gpus need to be physically close together or well networked?
ninjin 4y ago
Firstly, it is now pretty much exactly two years since OpenAI announced GPT-3 and it is great to see that access to these models – for scientists in particular – is becoming increasingly commonplace. I should also note that Hugging Face really have taken a “big tent” stance on the development of this model and their business in general. This is refreshing given that no other entity capable of raising the funds and manpower to develop these kinds of models have taken a similar stance (despite much deeper pockets…). Yes, Hugging Face is running on venture capital, but at least for now I see no reason to be cynical about their efforts that in every way appear genuine to me. Still, it is not all great. OpenAI, DeepMind, Google, etc. continues to produce models that are completely closed and thus can never be the subject to analysis from the rest of the community. Models which all dwarf GPT-3 at this point. My fear is that those of us arguing for access will continue to play catch up – maybe even forever. However, while BLOOM is more open than say FAIR’s OPT [1] – which had the gall to call itself both “open” and “democratising” despite restricting commercial usage in its “open” license – I believe there is a discussion to be had about how they (and many others in the community) use the word “open”. [1]: https://ai.facebook.com/blog/democratizing-access-to-large-scale-language-models-with-opt-175b https://ai.facebook.com/blog/democratizing-access-to-large-s... While I am not a lawyer, I am somewhat familiar with licenses and have gone through the BigScience RAIL License v1.0 [2], the associated blog post [3], and some background papers and documents to better understand how Hugging Face and its community motivate the licensing. [2]: https://huggingface.co/spaces/bigscience/license https://huggingface.co/spaces/bigscience/license [3]: https://bigscience.huggingface.co/blog/the-bigscience-rail-license https://bigscience.huggingface.co/blog/the-bigscience-rail-l... From the license itself: “[T]his License aims to strike a balance between both [open and responsible AI development] in order to enable responsible open-science…” I am of the opinion that this is impossible, as the definition of “responsible” they use (see Appendix A) is directly at odds with every definition of “open” that I am aware of. Furthermore, I find phrasings such as “Although the BigScience community does not aim to impose its values on potential users of this Model, it is determined to take tangible steps towards protecting the community from inappropriate uses of the work being developed by BigScience.” confusing, as it both claims not to seek to impose its values and to seek to impose its values in the very same sentence! In essence, I wish entities such as FAIR and Hugging Face would stop using the term “open” when they so clearly disagree with all accepted definitions of it. In the blog post related to their license [2], they explicitly state that they are incompatible with the OSI definition of open which for example the ethical source movement [4] also agrees with and thus avoids labelling themselves as “open”. [4]: https://ethicalsource.dev https://ethicalsource.dev Okay, so BLOOM is not “open” as in “open source”, then what is it “open” like? Putting limitations on usage makes it run afoul of definitions of “open access”: “free availability and unrestricted use” [5]. Although I have to admit that I am less familiar with how this community would view imposing ethical considerations on readers of science, I am fairly certain that barring access to scientific information based on anything other than very clear ethical considerations (say, easily-deployable, pandemic-level virus mutations) would run afoul of a great majority of the open access movement. [5]: http://legacy.earlham.edu/~peters/fos/overview.htm http://legacy.earlham.edu/~peters/fos/overview.htm Right, so it is not “open” as in “open source” and nor “open” as in “open access”. How about “open science”? To the best of my knowledge there is no widely accepted definition of open science, but most seem to model themselves around open access, open source, and open data. Thus falling back on OSI’s definition and to quote the Open Knowledge Foundation: “Knowledge is open if anyone is free to access, use, modify, and share it — subject, at most, to measures that preserve provenance and openness” [6]. Alternatively, their short definition makes it even clearer: “Open data and content can be freely used, modified, and shared by anyone for any purpose” [7]. [6]: http://opendefinition.org/od/2.1/en http://opendefinition.org/od/2.1/en [7]: http://opendefinition.org http://opendefinition.org In summary, I do not believe that you can argue that anything that is “ethically licensed” is “open” by any reasonable definition. This however is fine, as anyone is free to dictate how the fruit of their labour is to be used. However, even when I am being charitable, it is difficult for me not to feel that there is a desire to ride on the coattails of the positive semantics attached with the word “open” that has taken arguably more than 30 years for others to build up and I feel that it is ethically questionable to muddy the waters around a term that others have worked hard to define and build communities upon. You are “ethical”, “responsible”, or some other nice term, but not “open” – own it. Lastly, why do I as a researcher in this area object? Especially given that I think I can agree to every single ethical point in Appendix A. Firstly – as I have argued above – it annoys me greatly that multiple entities outside the “open” movements use the terms frivolously and I believe to their own benefit. This feels like appropriation if ever I saw it. Secondly, I believe that despite their good intentions multiple efforts motivated by ethics are reversing the direction of openness of where science has been heading over the last twenty years. Even if only marginally so. Thirdly (and lastly), I believe these efforts to achieve their ethical goals will prove to have little to no effect and that efforts are more strongly warranted elsewhere. Actors with the capability to cause harm will either be able to ignore the license or are likely to have already obtained these kinds of models independent of publicly released models (not to mention that cutting-edge models are developed solely by multi-national corporations without ethical quandaries). Thus, to me, taking a legalistic approach is akin to paying for indulgences and staying in our academic ivory towers. Rather, I think successful initiatives to restrict potential harm much be wider in scope akin to the Campaign to Stop Killer Robots [8] or, even better, to educate the general public about these models and work on developing countermeasures to support public trust and communication once these models inevitably become commonplace are necessary and receiving far too little attention in the community in favour of hypothetical risk and legal word play. [8]: https://en.wikipedia.org/wiki/Campaign_to_Stop_Killer_Robots https://en.wikipedia.org/wiki/Campaign_to_Stop_Killer_Robots Unrelated notes: While browsing the license I noted that it has patent clauses akin to Apache 2.0, which I found interesting. It also contains a somewhat vague requirement to keep your models in sync with the upstream: “You shall undertake reasonable efforts to use the latest version of the Model.” I assume that this is intended to be used if the model is “patched” to restrict harm. But it creates another ongoing relationship between users of the model and Hugging Face.
lostmsu 4y ago
Any model benchmarks on common tasks?
- lostmsu 4y ago
  They are available in this section: https://huggingface.co/bigscience/bloom#evaluation https://huggingface.co/bigscience/bloom#evaluation

160 points by georgehill 4y ago

You can try the model on HuggingFace: https://huggingface.co/bigscience/bloom https://huggingface.co/bigscience/bloom Though it requires an account, unlike other models on the hub (I guess they're hoping to get new users from the hype of this model). Bloom looks pretty exciting. It's reportedly as performant as GPT-3 (I haven't tested it enough to confirm, but what little testing I did gave okay results), but the model is completely open-source; if you can afford some cloud compute, you can just upload the model on your preferred cloud provider and just generate whatever you want from it, good or evil. I'm expecting the coming 12 months to be pretty interesting for text generation.

dekhn 4y ago
If you create a hugginface account, can you download the model and weights and run inference locally (I have a nvidia 3080 Ti on a 32-core machine with 64GB of RAM)? Or do they just host the inference server for you? IIUC the model design is open so I assume the weights are too.
- moyix 4y ago
  You can actually download the weights without an account: git clone https://huggingface.co/bigscience/bloom You will need Git LFS and ~330GB of free space though. It is possible to run inference locally, even without very much RAM/VRAM (though it will be very slow). Get the most recent versions of "transformers" and "accelerate" from git, and then: import torch from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained('./bloom') inputs = tokenizer("Hacker News is", return_tensors="pt") model = AutoModelForCausalLM.from_pretrained('./bloom', device_map="auto", offload_folder='offload', torch_dtype=torch.bfloat16) output = model.generate(inputs["input_ids"].to(0), min_length=30, max_length=30, do_sample=True) print(tokenizer.decode(output[0].tolist())) After the model is loaded you can also look at where it put each part of the model (GPU, CPU RAM, or on disk): print(model.hf_device_map) Here's a screenshot of it in action on my machine, which has 256GB RAM and 2xA6000 GPUs: https://twitter.com/moyix/status/1546961979566325760 https://twitter.com/moyix/status/1546961979566325760
  - dekhn 4y ago
    Thanks, that's what I was looking for (double thanks for the example inference code). I'll queue up the download and try to get the dependencies working. Comment 1: git-lfs is now close to saturating my gigabit ethernet connection, around 990Mb/sec
    - dekhn 4y ago
      Comment 2: filled up the disk on my first computer and had to find an actual hard drive since none of my SSDs have 330GB. It seems more like 500+GB on disk. Haven't gotten to inference yet.
      - moyix 4y ago
        I noticed this too -- I think it's a git LFS issue? The weights themselves are only 330GB, but the .git directory adds another 330GB: $ du -csh pytorch_model*.bin | tail -n 1 329G total $ du -sh .git 329G .git Not sure if there's a way to get git-lfs not to do that...
        dekhn 4y ago
        I haven't used LFS before but it seems to buffer incompletely downloaded files in .git/lfs/incomplete and a copy of "objects" in .git/lfs/objects (I am one of those people who absolutely no interest in how git's DB works, nor how its blob DB works).
  - epups 4y ago
    If I understand correctly, it took you 7 minutes to generate a single prompt in your machine?
    - moyix 4y ago
      Yep. The model is too big to fit on the two GPUs I have (even though each one is 48GB). So instead it gets split between the GPUs, CPU RAM, and some parts even have to sit on disk (SSD). This makes it much slower than if everything were on the GPU. I believe there are performance enhancements planned in Accelerate that will allow layers to be preloaded in parallel as other layers are executing inference, which should make things faster: https://github.com/huggingface/accelerate/issues/512#issuecomment-1182480255 https://github.com/huggingface/accelerate/issues/512#issueco...
- amrrs 4y ago
  You can use the model without Hugging Face account - https://www.youtube.com/watch?v=bGBY3K-_bio https://www.youtube.com/watch?v=bGBY3K-_bio 13:23
minimaxir 4y ago
> Though it requires an account, unlike other models on the hub (I guess they're hoping to get new users from the hype of this model). This model is larger than anything HF has done, so resource usage likely has to be constrained a bit (GPT-3 was restricted at first for similar reasons, in addition to the potential abuse issues)
schoen 4y ago
> but the model is completely open-source; if you can afford some cloud compute, you can just upload the model on your preferred cloud provider and just generate whatever you want from it, good or evil It seems that the developers' position is that it's not fully open-source, and that some potentially evil or unethical uses are actually restricted: https://huggingface.co/spaces/bigscience/license https://huggingface.co/spaces/bigscience/license
- PoignardAzur 4y ago
  Right, it's open source with a license that forbids using it for evil. Practically speaking, though, you can just download it on a private server and do evil and afaict nobody can stop you or know you're using BLOOM.
  - schoen 4y ago
    > Right, it's open source with a license that forbids using it for evil. That's not considered open source, though. https://opensource.org/osd-annotated https://opensource.org/osd-annotated Also compare https://web.archive.org/web/20130203112329/http://dev.hasenj.org/post/3272592502 https://web.archive.org/web/20130203112329/http://dev.hasenj... (although IBM wasn't asking specifically about the open source nature of the license).
    - PoignardAzur 4y ago
      Alright, fine. It's open-source-with-an-asterisk, which is completely different from open-source in every way, and happens to involve having access to the source of the thing. The asterisk says "I'm not allowed to say it's open source because some website I don't care about says it's not the right definition".

ceeplusplus 4y ago

bee_rider 4y ago
Was this message generated via BLOOM or GPT-3?
- r_hoods_ghost 4y ago
  Given the level of sophistication, I'm going with Eliza...

ldjkfkdsjnv 4y ago

I'm calling it now, these models will just grow and grow, their responses are reflexive probabilistic responses. HN will scream its not a AI since it doesn't understand the world, then we will discover that human beings are just reflexive probabilistic machines. And the dumber reflexivity (which is what the models have (or rather dumber humans have)) are not logically incorrect, they just arent complex enough to hold all the information, but still very good at spitting out what is there. A deep understanding of human intelligence I believe will reveal that this is the case, and humanity will be cracked open

tetris11 4y ago
After reading the Google Lamda transcripts, and genuinely believing the AI there to be sentient, I question how people judge the sentience of others
- zaptrem 4y ago
  Lambda and these language models are not currently capable maintaining an internal train of thought that isn't expressed as token context. It has no memory of the world other than what we tell it to have. It can only "think" as long as it takes to infer a token. I think these big language models might get closer to something we would all call sentience once they can refer directly to their weights' previous activations for context instead of just tokens.
  - naasking 4y ago
    > Lambda and these language models are not currently capable maintaining an internal train of thought that isn't expressed as token context. Why would that necessarily be insufficient for sentience? Sentience is just about the ability to "feel". Whatever "feeling" is, mechanistically, it will be some process that accepts some information and produces some output for these "feelings". It's not obvious that any kind of "memory" has to be part of the input to that process. For all we know, the transformer model is producing feelings as it computes, ie. that the next generated word is selected because it "feels right", where "feels right" simply means it's the most statistically likely word to follow the current token. Can you prove that this isn't basically what that gut feeling is for people? We don't actually know what "feelings" are fundamentally, so most of the people dismissing LaMDA's sentience don't really understand the can of worms they just stepped in.

Barrin92 4y ago

"BLOOM will be the first language model with over 100B parameters ever created. This is the culmination of a year of work involving over 1000 researchers from 70+ countries and 250+ institutions, leading to a final run of 117 days (March 11 - July 6) training the BLOOM model on the Jean Zay supercomputer in the south of Paris, France thanks to a compute grant worth an estimated €3M from French research agencies CNRS and GENCI." Thanks to the researchers, institutions and the French government for providing the resources to make that happen. I hope more countries on the continent follow this model of open access and funding for AI research.

justinc-md 4y ago

How distributable is training a model like this? Do all the gpus need to be physically close together or well networked?

ninjin 4y ago

Firstly, it is now pretty much exactly two years since OpenAI announced GPT-3 and it is great to see that access to these models – for scientists in particular – is becoming increasingly commonplace. I should also note that Hugging Face really have taken a “big tent” stance on the development of this model and their business in general. This is refreshing given that no other entity capable of raising the funds and manpower to develop these kinds of models have taken a similar stance (despite much deeper pockets…). Yes, Hugging Face is running on venture capital, but at least for now I see no reason to be cynical about their efforts that in every way appear genuine to me. Still, it is not all great. OpenAI, DeepMind, Google, etc. continues to produce models that are completely closed and thus can never be the subject to analysis from the rest of the community. Models which all dwarf GPT-3 at this point. My fear is that those of us arguing for access will continue to play catch up – maybe even forever. However, while BLOOM is more open than say FAIR’s OPT [1] – which had the gall to call itself both “open” and “democratising” despite restricting commercial usage in its “open” license – I believe there is a discussion to be had about how they (and many others in the community) use the word “open”. [1]: https://ai.facebook.com/blog/democratizing-access-to-large-scale-language-models-with-opt-175b https://ai.facebook.com/blog/democratizing-access-to-large-s... While I am not a lawyer, I am somewhat familiar with licenses and have gone through the BigScience RAIL License v1.0 [2], the associated blog post [3], and some background papers and documents to better understand how Hugging Face and its community motivate the licensing. [2]: https://huggingface.co/spaces/bigscience/license https://huggingface.co/spaces/bigscience/license [3]: https://bigscience.huggingface.co/blog/the-bigscience-rail-license https://bigscience.huggingface.co/blog/the-bigscience-rail-l... From the license itself: “[T]his License aims to strike a balance between both [open and responsible AI development] in order to enable responsible open-science…” I am of the opinion that this is impossible, as the definition of “responsible” they use (see Appendix A) is directly at odds with every definition of “open” that I am aware of. Furthermore, I find phrasings such as “Although the BigScience community does not aim to impose its values on potential users of this Model, it is determined to take tangible steps towards protecting the community from inappropriate uses of the work being developed by BigScience.” confusing, as it both claims not to seek to impose its values and to seek to impose its values in the very same sentence! In essence, I wish entities such as FAIR and Hugging Face would stop using the term “open” when they so clearly disagree with all accepted definitions of it. In the blog post related to their license [2], they explicitly state that they are incompatible with the OSI definition of open which for example the ethical source movement [4] also agrees with and thus avoids labelling themselves as “open”. [4]: https://ethicalsource.dev https://ethicalsource.dev Okay, so BLOOM is not “open” as in “open source”, then what is it “open” like? Putting limitations on usage makes it run afoul of definitions of “open access”: “free availability and unrestricted use” [5]. Although I have to admit that I am less familiar with how this community would view imposing ethical considerations on readers of science, I am fairly certain that barring access to scientific information based on anything other than very clear ethical considerations (say, easily-deployable, pandemic-level virus mutations) would run afoul of a great majority of the open access movement. [5]: http://legacy.earlham.edu/~peters/fos/overview.htm http://legacy.earlham.edu/~peters/fos/overview.htm Right, so it is not “open” as in “open source” and nor “open” as in “open access”. How about “open science”? To the best of my knowledge there is no widely accepted definition of open science, but most seem to model themselves around open access, open source, and open data. Thus falling back on OSI’s definition and to quote the Open Knowledge Foundation: “Knowledge is open if anyone is free to access, use, modify, and share it — subject, at most, to measures that preserve provenance and openness” [6]. Alternatively, their short definition makes it even clearer: “Open data and content can be freely used, modified, and shared by anyone for any purpose” [7]. [6]: http://opendefinition.org/od/2.1/en http://opendefinition.org/od/2.1/en [7]: http://opendefinition.org http://opendefinition.org In summary, I do not believe that you can argue that anything that is “ethically licensed” is “open” by any reasonable definition. This however is fine, as anyone is free to dictate how the fruit of their labour is to be used. However, even when I am being charitable, it is difficult for me not to feel that there is a desire to ride on the coattails of the positive semantics attached with the word “open” that has taken arguably more than 30 years for others to build up and I feel that it is ethically questionable to muddy the waters around a term that others have worked hard to define and build communities upon. You are “ethical”, “responsible”, or some other nice term, but not “open” – own it. Lastly, why do I as a researcher in this area object? Especially given that I think I can agree to every single ethical point in Appendix A. Firstly – as I have argued above – it annoys me greatly that multiple entities outside the “open” movements use the terms frivolously and I believe to their own benefit. This feels like appropriation if ever I saw it. Secondly, I believe that despite their good intentions multiple efforts motivated by ethics are reversing the direction of openness of where science has been heading over the last twenty years. Even if only marginally so. Thirdly (and lastly), I believe these efforts to achieve their ethical goals will prove to have little to no effect and that efforts are more strongly warranted elsewhere. Actors with the capability to cause harm will either be able to ignore the license or are likely to have already obtained these kinds of models independent of publicly released models (not to mention that cutting-edge models are developed solely by multi-national corporations without ethical quandaries). Thus, to me, taking a legalistic approach is akin to paying for indulgences and staying in our academic ivory towers. Rather, I think successful initiatives to restrict potential harm much be wider in scope akin to the Campaign to Stop Killer Robots [8] or, even better, to educate the general public about these models and work on developing countermeasures to support public trust and communication once these models inevitably become commonplace are necessary and receiving far too little attention in the community in favour of hypothetical risk and legal word play. [8]: https://en.wikipedia.org/wiki/Campaign_to_Stop_Killer_Robots https://en.wikipedia.org/wiki/Campaign_to_Stop_Killer_Robots Unrelated notes: While browsing the license I noted that it has patent clauses akin to Apache 2.0, which I found interesting. It also contains a somewhat vague requirement to keep your models in sync with the upstream: “You shall undertake reasonable efforts to use the latest version of the Model.” I assume that this is intended to be used if the model is “patched” to restrict harm. But it creates another ongoing relationship between users of the model and Hugging Face.

lostmsu 4y ago

Any model benchmarks on common tasks?

lostmsu 4y ago
They are available in this section: https://huggingface.co/bigscience/bloom#evaluation https://huggingface.co/bigscience/bloom#evaluation