5 ms·

Language-Agnostic Bert Sentence Embedding

74 points by theafh 6y ago

lacker 6y ago
After all the attention OpenAI got for making the GPT-3 API semi-publicly-available, I wonder if Google has considered making some of their research models available via API. It would be pretty neat to be able to try these things out, rather than just reading papers about them.
- MiroF 6y ago
  You've actually got it backwards. Releasing model weights has been common for a long time, it was OpenAI that regressed and refused to release its GPT models until it released GPT-3 hidden behind a semi-public API. The google BERT models are released pre-trained in full (as mentioned in the article), you can easily play around with them on your own. BERT isn't a forward LM like GPT, so it's a little less easy to play with for the uninitiated - although there are papers showing text generation with BERT.
  - colordrops 6y ago
    It's really ironic that they are regressing and keeping their research behind a curtain when the entire pretense for their existence (and their name) is to open things up.
    - est31 6y ago
      It's easier to hire people for a (claimed) mission driven company than for business evil corp that builds killer machines for the us military.
    - taneq 6y ago
      It’s not like it’s a new across-the-board policy for them. They were just being cautious in this particular case because they were worried about social ramifications. Arguably some other research should have been made less readily available like this (think deepfakes for example).
  - tmabraham 6y ago
    To be fair, IIRC OpenAI did release GPT-2-large after some community pressure, and it was somewhat doable for some people to actually train from scratch. GPT-3 is too large, so even if they released it, nobody apart from large companies like Google could do anything with it. If anything, they've made GPT-3 more open than it would have been if they just released the weights. At least that's my understanding. Feel free to correct me if I'm wrong.
    - MiroF 6y ago
      I'm not really up in arms about OpenAIs decision, but I just don't want people to frame this as "why can't more companies release models like OpenAI does" when in reality the opposite is pretty much true. GPT-2 wasn't really feasible for most actors to train from scratch, otherwise it would have been released by a third party. GPT-3 is technically feasible for a single actor to do inference with, although not really really.
      - tmabraham 6y ago
        I agree with your comment here but I think for the type and size of model, OpenAI has done the best they could with such large models. So I wouldn't say they've "regressed and refused to release its GPT models" they just had to take a different route with GPT models.
        tmabraham 6y ago
        People who disagree and are downvoting, may I ask why?
    - Der_Einzige 6y ago
      They only released GPT-2 large after other language models came out that were even larger, like T5...
    - liuliu 6y ago
      I don't know. It has 175 billion parameters, thus, about 500GiB for floating point parameters. If these are bfloat16, it is mere 300GiB. We know CPU is about 50 times slower than GPU for transformer models, hence, we are looking at 4 to 8 minutes per inference (parameter loading on demand from SSD takes 200 seconds or so, and probably the bottleneck here). If the parameters can be loaded into memory (seems you have to be on a >= 8-channel machine with unbuffered RAM or with buffered RAM), it will be probably 1 to 2 minutes per inference. Still in the realm of doing inference in homelab territory (barely).
  - chaoz_ 6y ago
    However, @lacker makes a valid point that most research papers lack interactive, well-designed API for people unfamiliar with the domain.
    - MiroF 6y ago
      by API we mean pretty UI? if not, huggingface/transformers is a pretty good API for the Google-brand transformers.
  - lettergram 6y ago
    I firmly believe what OpenAI is actually doing is building a dataset.
  - lopsidedBrain 6y ago
    For the uninitiated, WTH does semi-public mean? Did GPT-3 only provide an API but not the model?
    - IfOnlyYouKnew 6y ago
      They do only have an API and the (trained) model (parameters) are not released. But I believe the "semi-public" here refers to the fact that access to the API is limited to invitees (with the ability to add your name to a waitlist).
      - lopsidedBrain 6y ago
        Huh, for some reason I had just assumed that it must be open based on the buzz around it. Cool.
  - lacker 6y ago
    Releasing model weights is more powerful, but far less accessible than releasing an API. You can easily waste an hour trying to get a model from a research paper running, even when everything works smoothly, just having the wrong environment or dependencies. And that’s for people who are already savvy with ML. Releasing a Docker container is a lot closer to making things easily accessible but still not close to an API. If you look at the audience using GPT-3, a lot of it is software engineers who are experienced but don’t have AI-centric experience. In practice, making an API available is making the model available to many more people.
- thebeefytaco 6y ago
  As mentioned in the post, a pretrained model is available here: https://tfhub.dev/google/LaBSE/1 https://tfhub.dev/google/LaBSE/1
- accurrent 6y ago
  Learn to appreciate papers? This piece of research is far more interesting than openai's work which essentially involves spending more money on bigger compute and more data. The API is actually a massive turn off for me as I like to know how things work. While I understand your wanting to try things out via an API, I don't think that its the way things should be moving forward. Furthermore I think its very important that engineers using ML actually read the papers to understand what is going on otherwise you'll be finding idiots who apply facial recognition models to predict criminals running around in the wild saying that ML is some black magic when all its doing is reflecting social bias.
tasogare 6y ago
Typical ML-based paper where everything is about machine learning and nothing is about languages... The biggest issue is nothing is said about how the languages were selected. That’s the basis when dealing with multilingual languages as the features they share or not is important and will impact the results. In this respect the last figure is the more interesting one: it mentions ido and interlingua, two constructed languages. Why does they get included since NLP is supposed to deal with natural languages? It not a bad choice per se, but the selection criteria needs to be explain more than just picking languages to get a large number on a research paper. There is also no reflection on why some languages got better results than others. Again looking at the last figure in the top three best performing languages with no data two are Sinitic languages (Cantonese and Wu) which likely happens since their closeness to standard Chinese for which there is huge ressources available for training. On the reverse Breton for which there is probably very few apparented language data in the initial models and training set besides Irish get very poor results, which tends to show the model actually don’t transfer very well if at all...
- gleenn 6y ago
  In one of the graphs, the x-axis has the language code for the languages they used I think: https://1.bp.blogspot.com/-eqH1ZsnTP2o/XzwlDjy3X5I/AAAAAAAAGZw/_qFDiNyDi7cMeIBtUa-qoTJ9_6ODKtLtwCLcBGAsYHQ/s1621/image1.png https://1.bp.blogspot.com/-eqH1ZsnTP2o/XzwlDjy3X5I/AAAAAAAAG...
- mkasu 6y ago
  NLP is not my main field but still relevant to my work because I often use models and resources from NLP as tools. I'm also personally interested in Linguistics and Languages so I follow related news, sometimes attend NLP conferences and follow people in those fields on Social Media. It is very concerning how few thought is usually put into linguistic or language characteristics when dealing with these topics. I also rarely see cultural considerations etc. Basically everything is considered as "machine learning will hopefully get this right if having enough data" which is unfortunate (ML is a great tool but the conferences are about language processing). Another big issue I noticed is that a majority of research only targets or evaluates English texts. In many cases the language is not even specified (although it is clear they use English from figures or examples). I even heard people complaining that work on non-English data is treated as too minor by many reviewers so stuff like that often just gets rejected. I think this is a really weird development for a field which centers around natural languages.
richdougherty 6y ago
Paper: https://arxiv.org/abs/2007.01852 https://arxiv.org/abs/2007.01852 "We adapt multilingual BERT to produce language-agnostic sentence embeddings for 109 languages. %The state-of-the-art for numerous monolingual and multilingual NLP tasks is masked language model (MLM) pretraining followed by task specific fine-tuning. While English sentence embeddings have been obtained by fine-tuning a pretrained BERT model, such models have not been applied to multilingual sentence embeddings. Our model combines masked language model (MLM) and translation language model (TLM) pretraining with a translation ranking task using bi-directional dual encoders. The resulting multilingual sentence embeddings improve average bi-text retrieval accuracy over 112 languages to 83.7%, well above the 65.5% achieved by the prior state-of-the-art on Tatoeba. Our sentence embeddings also establish new state-of-the-art results on BUCC and UN bi-text retrieval." (Found via https://tfhub.dev/google/LaBSE/1 https://tfhub.dev/google/LaBSE/1)

74 points by theafh 6y ago

After all the attention OpenAI got for making the GPT-3 API semi-publicly-available, I wonder if Google has considered making some of their research models available via API. It would be pretty neat to be able to try these things out, rather than just reading papers about them.

MiroF 6y ago
You've actually got it backwards. Releasing model weights has been common for a long time, it was OpenAI that regressed and refused to release its GPT models until it released GPT-3 hidden behind a semi-public API. The google BERT models are released pre-trained in full (as mentioned in the article), you can easily play around with them on your own. BERT isn't a forward LM like GPT, so it's a little less easy to play with for the uninitiated - although there are papers showing text generation with BERT.
- colordrops 6y ago
  It's really ironic that they are regressing and keeping their research behind a curtain when the entire pretense for their existence (and their name) is to open things up.
  - est31 6y ago
    It's easier to hire people for a (claimed) mission driven company than for business evil corp that builds killer machines for the us military.
  - taneq 6y ago
    It’s not like it’s a new across-the-board policy for them. They were just being cautious in this particular case because they were worried about social ramifications. Arguably some other research should have been made less readily available like this (think deepfakes for example).
- tmabraham 6y ago
  To be fair, IIRC OpenAI did release GPT-2-large after some community pressure, and it was somewhat doable for some people to actually train from scratch. GPT-3 is too large, so even if they released it, nobody apart from large companies like Google could do anything with it. If anything, they've made GPT-3 more open than it would have been if they just released the weights. At least that's my understanding. Feel free to correct me if I'm wrong.
  - MiroF 6y ago
    I'm not really up in arms about OpenAIs decision, but I just don't want people to frame this as "why can't more companies release models like OpenAI does" when in reality the opposite is pretty much true. GPT-2 wasn't really feasible for most actors to train from scratch, otherwise it would have been released by a third party. GPT-3 is technically feasible for a single actor to do inference with, although not really really.
    - tmabraham 6y ago
      I agree with your comment here but I think for the type and size of model, OpenAI has done the best they could with such large models. So I wouldn't say they've "regressed and refused to release its GPT models" they just had to take a different route with GPT models.
      - tmabraham 6y ago
        People who disagree and are downvoting, may I ask why?
  - Der_Einzige 6y ago
    They only released GPT-2 large after other language models came out that were even larger, like T5...
  - liuliu 6y ago
    I don't know. It has 175 billion parameters, thus, about 500GiB for floating point parameters. If these are bfloat16, it is mere 300GiB. We know CPU is about 50 times slower than GPU for transformer models, hence, we are looking at 4 to 8 minutes per inference (parameter loading on demand from SSD takes 200 seconds or so, and probably the bottleneck here). If the parameters can be loaded into memory (seems you have to be on a >= 8-channel machine with unbuffered RAM or with buffered RAM), it will be probably 1 to 2 minutes per inference. Still in the realm of doing inference in homelab territory (barely).
- chaoz_ 6y ago
  However, @lacker makes a valid point that most research papers lack interactive, well-designed API for people unfamiliar with the domain.
  - MiroF 6y ago
    by API we mean pretty UI? if not, huggingface/transformers is a pretty good API for the Google-brand transformers.
- lettergram 6y ago
  I firmly believe what OpenAI is actually doing is building a dataset.
- lopsidedBrain 6y ago
  For the uninitiated, WTH does semi-public mean? Did GPT-3 only provide an API but not the model?
  - IfOnlyYouKnew 6y ago
    They do only have an API and the (trained) model (parameters) are not released. But I believe the "semi-public" here refers to the fact that access to the API is limited to invitees (with the ability to add your name to a waitlist).
    - lopsidedBrain 6y ago
      Huh, for some reason I had just assumed that it must be open based on the buzz around it. Cool.
- lacker 6y ago
  Releasing model weights is more powerful, but far less accessible than releasing an API. You can easily waste an hour trying to get a model from a research paper running, even when everything works smoothly, just having the wrong environment or dependencies. And that’s for people who are already savvy with ML. Releasing a Docker container is a lot closer to making things easily accessible but still not close to an API. If you look at the audience using GPT-3, a lot of it is software engineers who are experienced but don’t have AI-centric experience. In practice, making an API available is making the model available to many more people.
thebeefytaco 6y ago
As mentioned in the post, a pretrained model is available here: https://tfhub.dev/google/LaBSE/1 https://tfhub.dev/google/LaBSE/1
accurrent 6y ago
Learn to appreciate papers? This piece of research is far more interesting than openai's work which essentially involves spending more money on bigger compute and more data. The API is actually a massive turn off for me as I like to know how things work. While I understand your wanting to try things out via an API, I don't think that its the way things should be moving forward. Furthermore I think its very important that engineers using ML actually read the papers to understand what is going on otherwise you'll be finding idiots who apply facial recognition models to predict criminals running around in the wild saying that ML is some black magic when all its doing is reflecting social bias.

tasogare 6y ago

Typical ML-based paper where everything is about machine learning and nothing is about languages... The biggest issue is nothing is said about how the languages were selected. That’s the basis when dealing with multilingual languages as the features they share or not is important and will impact the results. In this respect the last figure is the more interesting one: it mentions ido and interlingua, two constructed languages. Why does they get included since NLP is supposed to deal with natural languages? It not a bad choice per se, but the selection criteria needs to be explain more than just picking languages to get a large number on a research paper. There is also no reflection on why some languages got better results than others. Again looking at the last figure in the top three best performing languages with no data two are Sinitic languages (Cantonese and Wu) which likely happens since their closeness to standard Chinese for which there is huge ressources available for training. On the reverse Breton for which there is probably very few apparented language data in the initial models and training set besides Irish get very poor results, which tends to show the model actually don’t transfer very well if at all...

gleenn 6y ago
In one of the graphs, the x-axis has the language code for the languages they used I think: https://1.bp.blogspot.com/-eqH1ZsnTP2o/XzwlDjy3X5I/AAAAAAAAGZw/_qFDiNyDi7cMeIBtUa-qoTJ9_6ODKtLtwCLcBGAsYHQ/s1621/image1.png https://1.bp.blogspot.com/-eqH1ZsnTP2o/XzwlDjy3X5I/AAAAAAAAG...
mkasu 6y ago
NLP is not my main field but still relevant to my work because I often use models and resources from NLP as tools. I'm also personally interested in Linguistics and Languages so I follow related news, sometimes attend NLP conferences and follow people in those fields on Social Media. It is very concerning how few thought is usually put into linguistic or language characteristics when dealing with these topics. I also rarely see cultural considerations etc. Basically everything is considered as "machine learning will hopefully get this right if having enough data" which is unfortunate (ML is a great tool but the conferences are about language processing). Another big issue I noticed is that a majority of research only targets or evaluates English texts. In many cases the language is not even specified (although it is clear they use English from figures or examples). I even heard people complaining that work on non-English data is treated as too minor by many reviewers so stuff like that often just gets rejected. I think this is a really weird development for a field which centers around natural languages.

richdougherty 6y ago

Paper: https://arxiv.org/abs/2007.01852 https://arxiv.org/abs/2007.01852 "We adapt multilingual BERT to produce language-agnostic sentence embeddings for 109 languages. %The state-of-the-art for numerous monolingual and multilingual NLP tasks is masked language model (MLM) pretraining followed by task specific fine-tuning. While English sentence embeddings have been obtained by fine-tuning a pretrained BERT model, such models have not been applied to multilingual sentence embeddings. Our model combines masked language model (MLM) and translation language model (TLM) pretraining with a translation ranking task using bi-directional dual encoders. The resulting multilingual sentence embeddings improve average bi-text retrieval accuracy over 112 languages to 83.7%, well above the 65.5% achieved by the prior state-of-the-art on Tatoeba. Our sentence embeddings also establish new state-of-the-art results on BUCC and UN bi-text retrieval." (Found via https://tfhub.dev/google/LaBSE/1 https://tfhub.dev/google/LaBSE/1)