5 ms·

The future is not as dark as it seems because of the rat race of megacorps. You can use reduced versions of language models with extremely good results. I was

by rg111 5y ago

The future is not as dark as it seems because of the rat race of megacorps. You can use reduced versions of language models with extremely good results. I was involved in training the first-ever GPT2 for Bengali language, but with 117 million parameters. It took a month's effort (training + writing code + setup) and about $6k in TPU cost, but Google Cloud covered it. Anyway, it is surprisingly good. We fine-tuned the model for several downstream tasks and we were shocked when we saw the quality of generated text. I fine-tuned this model to write Bengali poems with a dataset of just about 2k poems and ran the training for 20 minutes in GPU instance of Colab Pro. I was really blown away by the quality. The main training was done in JAX, and it is much faster and seamless than PyTorch XLA, much better than TensorFlow in every way. So, my point is, although everyone is talking hundreds of billions of parameters and millions in training cost, you can still derive practical value from language models, and that too, at a low cost.

cmrajan 5y ago
Good to know. We're trying to attempt something similar[1] but for Tamil. I'm also surprised how well the OSS language model & library AI4Bharat [2] performs for NLP tasks against SoTA systems. Is there a way to contact you? [1] https://vpt.ai/posts/about-us/ https://vpt.ai/posts/about-us/ [2] https://ai4bharat.org/projects/ https://ai4bharat.org/projects/
- rg111 5y ago
  Among a master's degree, a consultancy gig, personal research and study, and finding unis abroad- I am living a hectic life. I don't see how I can be of help. But I can talk. Leave me something through which I can reach you. And I will reach you within a week.
amelius 5y ago
> The future is not as dark as it seems because of the rat race of megacorps. Just wait until NVidia comes with a "Neural AppStore" and corresponding restrictions. Then wait until the other GPU manufacturers follow suit.
- rg111 5y ago
  Much of the work done is fully open source and are liberally licensed. DeepMind and OpenAI have a bad rep in this regard. But a lot is available for free (as in beer and speech). And most of the research papers are released in arXiv. It's very refreshing. The bottleneck is not the knowledge or code, but the compute. People are fighting this in innovative ways. I have been an inactive part of Neuropark that first demoed collaborative training. A bunch of folks (some of them close to laypeople) ran their free Colab instances and trained a huge model. You can even utilize a swarm of GT1030s or something like that. Also, if you have shown signs of success, you are very likely to have people willing to sponsor your compute needs, case in point- Eluether AI. The situation is far from ideal with this megacorps rat race [0], and NLP research being more and more inaccessible, but it is not completely dark. [0]: I, along with many respected figures tend to think that this scaling up stuff approach is not even useful. We can write good prose with GPT-3 nowadays, that are, for all intents and purposes, indistinguishable from text written by humans. But we are far, far away from true understanding. These models don't really understand anything and are not even "AI", so to speak. The Transformer architecture, the backbone of all these approaches- is too brute-force-y for my taste to be considered something that can mimic or, further- be intelligent.
zebraflask 5y ago
That is a fantastic result - nagging question - these work best on predictable things. How much of Bengali poetry is predictable?
- rg111 5y ago
  > these work best on predictable things Umm, not really. You are talking about single and multi-label classification tasks, maybe? Bengali poetry is just like poetry in any other rich languages like English, French, etc. What these models do, from a high level, is that they learn the distribution of the data. In this case, they learn the style of the poets. Writing poetry in specific styles has been done long before Transformer architectures came. See- - https://www.tensorflow.org/text/tutorials/text_generation https://www.tensorflow.org/text/tutorials/text_generation - https://machinelearningmastery.com/text-generation-lstm-recurrent-neural-networks-python-keras/ https://machinelearningmastery.com/text-generation-lstm-recu... The goal of poetry generation is to generate something that is unique, is in the poetic style, and is coherrent, grammaticallly correct, ideally indistinguishable in the eyes of a human.
  - zebraflask 5y ago
    Ah, I see the distinction now, thanks. It's an interesting subject, I think, since poetry has so many forms it can take and you need the output to capture the idiosyncratic aesthetics and "inner world" of a piece of verse. I've actually tried using GPT-3 to generate poetry and the naive approach of just sending prompts and text snippets through the API had wildly varying results. Some pretty good, some that were basically word salad and nonsense. But I also run a poetry journal on the side, so maybe that skews my understanding of it!
    - rg111 5y ago
      Start with LSTM or a distilled version of a language model that is trained for generation task, e.g. GPT2. There are a lot of pretrained language models (preferably at HuggingFace Hub). Take one, and fine-tune it on a much smaller dataset. You can then pass prompts to let this model write poetry. I would also suggest learning the fundamentals of Deep Learning, RNNs, LSTM, Transformer, etc. Because even if you know all these, this kind of taks is not trivial. I would use HuggingFace + PyTorch for these kind of tasks. Besides Deep Learning knowledge, you will also need to know your way around tokenizers, controlling parameters, evaluation metrics, etc. Having some basic understanding of poetry is helpful, but to be able to truly apply your knowledge of running your poetry journal, you will really need to be on the proverbial edge. I hope you get there, and maybe get some papers out! Although I worked on a language model because none existed in my language, my expertise (and employment) lies in Computer Vision. So, take what I said with a grain of salt.
ShamelessC 5y ago
> about $6k in TPU cost, but Google Cloud covered it. I'm glad this all worked out for you. This is unrelated, but I just want to say that I hate how many people Google managed to convert to TPU with their research program and that their managed TPU/GPU offerings are absolutely horrible and infuriating to work with unless you somehow get on their radar.
- rg111 5y ago
  I have never used TRC, but I heard from many that if you reach out to them, they are really helpful. I have not converted to TPU, because it is literally offered by only one company. It will be the height of "vendor lock-in". But, I must say that JAX on TPU is faster than anything and everything that I have ever seen.
totoglazer 5y ago
What library did you use for JAX on TPU? Also curious how much data you had?
- rg111 5y ago
  We used Flax, of course. We thought Haiku was cool, but none of us knew it, and learning resources were scarce. But we are very happy with Flax. We used the Bengali subset of the mC4 dataset [0] for training the GPT-2 model. [0]: https://huggingface.co/datasets/mc4 https://huggingface.co/datasets/mc4

by rg111 5y ago

cmrajan 5y ago

Good to know. We're trying to attempt something similar[1] but for Tamil. I'm also surprised how well the OSS language model & library AI4Bharat [2] performs for NLP tasks against SoTA systems. Is there a way to contact you? [1] https://vpt.ai/posts/about-us/ https://vpt.ai/posts/about-us/ [2] https://ai4bharat.org/projects/ https://ai4bharat.org/projects/

rg111 5y ago
Among a master's degree, a consultancy gig, personal research and study, and finding unis abroad- I am living a hectic life. I don't see how I can be of help. But I can talk. Leave me something through which I can reach you. And I will reach you within a week.

amelius 5y ago

> The future is not as dark as it seems because of the rat race of megacorps. Just wait until NVidia comes with a "Neural AppStore" and corresponding restrictions. Then wait until the other GPU manufacturers follow suit.

rg111 5y ago
Much of the work done is fully open source and are liberally licensed. DeepMind and OpenAI have a bad rep in this regard. But a lot is available for free (as in beer and speech). And most of the research papers are released in arXiv. It's very refreshing. The bottleneck is not the knowledge or code, but the compute. People are fighting this in innovative ways. I have been an inactive part of Neuropark that first demoed collaborative training. A bunch of folks (some of them close to laypeople) ran their free Colab instances and trained a huge model. You can even utilize a swarm of GT1030s or something like that. Also, if you have shown signs of success, you are very likely to have people willing to sponsor your compute needs, case in point- Eluether AI. The situation is far from ideal with this megacorps rat race [0], and NLP research being more and more inaccessible, but it is not completely dark. [0]: I, along with many respected figures tend to think that this scaling up stuff approach is not even useful. We can write good prose with GPT-3 nowadays, that are, for all intents and purposes, indistinguishable from text written by humans. But we are far, far away from true understanding. These models don't really understand anything and are not even "AI", so to speak. The Transformer architecture, the backbone of all these approaches- is too brute-force-y for my taste to be considered something that can mimic or, further- be intelligent.

zebraflask 5y ago

That is a fantastic result - nagging question - these work best on predictable things. How much of Bengali poetry is predictable?

rg111 5y ago
> these work best on predictable things Umm, not really. You are talking about single and multi-label classification tasks, maybe? Bengali poetry is just like poetry in any other rich languages like English, French, etc. What these models do, from a high level, is that they learn the distribution of the data. In this case, they learn the style of the poets. Writing poetry in specific styles has been done long before Transformer architectures came. See- - https://www.tensorflow.org/text/tutorials/text_generation https://www.tensorflow.org/text/tutorials/text_generation - https://machinelearningmastery.com/text-generation-lstm-recurrent-neural-networks-python-keras/ https://machinelearningmastery.com/text-generation-lstm-recu... The goal of poetry generation is to generate something that is unique, is in the poetic style, and is coherrent, grammaticallly correct, ideally indistinguishable in the eyes of a human.
- zebraflask 5y ago
  Ah, I see the distinction now, thanks. It's an interesting subject, I think, since poetry has so many forms it can take and you need the output to capture the idiosyncratic aesthetics and "inner world" of a piece of verse. I've actually tried using GPT-3 to generate poetry and the naive approach of just sending prompts and text snippets through the API had wildly varying results. Some pretty good, some that were basically word salad and nonsense. But I also run a poetry journal on the side, so maybe that skews my understanding of it!
  - rg111 5y ago
    Start with LSTM or a distilled version of a language model that is trained for generation task, e.g. GPT2. There are a lot of pretrained language models (preferably at HuggingFace Hub). Take one, and fine-tune it on a much smaller dataset. You can then pass prompts to let this model write poetry. I would also suggest learning the fundamentals of Deep Learning, RNNs, LSTM, Transformer, etc. Because even if you know all these, this kind of taks is not trivial. I would use HuggingFace + PyTorch for these kind of tasks. Besides Deep Learning knowledge, you will also need to know your way around tokenizers, controlling parameters, evaluation metrics, etc. Having some basic understanding of poetry is helpful, but to be able to truly apply your knowledge of running your poetry journal, you will really need to be on the proverbial edge. I hope you get there, and maybe get some papers out! Although I worked on a language model because none existed in my language, my expertise (and employment) lies in Computer Vision. So, take what I said with a grain of salt.

ShamelessC 5y ago

> about $6k in TPU cost, but Google Cloud covered it. I'm glad this all worked out for you. This is unrelated, but I just want to say that I hate how many people Google managed to convert to TPU with their research program and that their managed TPU/GPU offerings are absolutely horrible and infuriating to work with unless you somehow get on their radar.

rg111 5y ago
I have never used TRC, but I heard from many that if you reach out to them, they are really helpful. I have not converted to TPU, because it is literally offered by only one company. It will be the height of "vendor lock-in". But, I must say that JAX on TPU is faster than anything and everything that I have ever seen.

totoglazer 5y ago

What library did you use for JAX on TPU? Also curious how much data you had?

rg111 5y ago
We used Flax, of course. We thought Haiku was cool, but none of us knew it, and learning resources were scarce. But we are very happy with Flax. We used the Bengali subset of the mC4 dataset [0] for training the GPT-2 model. [0]: https://huggingface.co/datasets/mc4 https://huggingface.co/datasets/mc4