4 ms·

That is a fantastic result - nagging question - these work best on predictable things. How much of Bengali poetry is predictable?

rg111 5y ago
> these work best on predictable things Umm, not really. You are talking about single and multi-label classification tasks, maybe? Bengali poetry is just like poetry in any other rich languages like English, French, etc. What these models do, from a high level, is that they learn the distribution of the data. In this case, they learn the style of the poets. Writing poetry in specific styles has been done long before Transformer architectures came. See- - https://www.tensorflow.org/text/tutorials/text_generation https://www.tensorflow.org/text/tutorials/text_generation - https://machinelearningmastery.com/text-generation-lstm-recurrent-neural-networks-python-keras/ https://machinelearningmastery.com/text-generation-lstm-recu... The goal of poetry generation is to generate something that is unique, is in the poetic style, and is coherrent, grammaticallly correct, ideally indistinguishable in the eyes of a human.
- zebraflask 5y ago
  Ah, I see the distinction now, thanks. It's an interesting subject, I think, since poetry has so many forms it can take and you need the output to capture the idiosyncratic aesthetics and "inner world" of a piece of verse. I've actually tried using GPT-3 to generate poetry and the naive approach of just sending prompts and text snippets through the API had wildly varying results. Some pretty good, some that were basically word salad and nonsense. But I also run a poetry journal on the side, so maybe that skews my understanding of it!
  - rg111 5y ago
    Start with LSTM or a distilled version of a language model that is trained for generation task, e.g. GPT2. There are a lot of pretrained language models (preferably at HuggingFace Hub). Take one, and fine-tune it on a much smaller dataset. You can then pass prompts to let this model write poetry. I would also suggest learning the fundamentals of Deep Learning, RNNs, LSTM, Transformer, etc. Because even if you know all these, this kind of taks is not trivial. I would use HuggingFace + PyTorch for these kind of tasks. Besides Deep Learning knowledge, you will also need to know your way around tokenizers, controlling parameters, evaluation metrics, etc. Having some basic understanding of poetry is helpful, but to be able to truly apply your knowledge of running your poetry journal, you will really need to be on the proverbial edge. I hope you get there, and maybe get some papers out! Although I worked on a language model because none existed in my language, my expertise (and employment) lies in Computer Vision. So, take what I said with a grain of salt.
rg111 5y ago
I must tell you that your intuitions are not wrong. Many language models do predict. In this case, they either try to predict what the next word (or character, or sub-character in case of Chinese, Japanese, etc.- this is totally the decision of the DS) is , or what some "masked" word are. w_i becomes w_(i-1) in the sequence where w_i is the last word generated The ones that are trained to be able to predict the next word are the ones that are good generators.

rg111 5y ago

> these work best on predictable things Umm, not really. You are talking about single and multi-label classification tasks, maybe? Bengali poetry is just like poetry in any other rich languages like English, French, etc. What these models do, from a high level, is that they learn the distribution of the data. In this case, they learn the style of the poets. Writing poetry in specific styles has been done long before Transformer architectures came. See- - https://www.tensorflow.org/text/tutorials/text_generation https://www.tensorflow.org/text/tutorials/text_generation - https://machinelearningmastery.com/text-generation-lstm-recurrent-neural-networks-python-keras/ https://machinelearningmastery.com/text-generation-lstm-recu... The goal of poetry generation is to generate something that is unique, is in the poetic style, and is coherrent, grammaticallly correct, ideally indistinguishable in the eyes of a human.

zebraflask 5y ago
Ah, I see the distinction now, thanks. It's an interesting subject, I think, since poetry has so many forms it can take and you need the output to capture the idiosyncratic aesthetics and "inner world" of a piece of verse. I've actually tried using GPT-3 to generate poetry and the naive approach of just sending prompts and text snippets through the API had wildly varying results. Some pretty good, some that were basically word salad and nonsense. But I also run a poetry journal on the side, so maybe that skews my understanding of it!
- rg111 5y ago
  Start with LSTM or a distilled version of a language model that is trained for generation task, e.g. GPT2. There are a lot of pretrained language models (preferably at HuggingFace Hub). Take one, and fine-tune it on a much smaller dataset. You can then pass prompts to let this model write poetry. I would also suggest learning the fundamentals of Deep Learning, RNNs, LSTM, Transformer, etc. Because even if you know all these, this kind of taks is not trivial. I would use HuggingFace + PyTorch for these kind of tasks. Besides Deep Learning knowledge, you will also need to know your way around tokenizers, controlling parameters, evaluation metrics, etc. Having some basic understanding of poetry is helpful, but to be able to truly apply your knowledge of running your poetry journal, you will really need to be on the proverbial edge. I hope you get there, and maybe get some papers out! Although I worked on a language model because none existed in my language, my expertise (and employment) lies in Computer Vision. So, take what I said with a grain of salt.

rg111 5y ago

I must tell you that your intuitions are not wrong. Many language models do predict. In this case, they either try to predict what the next word (or character, or sub-character in case of Chinese, Japanese, etc.- this is totally the decision of the DS) is , or what some "masked" word are. w_i becomes w_(i-1) in the sequence where w_i is the last word generated The ones that are trained to be able to predict the next word are the ones that are good generators.