4 ms·
What's the best source/method to get the frequency distribution to build an Arithmetic coding model [0] for English (multilingual ideally) with emoticons? [0]
by espeed 10y ago
What's the best source/method to get the frequency distribution to build an Arithmetic coding model [0] for English (multilingual ideally) with emoticons?
[0] https://en.wikipedia.org/wiki/Arithmetic_coding#Defining_a_model https://en.wikipedia.org/wiki/Arithmetic_coding#Defining_a_m...
- web007 10y agoOnr of the main points of arithmetic coding is that you don't have to do a frequency count first like you would for Huffman. It's efficient enough that you can build your model piecewise as you go, and only sacrifice a few bits of coding inefficiency. You're not going to find many public datasets on SMS or other emoji-using entities regardless of language. Twitter or Instagram are your best bet, but building your model on those will only work best on those types of input.
- ooqr 10y agoLots of IRC servers host their logs online.
- espeed 10y agoJust discovered Matthew Rothenberg's Emojitracker project: * http://www.emojitracker.com/ http://www.emojitracker.com/ * http://github.com/mroth/emojitrack http://github.com/mroth/emojitrack * https://medium.com/@mroth/how-i-built-emojitracker-179cfd8238ac#.hhoymvyp1 https://medium.com/@mroth/how-i-built-emojitracker-179cfd823... * https://medium.com/@mroth/how-i-kept-building-emojitracker-c31378810136 https://medium.com/@mroth/how-i-kept-building-emojitracker-c...