4 ms·
How is a largely text-based app 3.47 GB? Is the dictionary/semantic DB just that large or is there other stuff going on?
by Jordan-117 1y ago
How is a largely text-based app 3.47 GB? Is the dictionary/semantic DB just that large or is there other stuff going on?
- michaeld123 1y agoI wish it were not so! Here's the breakdown: 1.5M headwords × ~2KB average per entry >= 3GB Each entry contains: 40 associations in the core graph Multiple senses (up to 8) × 17 associations each = up to 136 more Stems and morphological variants In-game clue definitions Longer definition entries with several types of related word lists. The only good news is it works offline.
- senkora 1y agoI do think that as a practical matter it might be better to store the word graph in the cloud and query it from the client. You could either store the word graph as a partitioned set of S3 buckets, or have a back-end that serves individual words and does rate-limiting. I guess that the back-end might be better to avoid surprise egress charges from anyone trying to download the entire dataset. I want to try out the game but I'm discouraged by the download size.
- kevin_thibedeau 1y agoThere would be significant data reduction if it was stored as a prefix trie flattened into an array.
- hx8 1y agoThe first really large text data I ever encountered was Google Ngram[0], the total size of which is about 3TB. I would have guessed it was closer to 3GB before I started downloading it. [0] https://storage.googleapis.com/books/ngrams/books/datasetsv3.html https://storage.googleapis.com/books/ngrams/books/datasetsv3...
- michaeld123 1y agoYes. I love Google Ngrams. We use the top google Ngrams in 2 ways. (a) we share it in the reference mode of our app, i.e. common words before or after; (b) we use longer N-grams, where possible, like a 4-gram, to choose literary examples that also show a common pattern.