3 ms·
I'm interested in how large you think the database might grow? One of the biggest benefits I can see in a site like this is the possibility of discovering 'hidd
by roxgib 4y ago
I'm interested in how large you think the database might grow? One of the biggest benefits I can see in a site like this is the possibility of discovering 'hidden gems' that aren't well know and thus hard to find, but of course you'll only find what's in the database.
I'm working on different product that involves processing book content, and trying to plan for the computational requirements of millions of books and it's a lot, but I'm guessing running an language model over them would be several magnitudes more demanding.
Good luck, seems like a really promising use of AI!
- aditiyadav 4y agoI am planning to increase my catalogue to 1Mn. books as I get more users on the website. Running a language model is challenging as it took ~22 hours for training on 30k books and I expect it to take at least a week on M1 GPU. On Size, I expect 10Mn books to be taking ~16GB storage in my database. Thanks for your comments and good luck to your data pipeline.
- roxgib 4y agoThat's actually not too bad, I would have guessed a lot more than that. Good luck!