4 ms·
We were planning on writing up a blog post to go over what our backend looks like. But essentially we have written a crawler to discover audio on the internet
by mathatoms 8y ago
We were planning on writing up a blog post to go over what our backend looks like. But essentially we have written a crawler to discover audio on the internet and a distributed processing framework to download, extract metadata, and transcribe the audio.
We've iterated through a few storage solutions and have settled on using GlusterFS+zfs running on Storinators. So far we have about 350TB of data indexed in our collection.
- dandancanfly 8y agoThat's pretty neat. After you download the audio and process it, do you delete the data, or store it for safe keeping? 350TB is a healthy chunk of data.
- mathatoms 8y agoWe have enough storage to hold on to the data. We keep the data around so we can retranscribe files as we update our language models.
- chatmasta 8y agoWow! Sounds awesome, I would love to read a blog post on that. Are you co-locating the hardware? What is bandwidth pricing like?
- mathatoms 8y agoThanks. Our blog is located at https://blog.bitplatter.com https://blog.bitplatter.com. We are co-locating some of our infrastructure. The backend that does the data processing is running in a rack on our own hardware. The user facing portions are hosted in GCE.