3 ms·

I posted this spreadsheet in another thread to give an idea of the cost of storing all of the nation's calls. I figured an annual cost of about $11 million. h

I posted this spreadsheet in another thread to give an idea of the cost of storing all of the nation's calls. I figured an annual cost of about $11 million. https://docs.google.com/spreadsheet/ccc?key=0AqWtA_3af-R0dE5hVGdZNUxkZlBYck83TkQ2NlJxQVE&usp=sharing#gid=0 https://docs.google.com/spreadsheet/ccc?key=0AqWtA_3af-R0dE5... (the spreadsheet allows editing, so you all can play with the numbers).

cklaus 13y ago
What does it cost to transcribe All VOICE calls ->TEXT for faster analysis, apply DEEP learning to it, and apply a graph of Everyone bigger than Facebook to these patterns?
- noblethrasher 13y ago
  I was just talking about the cost of storage, not acquisition or analysis.
- abecedarius 13y ago
  Just going by energy need I back-of-the-envelope estimated the Bluffdale center could run voice recognition on all U.S. voice calls, yes. I'd guess it's not the best use of that much power, currently, but it'll only get easier.
- PeterisP 13y ago
  The voice->text is inaccurate enough (especially because the many different languages that can be used) so you have to keep the original recording and not just the text; but yes, you can easily do things like "find all conversations where the word 'nakamoto' is heard" with mass scale voice/speech analysis.
intslack 13y ago
Brewster Kahle of archive.org ran the numbers last year as well: https://blog.archive.org/2013/06/15/cost-to-store-all-us-phonecalls-made-in-a-year-in-cloud-storage-so-it-could-be-datamined/ https://blog.archive.org/2013/06/15/cost-to-store-all-us-pho... https://docs.google.com/spreadsheet/ccc?key=0AuqlWHQKlooOdGJrSzhBVnh0WGlzWHpCZFNVcURkX0E https://docs.google.com/spreadsheet/ccc?key=0AuqlWHQKlooOdGJ...
JonSkeptic 13y ago
That's remarkably cheap.
spikels 13y ago
I'm pretty sure 96,000 bits/s vastly overstates the required storage. Common VoIP codecs uses as little as 5,300 bits/s[1] and military communication systems typically only require 2,400 bits/s and can go as low as 600 bits/sec[2]. Note: this is before all the transport overhead. And this is for live communications. Recorded voice communication can be compressed further. For example the popular G.729 VoIP codec compresses down to something like 1,000 bits/sec in archives from it's normal 8,000 bits/s required during a live call[3]. Although lower bandwidth codecs probably don't compress as much. Please correct me if I am wrong - not an expert. [1] http://www.cisco.com/c/en/us/support/docs/voice/voice-quality/7934-bwidth-consume.html#topic1 http://www.cisco.com/c/en/us/support/docs/voice/voice-qualit... [2] http://en.wikipedia.org/wiki/FNBDT#Voice_compression_using_Voice_Coders_.28vocoders.29 http://en.wikipedia.org/wiki/FNBDT#Voice_compression_using_V... [3] http://call-recording.tmcnet.com/topics/call-recording/articles/65721-voice-data-compression-recording-solutions-blessing-a-curse.htm http://call-recording.tmcnet.com/topics/call-recording/artic...
- noblethrasher 13y ago
  Agreed. I didn't have any reliable numbers at the time so I overestimated in favor of making it more expensive.

cklaus 13y ago

What does it cost to transcribe All VOICE calls ->TEXT for faster analysis, apply DEEP learning to it, and apply a graph of Everyone bigger than Facebook to these patterns?

noblethrasher 13y ago
I was just talking about the cost of storage, not acquisition or analysis.
abecedarius 13y ago
Just going by energy need I back-of-the-envelope estimated the Bluffdale center could run voice recognition on all U.S. voice calls, yes. I'd guess it's not the best use of that much power, currently, but it'll only get easier.
PeterisP 13y ago
The voice->text is inaccurate enough (especially because the many different languages that can be used) so you have to keep the original recording and not just the text; but yes, you can easily do things like "find all conversations where the word 'nakamoto' is heard" with mass scale voice/speech analysis.

intslack 13y ago

Brewster Kahle of archive.org ran the numbers last year as well: https://blog.archive.org/2013/06/15/cost-to-store-all-us-phonecalls-made-in-a-year-in-cloud-storage-so-it-could-be-datamined/ https://blog.archive.org/2013/06/15/cost-to-store-all-us-pho... https://docs.google.com/spreadsheet/ccc?key=0AuqlWHQKlooOdGJrSzhBVnh0WGlzWHpCZFNVcURkX0E https://docs.google.com/spreadsheet/ccc?key=0AuqlWHQKlooOdGJ...

JonSkeptic 13y ago

That's remarkably cheap.

spikels 13y ago

I'm pretty sure 96,000 bits/s vastly overstates the required storage. Common VoIP codecs uses as little as 5,300 bits/s[1] and military communication systems typically only require 2,400 bits/s and can go as low as 600 bits/sec[2]. Note: this is before all the transport overhead. And this is for live communications. Recorded voice communication can be compressed further. For example the popular G.729 VoIP codec compresses down to something like 1,000 bits/sec in archives from it's normal 8,000 bits/s required during a live call[3]. Although lower bandwidth codecs probably don't compress as much. Please correct me if I am wrong - not an expert. [1] http://www.cisco.com/c/en/us/support/docs/voice/voice-quality/7934-bwidth-consume.html#topic1 http://www.cisco.com/c/en/us/support/docs/voice/voice-qualit... [2] http://en.wikipedia.org/wiki/FNBDT#Voice_compression_using_Voice_Coders_.28vocoders.29 http://en.wikipedia.org/wiki/FNBDT#Voice_compression_using_V... [3] http://call-recording.tmcnet.com/topics/call-recording/articles/65721-voice-data-compression-recording-solutions-blessing-a-curse.htm http://call-recording.tmcnet.com/topics/call-recording/artic...

noblethrasher 13y ago
Agreed. I didn't have any reliable numbers at the time so I overestimated in favor of making it more expensive.