5 ms·
1.2b/s try that its far moerchallenging
by ziont 8y ago
1.2b/s try that its far moerchallenging
- emddudley 8y agoWhat kind of database are you writing 1+ GB/s to...
- ziont 8y agoi mean 1.2 billion records per second. not month. now that was a tough problem back when I was sane.
- reallydude 8y agothats the kind of problem i work on (millions per sec at peak). everything is a queue and we stream from servers via kafka and into processors into redshift, more or less. Its free once it's moving around in AWS, but redshift and processors get expensive and im interested in reliable ad hoc solutions... but havent seen it.
- scapecast 8y agocan you elaborate on the process of copying data into Redshift? How often do you run COPY commands? On a continuous basis? Asking since Redshift as an OLAP isn't really optimized for heavy writes, so if you've figured out a way to do it, please bottle that magic!