4 ms·
Streams is something Go is very good at. I recently did project where I was download a large CSV zipped (50gigs) file and had to run analytics and collect metri
by knodi 12y ago
Streams is something Go is very good at. I recently did project where I was download a large CSV zipped (50gigs) file and had to run analytics and collect metrics on the data. With io.Buffer i was able to download, unzip, parse and process the data all in realtime with ease. Love it, very powerful.
P.S: without any of the data hitting disk. :)
- sologoub 12y agoWhat kind of memory footprint would something like this have? I'm assuming you didn't use 50GB of RAM and the processing was line-by-line, which should be quite small.
- jchavannes 12y agoI built an ETL that uses a Go stream for the transform. Once the data has been processed it's written to disk on our analytics servers. Since it can keep up with the stream, memory footprint is very small (which also means we never have to hit disk on ETL servers). Go outperformed our previous solution by over 60x, making our new bottleneck disk I/O instead of CPU.
- taschenbillard 12y agowhat language was your previous solution in?
- deleted 12y ago[deleted]
- jchavannes 12y agoPHP. It was what I used initially so I could have access to our internal libs since our main application is in PHP. After crashing the ETL server a couple times though, I decided to try moving the heavy lifting to Go. When we benchmarked them, what would take PHP 30 minutes, Go could do in less than 30 seconds. Only the transform is written in Go, the rest of the ETL is still in PHP.
- lipoicacid 12y agohttp://jaxbot.me/articles/benchmarks_nodejs_vs_go_vs_php_3_14_2013 http://jaxbot.me/articles/benchmarks_nodejs_vs_go_vs_php_3_1... This bubble sort benchmark between PHP and Go supports your claims.
- knodi 12y agoYes, the app only took up 150MBs max as the data was processed in chunks.