3 ms·
I built an ETL that uses a Go stream for the transform. Once the data has been processed it's written to disk on our analytics servers. Since it can keep up w
by jchavannes 12y ago
I built an ETL that uses a Go stream for the transform. Once the data has been processed it's written to disk on our analytics servers. Since it can keep up with the stream, memory footprint is very small (which also means we never have to hit disk on ETL servers).
Go outperformed our previous solution by over 60x, making our new bottleneck disk I/O instead of CPU.
- taschenbillard 12y agowhat language was your previous solution in?
- deleted 12y ago[deleted]
- jchavannes 12y agoPHP. It was what I used initially so I could have access to our internal libs since our main application is in PHP. After crashing the ETL server a couple times though, I decided to try moving the heavy lifting to Go. When we benchmarked them, what would take PHP 30 minutes, Go could do in less than 30 seconds. Only the transform is written in Go, the rest of the ETL is still in PHP.
- lipoicacid 12y agohttp://jaxbot.me/articles/benchmarks_nodejs_vs_go_vs_php_3_14_2013 http://jaxbot.me/articles/benchmarks_nodejs_vs_go_vs_php_3_1... This bubble sort benchmark between PHP and Go supports your claims.