5 ms·
"Also, our entire logging and analysis infrastructure is being migrated to Go." This intrigues me. I've been moving a lot of ETL into Go (as it's pretty well
by ironchef 12y ago
"Also, our entire logging and analysis infrastructure is being migrated to Go." This intrigues me. I've been moving a lot of ETL into Go (as it's pretty well suited to this), but I still end up doing a lot of the analytics in Spark (for bigger work) or Pandas or R (for smaller bits of data). What are you guys planning on doing for the analytics? Or is it more of straight up time series work (so you can use influx + whatever). Just curious and keep up the good work :)
- jgrahamc 12y agoSince I'm not actually involved in that project I'm going to defer answering until we write a detailed blog post. Also, if you are interested in a job...
- voidlogic 12y ago"Also, our entire logging and analysis infrastructure is being migrated to Go." This intrigues me.... Go is great for this kind of stuff. Spark is great when you have a lot of custom queries, but if you have a few fixed queries writing a simple solution in Go have have huge gains. I worked to implement a web service + simple map reduce (with network transparency) + a time series DB (storage engine too, NOT using levelDB like influx) all in Go and the result was it is many times faster than an order of magnitude larger hadoop cluster, BigQuery, etc for the limited operations it supports.
- jobquestion123 12y agoDid you use a framework to run mapreduce with go or did you roll your own?
- voidlogic 12y agoWe wrote our own, Go 1.0 had just come out.