5 ms·

We use Elastic MapReduce at Snowplow to validate and enrich raw user events (collected in S3 from web, Lua, Arduino etc clients) into "full fat" schema'ed Snowp

We use Elastic MapReduce at Snowplow to validate and enrich raw user events (collected in S3 from web, Lua, Arduino etc clients) into "full fat" schema'ed Snowplow events containing geo-IP information, referer attribution etc. We then load those events from S3 into Redshift and Postgres. So we are solving the problem of processing raw user behavioural data at scale using MapReduce. All of our MapReduce code is written in Scalding, which is a Scala DSL on top of Cascading, which is an ETL/query framework for Hadoop. You can check out our MapReduce code here: https://github.com/snowplow/snowplow/tree/master/3-enrich/hadoop-etl https://github.com/snowplow/snowplow/tree/master/3-enrich/ha...

sandGorgon 13y ago
thanks for this. snowplow has been an amazing source of learning. I'm quite interested in the etl process than in the actual mapreduce. have you seen your etl used to pull data from Twitter or Facebook. I am wondering what is the state of art there considering throttling, etc.
- alexatkeplar 13y ago
  Hi sandGorgon! Thanks for the encouraging words. We haven't yet seen people use the existing Scalding ETL to pull data from Twitter or Facebook. As you suggest, there are some considerations around using Hadoop to access web APIs without getting throttled/banned. Here's a couple of links which might be helpful: - http://stackoverflow.com/questions/6206105/running-web-fetches-from-within-a-hadoop-cluster http://stackoverflow.com/questions/6206105/running-web-fetch... - http://petewarden.com/2011/05/02/using-hadoop-with-external-api-calls/ http://petewarden.com/2011/05/02/using-hadoop-with-external-... I think a gatekeeper service could make sense; or alternatively you could write something which runs prior to your MapReduce job and e.g. just loads your results into HDFS/HBase, for the MapReduce to then lookup into. Akka or maybe Storm could be choices here. We have done a small prototype project to pull data out of Twitter & Facebook - that was only a Python/RDS pilot, but it gave us some ideas for how a proper social feed into Snowplow could work.
  - sandGorgon 13y ago
    is your python code part of the snowplow repository/gist? it would be very interesting to take a look at it.

sandGorgon 13y ago

thanks for this. snowplow has been an amazing source of learning. I'm quite interested in the etl process than in the actual mapreduce. have you seen your etl used to pull data from Twitter or Facebook. I am wondering what is the state of art there considering throttling, etc.

alexatkeplar 13y ago
Hi sandGorgon! Thanks for the encouraging words. We haven't yet seen people use the existing Scalding ETL to pull data from Twitter or Facebook. As you suggest, there are some considerations around using Hadoop to access web APIs without getting throttled/banned. Here's a couple of links which might be helpful: - http://stackoverflow.com/questions/6206105/running-web-fetches-from-within-a-hadoop-cluster http://stackoverflow.com/questions/6206105/running-web-fetch... - http://petewarden.com/2011/05/02/using-hadoop-with-external-api-calls/ http://petewarden.com/2011/05/02/using-hadoop-with-external-... I think a gatekeeper service could make sense; or alternatively you could write something which runs prior to your MapReduce job and e.g. just loads your results into HDFS/HBase, for the MapReduce to then lookup into. Akka or maybe Storm could be choices here. We have done a small prototype project to pull data out of Twitter & Facebook - that was only a Python/RDS pilot, but it gave us some ideas for how a proper social feed into Snowplow could work.
- sandGorgon 13y ago
  is your python code part of the snowplow repository/gist? it would be very interesting to take a look at it.