5 ms·

There are many, many self-hosting analytics tools, from your own big data pipes to tools like the aforementioned piwik, as well as Open Web Analytics. I like

by mwexler 10y ago

There are many, many self-hosting analytics tools, from your own big data pipes to tools like the aforementioned piwik, as well as Open Web Analytics. I like Snowplow (http://snowplowanalytics.com/ http://snowplowanalytics.com/), but it's currently hard-coded to AWS. Hosting your own analytics data can be great, but there are lots of ways to get better accuracy and control over your data without having to host everything. Still, if you can, it's great.

yummyfajitas 10y ago
Snowplow is only loosely hard coded to AWS. I'm using it and breaking it free is only a few hundred lines of code. For example, rerouting Snowplow's Kinesis collector into Kafka is 114 lines, and that includes logging, metrics, etc - I basically just had to extend the AbstractSink object in their scala collector. Reading from Kafka is another couple of hundred lines, similarly writing to files.
- alexatkeplar 10y ago
  Thanks for sharing yummyfajitas - expect official Kafka support for Snowplow a little later this year [1] [2]; it's been long awaited! (Snowplow co-founder) [1] https://github.com/snowplow/snowplow/milestones/Kafka%20%231 https://github.com/snowplow/snowplow/milestones/Kafka%20%231 [2] https://github.com/snowplow/snowplow/milestones/Kafka%20%232 https://github.com/snowplow/snowplow/milestones/Kafka%20%232
  - yummyfajitas 10y ago
    Nice. If you build some sort of native maxmind or other geotargeting into the scala collector, that would also be cool. (Not that it was difficult to roll my own - so far snowplow is perfect for my needs - but obviously I'd rather use an official one.)
    - alexatkeplar 10y ago
      We do all enrichments like MaxMind, weather, arbitrary JavaScript etc downstream of collection, in our enrichment phase - the list of configurable enrichments is here: https://github.com/snowplow/snowplow/wiki/Configurable-enrichments https://github.com/snowplow/snowplow/wiki/Configurable-enric...
- shermozle 10y ago
  I hope you're going to push your ~200 back to the project! Would love to see it free of the AWS dependency.
zoomzoom 10y ago
Also one thing to add here is that the client libraries are a lot of the work and you can use the snowplow js/ios/python etc.. no matter what server-side setup you use. I like to think of snowplow as pushing the open-source analytics standard and then hopefully an ecosystem of server-side products grows around that, led by their own product. We're doing something as dumb as using logs from cloud storage and parsing those with a ~100 line python script into a DB. S3/GCS deal with the collector uptime and as long as you aren't time-sensitive it is a great solution. The biggest issue with self-hosted analytics is visualizing/sharing the results with non-tech parts of the team. Piwik has some advantages here because it's closer to Google Analytics or Mixpanel than to a DB of event rows...

by mwexler 10y ago

yummyfajitas 10y ago

Snowplow is only loosely hard coded to AWS. I'm using it and breaking it free is only a few hundred lines of code. For example, rerouting Snowplow's Kinesis collector into Kafka is 114 lines, and that includes logging, metrics, etc - I basically just had to extend the AbstractSink object in their scala collector. Reading from Kafka is another couple of hundred lines, similarly writing to files.

alexatkeplar 10y ago
Thanks for sharing yummyfajitas - expect official Kafka support for Snowplow a little later this year [1] [2]; it's been long awaited! (Snowplow co-founder) [1] https://github.com/snowplow/snowplow/milestones/Kafka%20%231 https://github.com/snowplow/snowplow/milestones/Kafka%20%231 [2] https://github.com/snowplow/snowplow/milestones/Kafka%20%232 https://github.com/snowplow/snowplow/milestones/Kafka%20%232
- yummyfajitas 10y ago
  Nice. If you build some sort of native maxmind or other geotargeting into the scala collector, that would also be cool. (Not that it was difficult to roll my own - so far snowplow is perfect for my needs - but obviously I'd rather use an official one.)
  - alexatkeplar 10y ago
    We do all enrichments like MaxMind, weather, arbitrary JavaScript etc downstream of collection, in our enrichment phase - the list of configurable enrichments is here: https://github.com/snowplow/snowplow/wiki/Configurable-enrichments https://github.com/snowplow/snowplow/wiki/Configurable-enric...
shermozle 10y ago
I hope you're going to push your ~200 back to the project! Would love to see it free of the AWS dependency.

zoomzoom 10y ago

Also one thing to add here is that the client libraries are a lot of the work and you can use the snowplow js/ios/python etc.. no matter what server-side setup you use. I like to think of snowplow as pushing the open-source analytics standard and then hopefully an ecosystem of server-side products grows around that, led by their own product. We're doing something as dumb as using logs from cloud storage and parsing those with a ~100 line python script into a DB. S3/GCS deal with the collector uptime and as long as you aren't time-sensitive it is a great solution. The biggest issue with self-hosted analytics is visualizing/sharing the results with non-tech parts of the team. Piwik has some advantages here because it's closer to Google Analytics or Mixpanel than to a DB of event rows...