23 ms·
Ask HN: Who is using a self hosted analytics system?
I'm so used to SASS based solutions these days like Mixpanel, GA, Kiss Metrics, etc. But with everything happening over the last few years in regards to privacy and security I've been looking into self hosted.
Is anyone else considering the same? I know this is standard for companies like Amazon and Facebook, but what about everyone else?
If you are using a self hosted setup, what is it? Custom built, open source, etc?
- ohgh1ieD 10y agoI'm actually on the same road right now, currently I'm testing Piwik. https://piwik.org https://piwik.org
- mfrye0 10y agoYeah I've been looking at Piwik. What do you think so far? The only aspect I didn't like was it was PHP. Not really my strong language - if I wanted to customize it.
- pesfandiar 10y agoWhat's your experience with it so far? I was playing around with it at a startup a few years ago. It seemed to have a strong community and features were popping up rather quickly. However, it didn't scale that well for us.
- ohgh1ieD 10y agoI'm very pleased so far, but as I said, it's too early for me to tell something. > However, it didn't scale that well for us. What kind of scaling problems did you encounter ?
- pesfandiar 10y agoIn fairness to Piwik, it was some of the custom metrics that required more processing power. We ended up doing delayed batch processing for them. Towards the end, IIRC, the calculation of unique visitors within a custom range was also slow.
- user5994461 10y agoLast I checked it only runs over a MySQL. I don't know how many visitors and page views you have, but that's a red flag for most established sites.
- __d 10y agoI've used https://piwik.org https://piwik.org successfully.
- mfrye0 10y agoThoughts using it so far?
- mfrye0 10y agoI've been talking to a friend about it and he mentioned using Elastic Search to build our own setup. Idk if it's the right use case though...
- mrgreenfur 10y agoIf you're serious about self-hosting analytics there is only one serious place to go: http://snowplowanalytics.com/ http://snowplowanalytics.com/ I don't use them, but they are building enterprise-grade self-hosting. Disclaimer: I am working on a project in the marketing analytics space. I don't use them and this isn't an endorsement, just a pointer to research more!
- mfrye0 10y agoThis looks awesome. Just what I was looking for. Seems like they have some decent companies using them too.
- mrgreenfur 10y agoAre you still working on thebigpicture?
- mfrye0 10y agoYeah. I'm actually asking the question as one of my friends mentioned we should offer a full self hosted option for enterprise. I don't know if we want to go that direction though. Plus I wasn't even sure who is actually self hosting these days. Kind of reminds me of the Silicon Valley show with the "box" in the data center dilemma.
- mrgreenfur 10y agoYeah, I hear you. Snow plow is the only commercial one I've heard of that allows self-hosted. It makes sense but you need a new billing model then (e.g. services). The bigger the data the more self-hosting will make sense for you, but the less for your customers.
- mfrye0 10y agoI'm not sure if I follow you there. What did you mean by, "The bigger the data the more self-hosting will make sense for you, but the less for your customers"?
- Nilef 10y agoWorking for a bank processing millions of calls a minute; using WebTrends
- mfrye0 10y agoI haven't heard of WebTrends before. I'm looking at their website now. Are you self-hosting it? I can't find info on that option.
- shakna 10y agoI don't yet have a product I've settled on, but I'm in the early stages of developing a website for a government body. With all the regulations and policies on data protection, using something not self-hosted is just not going to happen. (I believe that if there was a possibility that their users were affected by a 3rd party breach, the fine is around 1000x the project budget). If this means a little less information about the audience, that is perfectly acceptable.
- mfrye0 10y agoYeah that makes sense being a government org.
- zyzioziom 10y agoPiwik is pretty good, but like most open source, not so feature packed comparing to commercial tools. It's resource heavy, though. We had problems for big amounts of data which couldn't be loaded before timeouts. Caching helps, but it definitely needs stronger machine.
- tixocloud 10y agoTo give you some perspective, a startup I worked for began with a self-hosted web analytics option. In fact, most of the systems we had were all homegrown and while initially it was a great idea - the maintenance part of it took a lot of time away from optimizing on generating revenue. We spent a lot of time trying to figure out the structure of our self-hosted web analytics platform, how it tracks data, how it stores data, etc. The majority of our company were engineers but we still wasted a lot of time fiddling with the self-hosted analytics. There will likely be a subset of customers who would be interested in self-hosted but I'm willing to bet that they are more likely to be companies more engineering-oriented.
- mfrye0 10y agoGood feedback. Yeah I was thinking the same thing - that companies who are self hosting are more likely to be engineering-oriented. Otherwise like you said it's just too much of a pain to handle it yourself. May I ask, why did you self-host vs use an external service? To save money, for greater security and privacy?
- tixocloud 10y agoI believe we self-hosted because of privacy worries and the engineering culture (or myth that we can and should build everything). Ultimately though, the business demanded to switch to something more robust and we went with Omniture. I do believe that self-hosted is an option if it's easy to maintain and it also delivers on business features like reporting and data exploration.
- gesman 10y agoI built my own analytics App for Splunk to offer business insights for my wife's small business. Mostly how traffic correlates with purchases and where buyers are coming from. As a side effect same system detects malware and cyber attacks on other websites pretty well as well. https://splunkbase.splunk.com/app/2676/ https://splunkbase.splunk.com/app/2676/
- mfrye0 10y agoWhoa. That's badass. Do you pull any info regarding the IP addresses, or is it only the raw logs that you're going through?
- gesman 10y agoOnly raw logs. Splunk resolves IP to Country/Region/City (and geo coordinates if wanted to map these). Mostly playing with raw logs and then even RAW-er logs using Splunk Stream (thing that switches network interface in promiscuous mode and gives me all data for all protocols and any context I ever want). For example I can analyze anomalies in web hits and anomalies in web session to discover new, previously unknown traffic sources and patterns. It helped to discover 2 new classes of cyberattacks I didn't know were targeting my server.
- mfrye0 10y agoSounds really useful. I'll have to check that out.
- boyter 10y agoI have a custom built one. It's main feature is that it can email me the details I want on a scheduled basis to save me logging in. It tracks a few million records a month so not high scale and runs on a single $5 digital ocean instance. I did attempt to make it Sass for hosting resellers as an upsell but never made any progress. It's just running for myself these days.
- mfrye0 10y agoSounds pretty cool. What tech is it based on? Just a standard api and db? Also, can I ask how big your company is? A few million records is decent volume.
- boyter 10y agoNginx, Django/Python and MySQL. Nothing fancy. No company actually. Just side projects. The largest being searchcode.com Feel free to email me if you want further details on either. Details in my profile.
- mfrye0 10y agoSure thing. Thanks. I've actually been looking at Snowplow and that seems pretty badass so far. Might work for what I'm looking at.