9 ms·

Self-host analytics for better privacy and accuracy

193 points by FiloSottile 10y ago

d0ugie 10y ago
To mitigate security loss (Piwik is complex), run Piwik and serve its gif on another machine.
- ljoshua 10y ago
  Can you elaborate a bit more?
  - SwellJoe 10y ago
    I'd assume the prior poster is suggesting that running a third party tool on your primary web server(s) would increase the surface area for attacks by some (possibly not small) amount. e.g. if piwik is compromised, an attacker would have some sort of user access on your web server(s), which is generally a bad thing. I suppose some people also use the same database user for all web applications, which would potentially be disastrous. There are mitigations one can implement without going to that length (run piwik under a different user than any of your other web applications, using suexec, use a different database and user, etc.). At the least, putting Piwik in a container or VM makes sense, if any data on your web server(s) is critical or sensitive. I suspect for very large deployments this would go without saying. But, for users with only one web server, it might seem reasonable to drop it into the same virtual host and run it all as the same user (and it's probably safe enough to do so for many users, as long as they stay on top of updates). But, any web application you run adds surface area for attackers. Might as well isolate them as well as your skills and resources allow.
    - mattab 10y ago
      See also some of the official Piwik security tips: https://piwik.org/docs/how-to-secure-piwik/ https://piwik.org/docs/how-to-secure-piwik/
daedalus_j 10y ago
Not quite true though... Piwik, by default, includes EasyList, which has the following block rule: /piwik.$domain=~piwik.org,script This will block your self-hosted piwik.js file, unless you perform some redirection trickery.
- FiloSottile 10y ago
  The default Piwik Sandstorm install has "piwik" nowhere in the URL.
  - tirus 10y ago
    Yeah, unfortunately it shows up as https://ls4an735rucvfa6ps6bb.filippo.sandcats.io/embed.js https://ls4an735rucvfa6ps6bb.filippo.sandcats.io/embed.js - it's getting to the point where I need to start performing actual content inspection.
    - brettz 10y ago
      How come?
    - aorth 10y ago
      I run Piwik on my server log files. This is my right as a website owner / system admin. :)
- skrause 10y ago
  It's very easy to rename (or symlink) the piwik.js script to something else that is not blocked by any filter list. And unless you're a major site nobody will bother creating a special rule just for your self-hosted Piwik.
aluhut 10y ago
I would love to have Sandstorm on my RasPi but they don't support ARM. It sounds like a great home intranet.
- ocdtrekkie 10y ago
  Just notes on this for the curious: 1. Sandstorm doesn't support ARM currently because Sandstorm apps run native Linux binaries, and every app would have to be compiled for each architecture. 2. I honestly think you'd be running pretty crippled trying to do Sandstorm on a RasPi. It's a bit smaller scale than Sandstorm seems targeted for. Each open Sandstorm grain commonly uses 100 MB of RAM or more (on top of the RAM used by Linux and the Sandstorm server itself, of course), so with just a couple of Sandstorm grains running simultaneously, you can max out a RasPi pretty quickly.
  - MustardTiger 10y ago
    >Sandstorm doesn't support ARM currently because Sandstorm apps run native Linux binaries, and every app would have to be compiled for each architecture. That's true of any linux distro providing binary packages. They all support arm anyways, it is trivially simple to compile packages. Even small projects like openbsd compile tens of thousands of packages for a dozen arches.
    - kentonv 10y ago
      Yes but distros accomplish that by being highly opinionated on the build process you use to build packages whereas Sandstorm tries to be unopinionated on this point. Sandstorm will support ARM someday but it's going to require a large investment in tooling in order to be painless for developers.
      - MustardTiger 10y ago
        >Yes but distros accomplish that by being highly opinionated on the build process you use to build packages whereas Sandstorm tries to be unopinionated on this point. How's that exactly? I looked it over and can't see any difference at all.
Animats 10y ago
If you're serving the pages yourself (not, as this site does, through Cloudflare's caches), what does this tell you that isn't in the server logs?
- FiloSottile 10y ago
  Unless you are tracking events, outlinks, time spent on page or something similar, nothing as far as data goes. But obviously server logs lack statistics and aggregation. PIWIK actually supports using the server logs as its data source instead of the javascript tag.
- shermozle 10y ago
  Deduplicated reach (Unique Visitors). IP addresses change and are shared, so they aren't suitable for this.
viana007 10y ago
The biggest problem with Piwik is that is not scalable and the cost ($) with servers to store analytics data. I saw many cases that Piwik not supported the big volume of data.
- rhizome 10y ago
  Why isn't it scalable?
  - shermozle 10y ago
    MySQL
    - toomuchtodo 10y ago
      Put Rabbit or Kafka in front of MySQL then?
      - nacs 10y ago
        They have a Redis-based solution to sit in front of mySQL (seems to be a "Pro" feature though): https://piwik.org/faq/how-to/faq_19738/ https://piwik.org/faq/how-to/faq_19738/
        moehm 10y ago
        It's not a Pro feature as in you have to pay for it. It's developed by the devs at piwik.pro, but you can download it for free.
    - rhizome 10y ago
      I see. What is the upper bound of "big volume of data" possible under MySQL + Piwik?
      - shermozle 10y ago
        That's a difficult thing to answer. However the more important problem is loss of data whenever your DB isn't available due to downtime, upgrade etc. It depends how important data loss is for your user case. I'm a data completist but I'm in therapy for it ;) It's definitely worth playing with, and trivially easy to spin up. Other self-hosted options aren't anything like as simple to get up and running.
mwexler 10y ago
There are many, many self-hosting analytics tools, from your own big data pipes to tools like the aforementioned piwik, as well as Open Web Analytics. I like Snowplow (http://snowplowanalytics.com/ http://snowplowanalytics.com/), but it's currently hard-coded to AWS. Hosting your own analytics data can be great, but there are lots of ways to get better accuracy and control over your data without having to host everything. Still, if you can, it's great.
- yummyfajitas 10y ago
  Snowplow is only loosely hard coded to AWS. I'm using it and breaking it free is only a few hundred lines of code. For example, rerouting Snowplow's Kinesis collector into Kafka is 114 lines, and that includes logging, metrics, etc - I basically just had to extend the AbstractSink object in their scala collector. Reading from Kafka is another couple of hundred lines, similarly writing to files.
  - alexatkeplar 10y ago
    Thanks for sharing yummyfajitas - expect official Kafka support for Snowplow a little later this year [1] [2]; it's been long awaited! (Snowplow co-founder) [1] https://github.com/snowplow/snowplow/milestones/Kafka%20%231 https://github.com/snowplow/snowplow/milestones/Kafka%20%231 [2] https://github.com/snowplow/snowplow/milestones/Kafka%20%232 https://github.com/snowplow/snowplow/milestones/Kafka%20%232
    - yummyfajitas 10y ago
      Nice. If you build some sort of native maxmind or other geotargeting into the scala collector, that would also be cool. (Not that it was difficult to roll my own - so far snowplow is perfect for my needs - but obviously I'd rather use an official one.)
      - alexatkeplar 10y ago
        We do all enrichments like MaxMind, weather, arbitrary JavaScript etc downstream of collection, in our enrichment phase - the list of configurable enrichments is here: https://github.com/snowplow/snowplow/wiki/Configurable-enrichments https://github.com/snowplow/snowplow/wiki/Configurable-enric...
shermozle 10y ago
Piwik has the problem that it writes directly to MySQL as the activity happens. If your database is down, you lose data. If you have a spike of traffic above what your DB can handle in writes, you lose data. Snowplow doesn't have this problem.
- moehm 10y ago
  You can query your writes in Redis, so it won't be lost if your database goes down. https://piwik.org/faq/how-to/faq_19738/ https://piwik.org/faq/how-to/faq_19738/
  - shermozle 10y ago
    Cos your Redis will never go down? Or need to be upgraded? Patched?
    - educar 10y ago
      How does snowplow solve this problem? By writing to disk?
      - alexatkeplar 10y ago
        Snowplow is implemented as a unidirectional data pipeline: tracking -> collection -> enrichment -> storage Between each step there is typically some kind of persistent queue (mostly S3/Kinesis), and data won't be lost if a downstream component is not operational. Examples: * If your event collector is unavailable, raw events will be cached in the tracker in localStorage, SQLite or similar * If your Redshift database is read-only for maintenance, enriched events will be held back until Redshift is writeable again
        shermozle 10y ago
        There is of course the opportunity for a collector to be down, but the aim is to keep those components super simple and rely on really durable storage (Kinesis, S3) managed by someone else.
tirus 10y ago
For those of us that prefer not to be tracked by self-hosted Sandstorm apps, 2 new ublock rules (they're very rough, but I am no ublock wizard): ||sandcats.io/embed.js$script view.gif?page=$image Note that you would have to change the filters if the scripts and tracking pixel are renamed of course but this should catch a majority of the push button installs.
- dragonne 10y ago
  Why do you object to self-hosted analytics? I understand blocking centralized trackers (I do so myself), but self-hosted doesn't seem problematic in the same way GA being present on half the pages on the Internet is. It also strikes me as an unwinnable battle for all but the largest sites.
  - angry-hacker 10y ago
    Because OP is against all kind of tracking? And because he can...
    - Programmatic 10y ago
      I can't claim to speak for OP, but am also against most tracking. I would also tend to think that being against first party tracking would be an unwinnable battle. It also leaks less data than third-party tracking, since the third party can see your activity across multiple sites whereas first party can only see your activity on that site unless it's aggregated through a backend service (another poster mentioned the ability to upload server logs to GA). No matter what, they can see what you load from their site. Getting to first party hosting of more intrusive analytics (scroll location, etc) I think rather than disallowing certain scripts/URLs to run, you have to get back to behavioral-based blocking. Doing that in an environment that you allow any JS to execute seems tough since sandboxing something that can update the page based on location can "talk" to another part that can report back to the server. If you don't like intrusive first party analytics, just stop all JS.
- guelo 10y ago
  If you visit a web site the operators of that site will know that someone at your IP address visited the site, which pages you viewed, how long you lingered, which buttons you clicked, etc. If you want to prevent that your only option is to browse via things like proxies or Tor.
_jomo 10y ago
Another very simple alternative is goaccess [0] which is purely server-side. It gathers quite a lot of information by parsing the server log. It doesn't do all the JavaScript stuff to track every single click, but it gives you stats on how many users visit your site, which parts, when, which sites or which domains they're coming from and also how much bandwidth they use. It also shows which status codes are coming from which paths and a lot more. It supports various output formats such as HTML or an interactive htop-like terminal application. It's also being actively developed. I use it and find it very useful. 0: https://www.goaccess.io/ https://www.goaccess.io/
- herbst 10y ago
  GoAccess is really one of the greatest tools of these kind. I use it with GA on all my servers.
deleted 10y ago
[deleted]
davidw 10y ago
One potential problem: if you, say, test out self-hosted alongside GA, and get different numbers, people are going to question the value of the self-hosted thing.
- soared 10y ago
  They should question both, but maybe GA more. GA is an approximation and absolutely not completely correct.
  - davidw 10y ago
    Try selling that to a business guy or client or someone. Possible, but not easy, especially if there's ever a large discrepancy.
    - soared 10y ago
      Haha oh I know. Tell a client, "Yeah this is the data, it says this, BUT this is also how its wrong" and they always ignore the last part. Its data, that stuff can't be wrong!
__jal 10y ago
I've done my own analytics since before there were hosted services. Still use an old-school, proprietary tool called Sawmill[1]. One very nice aspect (out of many) about it is that it handles hundreds of other log formats out of the box, so it can report sensibly on switches, email, firewall logs... just about anything with very little effort. No connection, other than being a long-time happy customer. [1] http://sawmill.net/ http://sawmill.net/
- blacksmith_tb 10y ago
  I haven't used Sawmill in ages, but it wasn't bad, once upon a time. There's also GoAccess, which is a snazzy ncurses logfile analyzer: https://www.goaccess.io/ https://www.goaccess.io/
  - mattab 10y ago
    Piwik also handle dozens of Log Formats, check it out: http://piwik.org/log-analytics/ http://piwik.org/log-analytics/ Github project: https://github.com/piwik/piwik-log-analytics https://github.com/piwik/piwik-log-analytics
vizza 10y ago
A middle of the road approach is to use an analytics package like https://truestats.com https://truestats.com that allows you to download raw session info at any time.
pixelmonkey 10y ago
Piwik is a great project, but it tends not to work well for handling sites with millions of events per day. Your MySQL table starts to bust at the seams pretty quickly. For big sites, you'll want that event data in GiB's of plain raw logs that you can bulk load into tools like BigQuery or Redshift for analysis. My team has built/delivered a SaaS web content analytics platform for the past few years called Parse.ly. We instrument pageview events (like Google Analytics) automatically, and we also instrument time-on-page using heartbeat events. We collect 50 billion monthly page events for over 600 top-traffic sites, and display it all in a real-time dashboard. To emphasize that our customers own the data they send us, we recently launched a Raw Data Pipeline product: http://parse.ly/data-pipeline http://parse.ly/data-pipeline Basically, we host a secure S3 bucket and Kinesis stream for customers, and deliver their raw (enriched) event data there. From there, they typically load it into their own BigQuery or Redshift instance, or they analyze pieces of it directly with Python/R/Excel/etc. Our customers tell us this strikes the right balance among data ownership, query flexibility, and hassle-free infrastructure.
- educar 10y ago
  Is their any difference in data ownership between GA and parsely?
  - pixelmonkey 10y ago
    Data ownership in GA is a "gray area" that becomes less gray if you pay $150K/yr for "GA Premium". Google has mixed incentives in running its free analytics service. It gets web-wide analytics data, it uses data to help it sell more AdWords to customers, and it integrates GA with other services, like their display advertising products (DFP, etc.) From a practical standpoint, you don't "own" analytics data when a) you can't easily access it in raw form and b) the SaaS provider "leaks" your data to dilute its value to you. We address (a) and (b) directly through our products and public data privacy stance. See this blog post for our public view on analytics data privacy: http://blog.parsely.com/post/3394/analytics-privacy-without-compromise/ http://blog.parsely.com/post/3394/analytics-privacy-without-...
Ileca 10y ago
"Self-hosting analytics for better privacy and accuracy." Then why using google fonts which is listed by Disconnect.me (used by Firefox) as a tracking domain? Isn't that paradoxical?
- ocdtrekkie 10y ago
  I don't think* embedding Google Fonts allows Google much (any?) data collection beyond, presumably, that the font was requested, though it's definitely an unnecessary dependency. Given Google Fonts allows you to download all of the fonts, with license info provided, in a variety of formats, there's almost no good reason not to embed them directly in your own site. Right now, Sandstorm apps will currently work with Google Fonts, but the Sandstorm team intends to sandbox the client side better in the near-ish future, so I've already been nudging Sandstorm apps to make sure not to use such things when I see it. Note that while the author's blog uses Google Fonts, the Piwik Sandstorm package does not. (I checked.) *I don't know.
  - 0942v8653 10y ago
    Google Fonts does see the page you are on, in the `Referer` header. According to mitmproxy: host: fonts.googleapis.com Connection: keep-alive Proxy-Connection: keep-alive Accept: text/css,*/*;q=0.1 User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13E238 Safari/601.1 Accept-Language: en-us Referer: https://blog.filippo.io/self-host-analytics/ Accept-Encoding: gzip, deflate (I agree with the rest of your comment; the best thing to do is host the fonts yourself.)
    - ocdtrekkie 10y ago
      And it does appear Google just blankets this into it's general API terms of use that they can "use submitted data" in accordance with their general privacy policies. So yeah, I guess they can use it as part of their tracking. :/
  - ngrilly 10y ago
buremba 10y ago
Shameless plug: We're also working on an on-premise custom analytics platform that can be deployed either Heroku or AWS with Cloudformation. https://github.com/rakam-io/rakam https://github.com/rakam-io/rakam
manigandham 10y ago
Google Analytics has Measurement Protocol which allows for posting data server-side. All you need to do is proxy the data collection and then send it to them, taking advantage of all the scalability and features they have. https://developers.google.com/analytics/devguides/collection/protocol/v1/ https://developers.google.com/analytics/devguides/collection...
philliphaydon 10y ago
Off topic a bit. But with GA being blocked, all these blocks work by blocking URLs and domains at the time they are loaded, but not requests made after load of the requested resource. Couldn't you just proxy the URL to GA so it's not blocked?
- dest 10y ago
  By "requests made after load of the requested resource", you mean JS XHR requests for example? I'd guess they are also filtered by adblockers, aren't they?

193 points by FiloSottile 10y ago

d0ugie 10y ago

To mitigate security loss (Piwik is complex), run Piwik and serve its gif on another machine.

ljoshua 10y ago
Can you elaborate a bit more?
- SwellJoe 10y ago
  I'd assume the prior poster is suggesting that running a third party tool on your primary web server(s) would increase the surface area for attacks by some (possibly not small) amount. e.g. if piwik is compromised, an attacker would have some sort of user access on your web server(s), which is generally a bad thing. I suppose some people also use the same database user for all web applications, which would potentially be disastrous. There are mitigations one can implement without going to that length (run piwik under a different user than any of your other web applications, using suexec, use a different database and user, etc.). At the least, putting Piwik in a container or VM makes sense, if any data on your web server(s) is critical or sensitive. I suspect for very large deployments this would go without saying. But, for users with only one web server, it might seem reasonable to drop it into the same virtual host and run it all as the same user (and it's probably safe enough to do so for many users, as long as they stay on top of updates). But, any web application you run adds surface area for attackers. Might as well isolate them as well as your skills and resources allow.
  - mattab 10y ago
    See also some of the official Piwik security tips: https://piwik.org/docs/how-to-secure-piwik/ https://piwik.org/docs/how-to-secure-piwik/

daedalus_j 10y ago

Not quite true though... Piwik, by default, includes EasyList, which has the following block rule: /piwik.$domain=~piwik.org,script This will block your self-hosted piwik.js file, unless you perform some redirection trickery.

FiloSottile 10y ago
The default Piwik Sandstorm install has "piwik" nowhere in the URL.
- tirus 10y ago
  Yeah, unfortunately it shows up as https://ls4an735rucvfa6ps6bb.filippo.sandcats.io/embed.js https://ls4an735rucvfa6ps6bb.filippo.sandcats.io/embed.js - it's getting to the point where I need to start performing actual content inspection.
  - brettz 10y ago
    How come?
  - aorth 10y ago
    I run Piwik on my server log files. This is my right as a website owner / system admin. :)
skrause 10y ago
It's very easy to rename (or symlink) the piwik.js script to something else that is not blocked by any filter list. And unless you're a major site nobody will bother creating a special rule just for your self-hosted Piwik.

aluhut 10y ago

I would love to have Sandstorm on my RasPi but they don't support ARM. It sounds like a great home intranet.

ocdtrekkie 10y ago
Just notes on this for the curious: 1. Sandstorm doesn't support ARM currently because Sandstorm apps run native Linux binaries, and every app would have to be compiled for each architecture. 2. I honestly think you'd be running pretty crippled trying to do Sandstorm on a RasPi. It's a bit smaller scale than Sandstorm seems targeted for. Each open Sandstorm grain commonly uses 100 MB of RAM or more (on top of the RAM used by Linux and the Sandstorm server itself, of course), so with just a couple of Sandstorm grains running simultaneously, you can max out a RasPi pretty quickly.
- MustardTiger 10y ago
  >Sandstorm doesn't support ARM currently because Sandstorm apps run native Linux binaries, and every app would have to be compiled for each architecture. That's true of any linux distro providing binary packages. They all support arm anyways, it is trivially simple to compile packages. Even small projects like openbsd compile tens of thousands of packages for a dozen arches.
  - kentonv 10y ago
    Yes but distros accomplish that by being highly opinionated on the build process you use to build packages whereas Sandstorm tries to be unopinionated on this point. Sandstorm will support ARM someday but it's going to require a large investment in tooling in order to be painless for developers.
    - MustardTiger 10y ago
      >Yes but distros accomplish that by being highly opinionated on the build process you use to build packages whereas Sandstorm tries to be unopinionated on this point. How's that exactly? I looked it over and can't see any difference at all.

Animats 10y ago

If you're serving the pages yourself (not, as this site does, through Cloudflare's caches), what does this tell you that isn't in the server logs?

FiloSottile 10y ago
Unless you are tracking events, outlinks, time spent on page or something similar, nothing as far as data goes. But obviously server logs lack statistics and aggregation. PIWIK actually supports using the server logs as its data source instead of the javascript tag.
shermozle 10y ago
Deduplicated reach (Unique Visitors). IP addresses change and are shared, so they aren't suitable for this.

viana007 10y ago

The biggest problem with Piwik is that is not scalable and the cost ($) with servers to store analytics data. I saw many cases that Piwik not supported the big volume of data.

rhizome 10y ago
Why isn't it scalable?
- shermozle 10y ago
  MySQL
  - toomuchtodo 10y ago
    Put Rabbit or Kafka in front of MySQL then?
    - nacs 10y ago
      They have a Redis-based solution to sit in front of mySQL (seems to be a "Pro" feature though): https://piwik.org/faq/how-to/faq_19738/ https://piwik.org/faq/how-to/faq_19738/
      - moehm 10y ago
        It's not a Pro feature as in you have to pay for it. It's developed by the devs at piwik.pro, but you can download it for free.
  - rhizome 10y ago
    I see. What is the upper bound of "big volume of data" possible under MySQL + Piwik?
    - shermozle 10y ago
      That's a difficult thing to answer. However the more important problem is loss of data whenever your DB isn't available due to downtime, upgrade etc. It depends how important data loss is for your user case. I'm a data completist but I'm in therapy for it ;) It's definitely worth playing with, and trivially easy to spin up. Other self-hosted options aren't anything like as simple to get up and running.

mwexler 10y ago

There are many, many self-hosting analytics tools, from your own big data pipes to tools like the aforementioned piwik, as well as Open Web Analytics. I like Snowplow (http://snowplowanalytics.com/ http://snowplowanalytics.com/), but it's currently hard-coded to AWS. Hosting your own analytics data can be great, but there are lots of ways to get better accuracy and control over your data without having to host everything. Still, if you can, it's great.

yummyfajitas 10y ago
Snowplow is only loosely hard coded to AWS. I'm using it and breaking it free is only a few hundred lines of code. For example, rerouting Snowplow's Kinesis collector into Kafka is 114 lines, and that includes logging, metrics, etc - I basically just had to extend the AbstractSink object in their scala collector. Reading from Kafka is another couple of hundred lines, similarly writing to files.
- alexatkeplar 10y ago
  Thanks for sharing yummyfajitas - expect official Kafka support for Snowplow a little later this year [1] [2]; it's been long awaited! (Snowplow co-founder) [1] https://github.com/snowplow/snowplow/milestones/Kafka%20%231 https://github.com/snowplow/snowplow/milestones/Kafka%20%231 [2] https://github.com/snowplow/snowplow/milestones/Kafka%20%232 https://github.com/snowplow/snowplow/milestones/Kafka%20%232
  - yummyfajitas 10y ago
    Nice. If you build some sort of native maxmind or other geotargeting into the scala collector, that would also be cool. (Not that it was difficult to roll my own - so far snowplow is perfect for my needs - but obviously I'd rather use an official one.)
    - alexatkeplar 10y ago
      We do all enrichments like MaxMind, weather, arbitrary JavaScript etc downstream of collection, in our enrichment phase - the list of configurable enrichments is here: https://github.com/snowplow/snowplow/wiki/Configurable-enrichments https://github.com/snowplow/snowplow/wiki/Configurable-enric...

shermozle 10y ago

Piwik has the problem that it writes directly to MySQL as the activity happens. If your database is down, you lose data. If you have a spike of traffic above what your DB can handle in writes, you lose data. Snowplow doesn't have this problem.

moehm 10y ago
You can query your writes in Redis, so it won't be lost if your database goes down. https://piwik.org/faq/how-to/faq_19738/ https://piwik.org/faq/how-to/faq_19738/
- shermozle 10y ago
  Cos your Redis will never go down? Or need to be upgraded? Patched?
  - educar 10y ago
    How does snowplow solve this problem? By writing to disk?
    - alexatkeplar 10y ago
      Snowplow is implemented as a unidirectional data pipeline: tracking -> collection -> enrichment -> storage Between each step there is typically some kind of persistent queue (mostly S3/Kinesis), and data won't be lost if a downstream component is not operational. Examples: * If your event collector is unavailable, raw events will be cached in the tracker in localStorage, SQLite or similar * If your Redshift database is read-only for maintenance, enriched events will be held back until Redshift is writeable again
      - shermozle 10y ago
        There is of course the opportunity for a collector to be down, but the aim is to keep those components super simple and rely on really durable storage (Kinesis, S3) managed by someone else.

tirus 10y ago

For those of us that prefer not to be tracked by self-hosted Sandstorm apps, 2 new ublock rules (they're very rough, but I am no ublock wizard): ||sandcats.io/embed.js$script view.gif?page=$image Note that you would have to change the filters if the scripts and tracking pixel are renamed of course but this should catch a majority of the push button installs.

dragonne 10y ago
Why do you object to self-hosted analytics? I understand blocking centralized trackers (I do so myself), but self-hosted doesn't seem problematic in the same way GA being present on half the pages on the Internet is. It also strikes me as an unwinnable battle for all but the largest sites.
- angry-hacker 10y ago
  Because OP is against all kind of tracking? And because he can...
  - Programmatic 10y ago
    I can't claim to speak for OP, but am also against most tracking. I would also tend to think that being against first party tracking would be an unwinnable battle. It also leaks less data than third-party tracking, since the third party can see your activity across multiple sites whereas first party can only see your activity on that site unless it's aggregated through a backend service (another poster mentioned the ability to upload server logs to GA). No matter what, they can see what you load from their site. Getting to first party hosting of more intrusive analytics (scroll location, etc) I think rather than disallowing certain scripts/URLs to run, you have to get back to behavioral-based blocking. Doing that in an environment that you allow any JS to execute seems tough since sandboxing something that can update the page based on location can "talk" to another part that can report back to the server. If you don't like intrusive first party analytics, just stop all JS.
guelo 10y ago
If you visit a web site the operators of that site will know that someone at your IP address visited the site, which pages you viewed, how long you lingered, which buttons you clicked, etc. If you want to prevent that your only option is to browse via things like proxies or Tor.

_jomo 10y ago

Another very simple alternative is goaccess [0] which is purely server-side. It gathers quite a lot of information by parsing the server log. It doesn't do all the JavaScript stuff to track every single click, but it gives you stats on how many users visit your site, which parts, when, which sites or which domains they're coming from and also how much bandwidth they use. It also shows which status codes are coming from which paths and a lot more. It supports various output formats such as HTML or an interactive htop-like terminal application. It's also being actively developed. I use it and find it very useful. 0: https://www.goaccess.io/ https://www.goaccess.io/

herbst 10y ago
GoAccess is really one of the greatest tools of these kind. I use it with GA on all my servers.

deleted 10y ago

[deleted]

davidw 10y ago

One potential problem: if you, say, test out self-hosted alongside GA, and get different numbers, people are going to question the value of the self-hosted thing.

soared 10y ago
They should question both, but maybe GA more. GA is an approximation and absolutely not completely correct.
- davidw 10y ago
  Try selling that to a business guy or client or someone. Possible, but not easy, especially if there's ever a large discrepancy.
  - soared 10y ago
    Haha oh I know. Tell a client, "Yeah this is the data, it says this, BUT this is also how its wrong" and they always ignore the last part. Its data, that stuff can't be wrong!

__jal 10y ago

I've done my own analytics since before there were hosted services. Still use an old-school, proprietary tool called Sawmill[1]. One very nice aspect (out of many) about it is that it handles hundreds of other log formats out of the box, so it can report sensibly on switches, email, firewall logs... just about anything with very little effort. No connection, other than being a long-time happy customer. [1] http://sawmill.net/ http://sawmill.net/

blacksmith_tb 10y ago
I haven't used Sawmill in ages, but it wasn't bad, once upon a time. There's also GoAccess, which is a snazzy ncurses logfile analyzer: https://www.goaccess.io/ https://www.goaccess.io/
- mattab 10y ago
  Piwik also handle dozens of Log Formats, check it out: http://piwik.org/log-analytics/ http://piwik.org/log-analytics/ Github project: https://github.com/piwik/piwik-log-analytics https://github.com/piwik/piwik-log-analytics

vizza 10y ago

A middle of the road approach is to use an analytics package like https://truestats.com https://truestats.com that allows you to download raw session info at any time.

pixelmonkey 10y ago

Piwik is a great project, but it tends not to work well for handling sites with millions of events per day. Your MySQL table starts to bust at the seams pretty quickly. For big sites, you'll want that event data in GiB's of plain raw logs that you can bulk load into tools like BigQuery or Redshift for analysis. My team has built/delivered a SaaS web content analytics platform for the past few years called Parse.ly. We instrument pageview events (like Google Analytics) automatically, and we also instrument time-on-page using heartbeat events. We collect 50 billion monthly page events for over 600 top-traffic sites, and display it all in a real-time dashboard. To emphasize that our customers own the data they send us, we recently launched a Raw Data Pipeline product: http://parse.ly/data-pipeline http://parse.ly/data-pipeline Basically, we host a secure S3 bucket and Kinesis stream for customers, and deliver their raw (enriched) event data there. From there, they typically load it into their own BigQuery or Redshift instance, or they analyze pieces of it directly with Python/R/Excel/etc. Our customers tell us this strikes the right balance among data ownership, query flexibility, and hassle-free infrastructure.

educar 10y ago
Is their any difference in data ownership between GA and parsely?
- pixelmonkey 10y ago
  Data ownership in GA is a "gray area" that becomes less gray if you pay $150K/yr for "GA Premium". Google has mixed incentives in running its free analytics service. It gets web-wide analytics data, it uses data to help it sell more AdWords to customers, and it integrates GA with other services, like their display advertising products (DFP, etc.) From a practical standpoint, you don't "own" analytics data when a) you can't easily access it in raw form and b) the SaaS provider "leaks" your data to dilute its value to you. We address (a) and (b) directly through our products and public data privacy stance. See this blog post for our public view on analytics data privacy: http://blog.parsely.com/post/3394/analytics-privacy-without-compromise/ http://blog.parsely.com/post/3394/analytics-privacy-without-...

Ileca 10y ago

"Self-hosting analytics for better privacy and accuracy." Then why using google fonts which is listed by Disconnect.me (used by Firefox) as a tracking domain? Isn't that paradoxical?

ocdtrekkie 10y ago
I don't think* embedding Google Fonts allows Google much (any?) data collection beyond, presumably, that the font was requested, though it's definitely an unnecessary dependency. Given Google Fonts allows you to download all of the fonts, with license info provided, in a variety of formats, there's almost no good reason not to embed them directly in your own site. Right now, Sandstorm apps will currently work with Google Fonts, but the Sandstorm team intends to sandbox the client side better in the near-ish future, so I've already been nudging Sandstorm apps to make sure not to use such things when I see it. Note that while the author's blog uses Google Fonts, the Piwik Sandstorm package does not. (I checked.) *I don't know.
- 0942v8653 10y ago
  Google Fonts does see the page you are on, in the `Referer` header. According to mitmproxy: host: fonts.googleapis.com Connection: keep-alive Proxy-Connection: keep-alive Accept: text/css,*/*;q=0.1 User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13E238 Safari/601.1 Accept-Language: en-us Referer: https://blog.filippo.io/self-host-analytics/ Accept-Encoding: gzip, deflate (I agree with the rest of your comment; the best thing to do is host the fonts yourself.)
  - ocdtrekkie 10y ago
    And it does appear Google just blankets this into it's general API terms of use that they can "use submitted data" in accordance with their general privacy policies. So yeah, I guess they can use it as part of their tracking. :/
- ngrilly 10y ago

buremba 10y ago

Shameless plug: We're also working on an on-premise custom analytics platform that can be deployed either Heroku or AWS with Cloudformation. https://github.com/rakam-io/rakam https://github.com/rakam-io/rakam

manigandham 10y ago

Google Analytics has Measurement Protocol which allows for posting data server-side. All you need to do is proxy the data collection and then send it to them, taking advantage of all the scalability and features they have. https://developers.google.com/analytics/devguides/collection/protocol/v1/ https://developers.google.com/analytics/devguides/collection...

philliphaydon 10y ago

Off topic a bit. But with GA being blocked, all these blocks work by blocking URLs and domains at the time they are loaded, but not requests made after load of the requested resource. Couldn't you just proxy the URL to GA so it's not blocked?

dest 10y ago
By "requests made after load of the requested resource", you mean JS XHR requests for example? I'd guess they are also filtered by adblockers, aren't they?