7 ms·

How we built analytics.usa.gov

158 points by taylorbuley 12y ago

smacktoward 12y ago
> This data comes from a unified Google Analytics profile I wish my tax dollars were not being spent to help a for-profit entity track me across the Web.
- thomasjames 12y ago
  I think using a pre-existing software service as a base instead of a massive handmade distributed system may not be a bad idea for the USG. Remember the healthcare.gov rollout? Although it is certainly a government contract to a private corporation, the cause is warranted since it helps the federal government better allocate resources and make funding decisions. Your comment does raise the question, though, of whether the Digital Analytics Program managing the account requested any safeguards for user privacy or are just using the standard free or premium service. There was an article earlier around the launch of healthcare.gov that raised the same privacy issues you are now relating to Google Analytics, but I can't remember who published it.
- wpietri 12y ago
  If you'd like to opt out, try this: https://tools.google.com/dlpage/gaoptout https://tools.google.com/dlpage/gaoptout It says, "Available for Microsoft Internet Explorer 8-11, Google Chrome, Mozilla Firefox, Apple Safari and Opera". Personally, I prefer this to having my tax dollars spent on reinventing web analytics. Especially given that it wouldn't be cheap; Google bought what is now Google Analytics in 2005 for circa $30 million [1] and has put a lot into improving it since. And now that I think about it, I'm not sure that tax dollars are, net, being spent here. GA is very easy to install, and good analytics typically help save money through better decisions and focusing effort on what's actually being used. [1] http://searchenginewatch.com/sew/news/2062461/google-to-acquire-urchin-web-analytics-firm http://searchenginewatch.com/sew/news/2062461/google-to-acqu...
  - teacup50 12y ago
    Opt-out, really? We shouldn't have to opt-out of something that shouldn't be happening in the first place. The government shouldn't be inserting a 3rd-party -- Google -- into my communications with them! As for this: > ... and good analytics typically help save money through better decisions and focusing effort on what's actually being used. Pffft. That's only true for startups whose sole goal is maximizing traffic/retention/whatever while absolutely minimizing the amount of creativity they expend on moving those "key performance metrics".
    - nemothekid 12y ago
      >The government shouldn't be inserting a 3rd-party -- Google -- into my communications with them! I'd think you would be surprised to discover that the government has been using 3rd parties for years to manage various services. Did you know the government also uses Oracle databases to store your information?
      - coroutines 12y ago
        Similarly, we should stop caring about domestic surveillance by the gov because the gov has been doing it for years and we've only learned of it now. /s Btw, using Google Analytics is a lot different than using an Oracle database - Oracle isn't privy to what you're storing.
        nemothekid 12y ago
        >Oracle isn't privy to what you're storing. Do you work at Oracle? Are you 100% sure this is true? Have you audited all the hosted 3rd party solutions for the Government, and interviewed every employee to ensure that they aren't privy to your data? Have you ensured that all code written for the US govt has no cryptographic errors? The claim that the US Govt. shouldn't use google analytics because Google can read their data is hardly any different than arguing they shouldn't use managed Oracle solutions because some Oracle exec has access to their data. Where does it end? Should the US Govt. now spend and build their own analytics infrastructure? Should the US Govt. write their own databases? Print their own silicon? And what about everyone else? If a US company isn't trusted by the US Govt to keep data secure, what about insurance companies? Or Healthcare companies? Should these companies too, be also printing their own silicon? How is this a solution? If the problem is privacy, the solution is to create processes that respect privacy. Sitting in the dark because electricity might burn your house down is not a solution. And God forbid that the Govt coerces Google to share data on how people are using their own servers and that they are already making public.
- corobo 12y ago
  You'd rather them pay {m,b}illions more making their own system?
  - mycroft-holmes 12y ago
    I'd rather they spend nothing.
    - corobo 12y ago
      So now they're spending nothing they have no way to know if people are using their sites. They have no idea what sites are useful and what sites are not. Do they assume sites are not being used and have just a "Hi we are the USA" homepage or do they spend money on sites that are never visited Even the government needs analytics friend
      - mycroft-holmes 12y ago
        I'm all for the government not spending any money on any sites.
err4nt 12y ago
This is truly beautiful! I have also been experimenting with building HTML pages that load all content from a BaaS provider. I can host a single static HTML page on S3 and include a login view and content editor view in the same file to let authenticated users make changes to the content that loads into the page for public users. I was so impressed by your dashboard earlier, to read that you built it using the same techniques (empty page,populated by JSON) makes me really happy!
orasis 12y ago
Dang. I wistfully hoped this would be drill down real time analytics for the U.S. economy and government operations.
gandalfu 12y ago
Why will the US government send all their traffic stats to Google? Why not use piwik? http://piwik.org/ http://piwik.org/ Im I being too paranoid? This is after all third party software served by the government.
- mlissner 12y ago
  I think it's a reasonable question and I'm a big fan of piwik, but running it at this kind of scale is very hard. We get about 10k hits/day on the piwik instance we run and it's consistently taking more resources than the application it's tracking.
  - bjelkeman-again 12y ago
    Are there any other viable open source traffic analysis tools than Piwik? I'd hate to have to roll my own.
    - jbnicolai 12y ago
      https://github.com/divolte/divolte-collector https://github.com/divolte/divolte-collector is quite nice and can handle extreme loads
      - bjelkeman-again 12y ago
        That is interesting too for us, as Kafka is possibly in our future too. Thanks!
    - alexatkeplar 12y ago
      Snowplow (I'm a co-founder) can happily scale to billions of events per day: https://github.com/snowplow/snowplow https://github.com/snowplow/snowplow
      - e12e 12y ago
        Judging from the repo: "Collectors receive Snowplow events from trackers. Currently we have three different event collectors, sinking events either to Amazon S3 or Amazon Kinesis" (etc) -- it's still not viable to self-host snowplow on own hardware/internal cloud etc? Or is it possible, but you need to run a full cloud? (I understand why one would want a setup that runs on Amazon, if one uses amazon, but when you host your own infrastructure, a self-host option would be nice ... if viable). Without an option to self-host, snowplow isn't really an alternative to pwiki.
nbevans 12y ago
TL;DR is "we just used AWS in a completely unexciting way".
- snarkyturtle 12y ago
  Speed is sexy...
  - chrisan 12y ago
    And simplicity!
snarkyturtle 12y ago
Also, they used Neat/Bourbon as their css framework!
deleted 12y ago
[deleted]
gagabity 12y ago
Can you really overload S3? Also how long is the expiration time of CloudFront?
- chrisan 12y ago
  Cloudfront's default is 24 hours unless you specify otherwise
  - konklone 12y ago
    As of launch day, we started specifying a 5 minute cache time: https://github.com/GSA/analytics.usa.gov#deploying-the-app https://github.com/GSA/analytics.usa.gov#deploying-the-app We might increase that over time, but having fixes take effect within 5 minutes while the site is under the spotlight is a nice thing.
- tobz 12y ago
  I'm guessing they misspoke or misattributed their usage of CloudFront to protect S3. It's almost certainly used for predictable performance, since S3 is measurably worst in that respect.
vijayr 12y ago
Testers often assumed any three-digit number was actually short for a six-digit number. This is interesting, I haven't come across this before. What is the best way to display big numbers? 550K or 550,000? 5M or 5,000,000?
diminoten 12y ago
Just a technical question, but is '90daysago' and 'yesterday' a common way of representing time in the node world, or are they just trying to show a generic case? Also I know there's some turmoil in the Node.js/IO.js fork, what kind of practical implications does this have on projects like these?
- tobz 12y ago
  There's at least one other language (PHP, [1]) that accepts these natural language versions of times. It's not quite the same syntax, but I don't think it's necessarily uncommon for some libraries to support these sorts of queries, for lack of a better word. On the flipside, there's definitely a lot of libraries that support displaying times in this way (like moment.js, [2]). [1] http://php.net/manual/en/function.strtotime.php http://php.net/manual/en/function.strtotime.php [2] http://momentjs.com/ http://momentjs.com/
  - emmelaich 12y ago
    The GNU versions of find and date accept natural language for times. For find in the --newermt and similar predicates and for date in the the --date option.
    - tobz 12y ago
      Whattttttttt? This is amazing.
- konklone 12y ago
  That's actually the Google Analytics Reporting API's formal syntax: https://developers.google.com/analytics/devguides/reporting/core/v3/reference#q_summary https://developers.google.com/analytics/devguides/reporting/... It's pretty handy, since queries are so commonly exactly that: "the last 30 days" or whatever, and it saves the client from having to perform date calculation code to figure out just what that is.
gesman 12y ago
>> visitor IP addresses are anonymized before they are ever stored in Google Analytics.. That applies to data available to GA user. Google of course has access to more detailed data.
- konklone 12y ago
  Google claims not to, that they anonymize the IP before it ever hits disk: https://support.google.com/analytics/answer/2763052?hl=en https://support.google.com/analytics/answer/2763052?hl=en You have to trust Google to follow through, but my understanding is this was originally implemented to satisfy German privacy laws, so there's reason to think they take it seriously. How that impacts logging at their outermost network edge, and whether logs at that level could be feasibly correlated with anything at the application layer, I have no idea. It'd be nice if they clarified that.

158 points by taylorbuley 12y ago

> This data comes from a unified Google Analytics profile I wish my tax dollars were not being spent to help a for-profit entity track me across the Web.

thomasjames 12y ago
I think using a pre-existing software service as a base instead of a massive handmade distributed system may not be a bad idea for the USG. Remember the healthcare.gov rollout? Although it is certainly a government contract to a private corporation, the cause is warranted since it helps the federal government better allocate resources and make funding decisions. Your comment does raise the question, though, of whether the Digital Analytics Program managing the account requested any safeguards for user privacy or are just using the standard free or premium service. There was an article earlier around the launch of healthcare.gov that raised the same privacy issues you are now relating to Google Analytics, but I can't remember who published it.
wpietri 12y ago
If you'd like to opt out, try this: https://tools.google.com/dlpage/gaoptout https://tools.google.com/dlpage/gaoptout It says, "Available for Microsoft Internet Explorer 8-11, Google Chrome, Mozilla Firefox, Apple Safari and Opera". Personally, I prefer this to having my tax dollars spent on reinventing web analytics. Especially given that it wouldn't be cheap; Google bought what is now Google Analytics in 2005 for circa $30 million [1] and has put a lot into improving it since. And now that I think about it, I'm not sure that tax dollars are, net, being spent here. GA is very easy to install, and good analytics typically help save money through better decisions and focusing effort on what's actually being used. [1] http://searchenginewatch.com/sew/news/2062461/google-to-acquire-urchin-web-analytics-firm http://searchenginewatch.com/sew/news/2062461/google-to-acqu...
- teacup50 12y ago
  Opt-out, really? We shouldn't have to opt-out of something that shouldn't be happening in the first place. The government shouldn't be inserting a 3rd-party -- Google -- into my communications with them! As for this: > ... and good analytics typically help save money through better decisions and focusing effort on what's actually being used. Pffft. That's only true for startups whose sole goal is maximizing traffic/retention/whatever while absolutely minimizing the amount of creativity they expend on moving those "key performance metrics".
  - nemothekid 12y ago
    >The government shouldn't be inserting a 3rd-party -- Google -- into my communications with them! I'd think you would be surprised to discover that the government has been using 3rd parties for years to manage various services. Did you know the government also uses Oracle databases to store your information?
    - coroutines 12y ago
      Similarly, we should stop caring about domestic surveillance by the gov because the gov has been doing it for years and we've only learned of it now. /s Btw, using Google Analytics is a lot different than using an Oracle database - Oracle isn't privy to what you're storing.
      - nemothekid 12y ago
        >Oracle isn't privy to what you're storing. Do you work at Oracle? Are you 100% sure this is true? Have you audited all the hosted 3rd party solutions for the Government, and interviewed every employee to ensure that they aren't privy to your data? Have you ensured that all code written for the US govt has no cryptographic errors? The claim that the US Govt. shouldn't use google analytics because Google can read their data is hardly any different than arguing they shouldn't use managed Oracle solutions because some Oracle exec has access to their data. Where does it end? Should the US Govt. now spend and build their own analytics infrastructure? Should the US Govt. write their own databases? Print their own silicon? And what about everyone else? If a US company isn't trusted by the US Govt to keep data secure, what about insurance companies? Or Healthcare companies? Should these companies too, be also printing their own silicon? How is this a solution? If the problem is privacy, the solution is to create processes that respect privacy. Sitting in the dark because electricity might burn your house down is not a solution. And God forbid that the Govt coerces Google to share data on how people are using their own servers and that they are already making public.
corobo 12y ago
You'd rather them pay {m,b}illions more making their own system?
- mycroft-holmes 12y ago
  I'd rather they spend nothing.
  - corobo 12y ago
    So now they're spending nothing they have no way to know if people are using their sites. They have no idea what sites are useful and what sites are not. Do they assume sites are not being used and have just a "Hi we are the USA" homepage or do they spend money on sites that are never visited Even the government needs analytics friend
    - mycroft-holmes 12y ago
      I'm all for the government not spending any money on any sites.

err4nt 12y ago

This is truly beautiful! I have also been experimenting with building HTML pages that load all content from a BaaS provider. I can host a single static HTML page on S3 and include a login view and content editor view in the same file to let authenticated users make changes to the content that loads into the page for public users. I was so impressed by your dashboard earlier, to read that you built it using the same techniques (empty page,populated by JSON) makes me really happy!

orasis 12y ago

Dang. I wistfully hoped this would be drill down real time analytics for the U.S. economy and government operations.

gandalfu 12y ago

Why will the US government send all their traffic stats to Google? Why not use piwik? http://piwik.org/ http://piwik.org/ Im I being too paranoid? This is after all third party software served by the government.

mlissner 12y ago
I think it's a reasonable question and I'm a big fan of piwik, but running it at this kind of scale is very hard. We get about 10k hits/day on the piwik instance we run and it's consistently taking more resources than the application it's tracking.
- bjelkeman-again 12y ago
  Are there any other viable open source traffic analysis tools than Piwik? I'd hate to have to roll my own.
  - jbnicolai 12y ago
    https://github.com/divolte/divolte-collector https://github.com/divolte/divolte-collector is quite nice and can handle extreme loads
    - bjelkeman-again 12y ago
      That is interesting too for us, as Kafka is possibly in our future too. Thanks!
  - alexatkeplar 12y ago
    Snowplow (I'm a co-founder) can happily scale to billions of events per day: https://github.com/snowplow/snowplow https://github.com/snowplow/snowplow
    - e12e 12y ago
      Judging from the repo: "Collectors receive Snowplow events from trackers. Currently we have three different event collectors, sinking events either to Amazon S3 or Amazon Kinesis" (etc) -- it's still not viable to self-host snowplow on own hardware/internal cloud etc? Or is it possible, but you need to run a full cloud? (I understand why one would want a setup that runs on Amazon, if one uses amazon, but when you host your own infrastructure, a self-host option would be nice ... if viable). Without an option to self-host, snowplow isn't really an alternative to pwiki.

nbevans 12y ago

TL;DR is "we just used AWS in a completely unexciting way".

snarkyturtle 12y ago
Speed is sexy...
- chrisan 12y ago
  And simplicity!

snarkyturtle 12y ago

Also, they used Neat/Bourbon as their css framework!

deleted 12y ago

[deleted]

gagabity 12y ago

Can you really overload S3? Also how long is the expiration time of CloudFront?

chrisan 12y ago
Cloudfront's default is 24 hours unless you specify otherwise
- konklone 12y ago
  As of launch day, we started specifying a 5 minute cache time: https://github.com/GSA/analytics.usa.gov#deploying-the-app https://github.com/GSA/analytics.usa.gov#deploying-the-app We might increase that over time, but having fixes take effect within 5 minutes while the site is under the spotlight is a nice thing.
tobz 12y ago
I'm guessing they misspoke or misattributed their usage of CloudFront to protect S3. It's almost certainly used for predictable performance, since S3 is measurably worst in that respect.

vijayr 12y ago

Testers often assumed any three-digit number was actually short for a six-digit number. This is interesting, I haven't come across this before. What is the best way to display big numbers? 550K or 550,000? 5M or 5,000,000?

diminoten 12y ago

Just a technical question, but is '90daysago' and 'yesterday' a common way of representing time in the node world, or are they just trying to show a generic case? Also I know there's some turmoil in the Node.js/IO.js fork, what kind of practical implications does this have on projects like these?

tobz 12y ago
There's at least one other language (PHP, [1]) that accepts these natural language versions of times. It's not quite the same syntax, but I don't think it's necessarily uncommon for some libraries to support these sorts of queries, for lack of a better word. On the flipside, there's definitely a lot of libraries that support displaying times in this way (like moment.js, [2]). [1] http://php.net/manual/en/function.strtotime.php http://php.net/manual/en/function.strtotime.php [2] http://momentjs.com/ http://momentjs.com/
- emmelaich 12y ago
  The GNU versions of find and date accept natural language for times. For find in the --newermt and similar predicates and for date in the the --date option.
  - tobz 12y ago
    Whattttttttt? This is amazing.
konklone 12y ago
That's actually the Google Analytics Reporting API's formal syntax: https://developers.google.com/analytics/devguides/reporting/core/v3/reference#q_summary https://developers.google.com/analytics/devguides/reporting/... It's pretty handy, since queries are so commonly exactly that: "the last 30 days" or whatever, and it saves the client from having to perform date calculation code to figure out just what that is.

gesman 12y ago

>> visitor IP addresses are anonymized before they are ever stored in Google Analytics.. That applies to data available to GA user. Google of course has access to more detailed data.

konklone 12y ago
Google claims not to, that they anonymize the IP before it ever hits disk: https://support.google.com/analytics/answer/2763052?hl=en https://support.google.com/analytics/answer/2763052?hl=en You have to trust Google to follow through, but my understanding is this was originally implemented to satisfy German privacy laws, so there's reason to think they take it seriously. How that impacts logging at their outermost network edge, and whether logs at that level could be feasibly correlated with anything at the application layer, I have no idea. It'd be nice if they clarified that.