4 ms·

You know for things like being able to do analysis. For example, last week I wanted to know where to locate our next local server and so needed to know where th

You know for things like being able to do analysis. For example, last week I wanted to know where to locate our next local server and so needed to know where the usage growth was coming from. I happen to noticed that the usage over the last few weeks was quite different to the historical data. If I only had 2 weeks of data I would have made our service worse for our customers.

e12e 9y ago
You might be a good candidate for storing aggregated (anonymized) data. [ed: incidentally you're kinda sorta in the area GDPR wants to combat: "it'd be nice to know what x want to buy next, and where - for logistical and marketing purposes. Why can't we just store a full profile of everything x does, where and when?]
- danieltillett 9y ago
  The problem becomes if you make a mistake or aggregate in the wrong way you end up losing the ability to do later analysis. If I had thrown out my logs I would have provided my customers with a worse experience. I am supportive of the intent of the GDPR and we have always gone out of our way to minimise the data we collect, but as currently written the law has a whole lot of very negative downstream effects. Really the law seems to have been written to catch a few bad players and has ended up netting everyone in the world.
  - e12e 9y ago
    Re: "Where usage growth was coming from" sounds like the kind of things a log of performance data pr server might also answer. Can't argue with the fact that storing less and corner grained data will make certain ad hoc queries harder. That's kinda the point.
    - danieltillett 9y ago
      Performance data is not going to tell me where the users are located. Yes it was only because I had the logs was I able to do this analysis. The issue is not over these sort of analyses which I am sure nobody would object to, but that GDPR casts such a wide net.
      - e12e 9y ago
        You implied you already had "local" servers - it sounded like bandwidth/load information might indicate the area were load was increasing.
        danieltillett 9y ago
        Local to the market - actually they are regional. The complication is that I have a client-based fallover where when one server becomes overloaded the client is switched automatically over to a different server (client based load balancing). This make it hard to sort out where the load is coming from without looking at the IP address.
pbowyer 9y ago
You can't use one of the lawful basis of consent for this? https://ico.org.uk/for-organisations/guide-to-the-general-data-protection-regulation-gdpr/lawful-basis-for-processing/ https://ico.org.uk/for-organisations/guide-to-the-general-da...
- danieltillett 9y ago
  The lawful basis of consent are a joke. You could capture all data under "legitimate interests" and then allow all further processing under "statistical purposes”.

e12e 9y ago

You might be a good candidate for storing aggregated (anonymized) data. [ed: incidentally you're kinda sorta in the area GDPR wants to combat: "it'd be nice to know what x want to buy next, and where - for logistical and marketing purposes. Why can't we just store a full profile of everything x does, where and when?]

danieltillett 9y ago
The problem becomes if you make a mistake or aggregate in the wrong way you end up losing the ability to do later analysis. If I had thrown out my logs I would have provided my customers with a worse experience. I am supportive of the intent of the GDPR and we have always gone out of our way to minimise the data we collect, but as currently written the law has a whole lot of very negative downstream effects. Really the law seems to have been written to catch a few bad players and has ended up netting everyone in the world.
- e12e 9y ago
  Re: "Where usage growth was coming from" sounds like the kind of things a log of performance data pr server might also answer. Can't argue with the fact that storing less and corner grained data will make certain ad hoc queries harder. That's kinda the point.
  - danieltillett 9y ago
    Performance data is not going to tell me where the users are located. Yes it was only because I had the logs was I able to do this analysis. The issue is not over these sort of analyses which I am sure nobody would object to, but that GDPR casts such a wide net.
    - e12e 9y ago
      You implied you already had "local" servers - it sounded like bandwidth/load information might indicate the area were load was increasing.
      - danieltillett 9y ago
        Local to the market - actually they are regional. The complication is that I have a client-based fallover where when one server becomes overloaded the client is switched automatically over to a different server (client based load balancing). This make it hard to sort out where the load is coming from without looking at the IP address.

pbowyer 9y ago

You can't use one of the lawful basis of consent for this? https://ico.org.uk/for-organisations/guide-to-the-general-data-protection-regulation-gdpr/lawful-basis-for-processing/ https://ico.org.uk/for-organisations/guide-to-the-general-da...

danieltillett 9y ago
The lawful basis of consent are a joke. You could capture all data under "legitimate interests" and then allow all further processing under "statistical purposes”.