3 ms·

I'll bet one of the reasons is the amount of data. At a previous employer I built a system similar to this which produced nightly, weekly, monthly, and annual r

I'll bet one of the reasons is the amount of data. At a previous employer I built a system similar to this which produced nightly, weekly, monthly, and annual reports of weblog analysis for an app that had 6 million+ http requests per day. It was tough enough to consolidate those logs across the load-balanced servers in a single data-center. We never consolidated across data-centers (separate reports for each instead) and I doubt we could have shipped all of that data to a cloud service in a reasonable amount of time. BTW, we made heavy use of setting and logging http headers too. One trick I liked was capturing performance timing metrics as a request was processed and stuffing it into a response header as the response went out. We then logged the response headers, which gave us the ability to report on the performance metrics. We also had a debug mode in the app on the browser side so we could see the performance metrics from the headers there too.

dvanduzer 13y ago
When was this? ~70 events per second doesn't sound like much to capture and aggregate. How much of this parsing did you need to perform in real time? Creating a unique token to pair requests/responses shouldn't add much overhead at all.
- DougWebb 13y ago
  Development started around 2000, and stabilized around 2008. As far as I know the reporting scripts are still being run every day. During this period we had purchased a 1TB storage rack from EMC for a million dollars, to give you some perspective on the differences between then and now. - No real-time parsing; it's all nightly batch processing after devops rotates the Apache server logs to a storage volume. The logs sit there for a while then get compressed and moved to offline tape archives. - No DB storage of the logs; space was too expensive and the Oracle database we had couldn't have kept up. It was already heavily burdened with a completely separate usage statistics system that fed into user-facing reporting and billing, which had a much higher event rate, about 100x higher, than the http logs. - We had unique tokens, but they identified a particular user session that tied together all of the user's http requests from login to logoff/abandonment, and which also tied into the Oracle-based statistics for that user, that user's organization, and the customer responsible for the user (often multi-organization). My reports had breakdowns for individual user experiences, session-level metrics, and user type/organization/customer/region/etc metrics. - I don't recall how long the analysis took; it was between half an hour to two hours I think. A lot of that time was spent on disk I/O reading the logs. I had optimized the parsing, analysis, and results recording about as much as I could. - This stuff was written in Perl, and ran on Solaris servers from that time era... probably not a lot more powerful than a handful of smartphones today, though they did have lots of cpus. I don't think traffic has grown much since I left the company (we had pretty full market penetration already) so it's likely those servers haven't been upgraded.
  - dvanduzer 13y ago
    I suspected it would have to be a system of that era. I think I have a good idea of how businesses (at a high level) have failed to understand Moore's law from 2000-present. I'm curious what those failures of understanding were like from 1985-2000. We all know that technology has been advancing rapidly, but these specific anecdotes of organizations paying a million dollars just for the backing storage of a system that you can essentially get for free from Google now...
    - DougWebb 13y ago
      Yeah, it's pretty amazing how much things have changed. That raw log data was about 250GB/year which is nothing today but when we started collecting it we were paying $1000/GB. Actually, they're probably still paying over $100/GB. The whole datacenter was outsourced to Perot Systems in the mid-2000s, and the storage fees were astronomical. We calculated that Perot must pay a separate tech to stare at each individual hard drive with a replacement in-hand in case any errors were reported. At least, they could afford to do that with what we were paying them for storage.

dvanduzer 13y ago

When was this? ~70 events per second doesn't sound like much to capture and aggregate. How much of this parsing did you need to perform in real time? Creating a unique token to pair requests/responses shouldn't add much overhead at all.

DougWebb 13y ago
Development started around 2000, and stabilized around 2008. As far as I know the reporting scripts are still being run every day. During this period we had purchased a 1TB storage rack from EMC for a million dollars, to give you some perspective on the differences between then and now. - No real-time parsing; it's all nightly batch processing after devops rotates the Apache server logs to a storage volume. The logs sit there for a while then get compressed and moved to offline tape archives. - No DB storage of the logs; space was too expensive and the Oracle database we had couldn't have kept up. It was already heavily burdened with a completely separate usage statistics system that fed into user-facing reporting and billing, which had a much higher event rate, about 100x higher, than the http logs. - We had unique tokens, but they identified a particular user session that tied together all of the user's http requests from login to logoff/abandonment, and which also tied into the Oracle-based statistics for that user, that user's organization, and the customer responsible for the user (often multi-organization). My reports had breakdowns for individual user experiences, session-level metrics, and user type/organization/customer/region/etc metrics. - I don't recall how long the analysis took; it was between half an hour to two hours I think. A lot of that time was spent on disk I/O reading the logs. I had optimized the parsing, analysis, and results recording about as much as I could. - This stuff was written in Perl, and ran on Solaris servers from that time era... probably not a lot more powerful than a handful of smartphones today, though they did have lots of cpus. I don't think traffic has grown much since I left the company (we had pretty full market penetration already) so it's likely those servers haven't been upgraded.
- dvanduzer 13y ago
  I suspected it would have to be a system of that era. I think I have a good idea of how businesses (at a high level) have failed to understand Moore's law from 2000-present. I'm curious what those failures of understanding were like from 1985-2000. We all know that technology has been advancing rapidly, but these specific anecdotes of organizations paying a million dollars just for the backing storage of a system that you can essentially get for free from Google now...
  - DougWebb 13y ago
    Yeah, it's pretty amazing how much things have changed. That raw log data was about 250GB/year which is nothing today but when we started collecting it we were paying $1000/GB. Actually, they're probably still paying over $100/GB. The whole datacenter was outsourced to Perot Systems in the mid-2000s, and the storage fees were astronomical. We calculated that Perot must pay a separate tech to stare at each individual hard drive with a replacement in-hand in case any errors were reported. At least, they could afford to do that with what we were paying them for storage.