4 ms·

"We don't really like the idea of cloud hosted monitoring" Can you elaborate on the reasons?

by berkay 13y ago

DougWebb 13y ago
I'll bet one of the reasons is the amount of data. At a previous employer I built a system similar to this which produced nightly, weekly, monthly, and annual reports of weblog analysis for an app that had 6 million+ http requests per day. It was tough enough to consolidate those logs across the load-balanced servers in a single data-center. We never consolidated across data-centers (separate reports for each instead) and I doubt we could have shipped all of that data to a cloud service in a reasonable amount of time. BTW, we made heavy use of setting and logging http headers too. One trick I liked was capturing performance timing metrics as a request was processed and stuffing it into a response header as the response went out. We then logged the response headers, which gave us the ability to report on the performance metrics. We also had a debug mode in the app on the browser side so we could see the performance metrics from the headers there too.
- dvanduzer 13y ago
  When was this? ~70 events per second doesn't sound like much to capture and aggregate. How much of this parsing did you need to perform in real time? Creating a unique token to pair requests/responses shouldn't add much overhead at all.
  - DougWebb 13y ago
    Development started around 2000, and stabilized around 2008. As far as I know the reporting scripts are still being run every day. During this period we had purchased a 1TB storage rack from EMC for a million dollars, to give you some perspective on the differences between then and now. - No real-time parsing; it's all nightly batch processing after devops rotates the Apache server logs to a storage volume. The logs sit there for a while then get compressed and moved to offline tape archives. - No DB storage of the logs; space was too expensive and the Oracle database we had couldn't have kept up. It was already heavily burdened with a completely separate usage statistics system that fed into user-facing reporting and billing, which had a much higher event rate, about 100x higher, than the http logs. - We had unique tokens, but they identified a particular user session that tied together all of the user's http requests from login to logoff/abandonment, and which also tied into the Oracle-based statistics for that user, that user's organization, and the customer responsible for the user (often multi-organization). My reports had breakdowns for individual user experiences, session-level metrics, and user type/organization/customer/region/etc metrics. - I don't recall how long the analysis took; it was between half an hour to two hours I think. A lot of that time was spent on disk I/O reading the logs. I had optimized the parsing, analysis, and results recording about as much as I could. - This stuff was written in Perl, and ran on Solaris servers from that time era... probably not a lot more powerful than a handful of smartphones today, though they did have lots of cpus. I don't think traffic has grown much since I left the company (we had pretty full market penetration already) so it's likely those servers haven't been upgraded.
    - dvanduzer 13y ago
      I suspected it would have to be a system of that era. I think I have a good idea of how businesses (at a high level) have failed to understand Moore's law from 2000-present. I'm curious what those failures of understanding were like from 1985-2000. We all know that technology has been advancing rapidly, but these specific anecdotes of organizations paying a million dollars just for the backing storage of a system that you can essentially get for free from Google now...
      - DougWebb 13y ago
        Yeah, it's pretty amazing how much things have changed. That raw log data was about 250GB/year which is nothing today but when we started collecting it we were paying $1000/GB. Actually, they're probably still paying over $100/GB. The whole datacenter was outsourced to Perot Systems in the mid-2000s, and the storage fees were astronomical. We calculated that Perot must pay a separate tech to stare at each individual hard drive with a replacement in-hand in case any errors were reported. At least, they could afford to do that with what we were paying them for storage.
jmelloy 13y ago
We're doing an evaluation of a bunch at work, and we usually come down to: 1) most charge per server per month, making costs go up quickly 2) you don't own the data & the API May not exist or be wonky with implementation details 3) most install agents, meaning your production servers have another piece of code on them, and it's hard to test how they behave under load
KyleBrandt 13y ago
1. The amount of data would be quite large when it comes to the logs 2. System lock-in, I like to have the data and be able to query it as I see fit 3. If an event happens where the facility is cut off from the Internet you won't be able to tell what happened inside the facility (unless maybe there is a store and forward agent, but even then it is only useful after the event. 4. Latency monitoring (again maybe an agent can help) but if I see changes in response time I can't tell if that is the WAN or not.
- dvanduzer 13y ago
  One point 3, that makes high level (non-agent-based) cloud monitoring a must in addition to your primary monitoring. In that kind of event, you need local and global monitoring.
  - KyleBrandt 13y ago
    I like the idea of a totally independent 3rd party monitoring system. For that we use pingdom, and its job is basically to tell us if our public services are up from to the outside world. Basically it is an additional perspective and a redundant level of monitoring. But I view it mostly as an up/down layer of monitoring. Extremely important, but simple and not really meant to give insight into the complexity of our system.

"We don't really like the idea of cloud hosted monitoring" Can you elaborate on the reasons?

by berkay 13y ago

"We don't really like the idea of cloud hosted monitoring" Can you elaborate on the reasons?

I'll bet one of the reasons is the amount of data. At a previous employer I built a system similar to this which produced nightly, weekly, monthly, and annual reports of weblog analysis for an app that had 6 million+ http requests per day. It was tough enough to consolidate those logs across the load-balanced servers in a single data-center. We never consolidated across data-centers (separate reports for each instead) and I doubt we could have shipped all of that data to a cloud service in a reasonable amount of time. BTW, we made heavy use of setting and logging http headers too. One trick I liked was capturing performance timing metrics as a request was processed and stuffing it into a response header as the response went out. We then logged the response headers, which gave us the ability to report on the performance metrics. We also had a debug mode in the app on the browser side so we could see the performance metrics from the headers there too.

dvanduzer 13y ago
When was this? ~70 events per second doesn't sound like much to capture and aggregate. How much of this parsing did you need to perform in real time? Creating a unique token to pair requests/responses shouldn't add much overhead at all.
- DougWebb 13y ago
  Development started around 2000, and stabilized around 2008. As far as I know the reporting scripts are still being run every day. During this period we had purchased a 1TB storage rack from EMC for a million dollars, to give you some perspective on the differences between then and now. - No real-time parsing; it's all nightly batch processing after devops rotates the Apache server logs to a storage volume. The logs sit there for a while then get compressed and moved to offline tape archives. - No DB storage of the logs; space was too expensive and the Oracle database we had couldn't have kept up. It was already heavily burdened with a completely separate usage statistics system that fed into user-facing reporting and billing, which had a much higher event rate, about 100x higher, than the http logs. - We had unique tokens, but they identified a particular user session that tied together all of the user's http requests from login to logoff/abandonment, and which also tied into the Oracle-based statistics for that user, that user's organization, and the customer responsible for the user (often multi-organization). My reports had breakdowns for individual user experiences, session-level metrics, and user type/organization/customer/region/etc metrics. - I don't recall how long the analysis took; it was between half an hour to two hours I think. A lot of that time was spent on disk I/O reading the logs. I had optimized the parsing, analysis, and results recording about as much as I could. - This stuff was written in Perl, and ran on Solaris servers from that time era... probably not a lot more powerful than a handful of smartphones today, though they did have lots of cpus. I don't think traffic has grown much since I left the company (we had pretty full market penetration already) so it's likely those servers haven't been upgraded.
  - dvanduzer 13y ago
    I suspected it would have to be a system of that era. I think I have a good idea of how businesses (at a high level) have failed to understand Moore's law from 2000-present. I'm curious what those failures of understanding were like from 1985-2000. We all know that technology has been advancing rapidly, but these specific anecdotes of organizations paying a million dollars just for the backing storage of a system that you can essentially get for free from Google now...
    - DougWebb 13y ago
      Yeah, it's pretty amazing how much things have changed. That raw log data was about 250GB/year which is nothing today but when we started collecting it we were paying $1000/GB. Actually, they're probably still paying over $100/GB. The whole datacenter was outsourced to Perot Systems in the mid-2000s, and the storage fees were astronomical. We calculated that Perot must pay a separate tech to stare at each individual hard drive with a replacement in-hand in case any errors were reported. At least, they could afford to do that with what we were paying them for storage.

jmelloy 13y ago

We're doing an evaluation of a bunch at work, and we usually come down to: 1) most charge per server per month, making costs go up quickly 2) you don't own the data & the API May not exist or be wonky with implementation details 3) most install agents, meaning your production servers have another piece of code on them, and it's hard to test how they behave under load

KyleBrandt 13y ago

1. The amount of data would be quite large when it comes to the logs 2. System lock-in, I like to have the data and be able to query it as I see fit 3. If an event happens where the facility is cut off from the Internet you won't be able to tell what happened inside the facility (unless maybe there is a store and forward agent, but even then it is only useful after the event. 4. Latency monitoring (again maybe an agent can help) but if I see changes in response time I can't tell if that is the WAN or not.

dvanduzer 13y ago
One point 3, that makes high level (non-agent-based) cloud monitoring a must in addition to your primary monitoring. In that kind of event, you need local and global monitoring.
- KyleBrandt 13y ago
  I like the idea of a totally independent 3rd party monitoring system. For that we use pingdom, and its job is basically to tell us if our public services are up from to the outside world. Basically it is an additional perspective and a redundant level of monitoring. But I view it mostly as an up/down layer of monitoring. Extremely important, but simple and not really meant to give insight into the complexity of our system.