3 ms·

Relevant posts about how other companies do it Cloudflare - https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/ https://blo

by flurly 5y ago

Relevant posts about how other companies do it Cloudflare - https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/ https://blog.cloudflare.com/http-analytics-for-6m-requests-p... Uber - https://eng.uber.com/logging/ https://eng.uber.com/logging/ Facebook - https://research.facebook.com/publications/scuba-diving-into-data-at-facebook/ https://research.facebook.com/publications/scuba-diving-into... For those who are smaller and don't have the money to pay for Splunk enterprise, and don't have the headcount to build your own logging infrastructure, I built a a service called GraphJSON that makes it super easy to log and process any type of data. You can read more about how and why I built it here https://www.graphjson.com/guides/about https://www.graphjson.com/guides/about

sockpuppet69 5y ago
bckr 5y ago
Thanks for the links! I was hoping TFA would break down their log-based observability strategy and go into things like trace ids, structured logs.... Instead I am disappointed to learn that engineering at Twitter still sounds.... suboptimal. This from the company that brought us the fail whale (I can't find the blog post now, but I recall Twitter's instability was due to a massive kludge that went unnoticed for years).
falafelite 5y ago
Just started my 2 week trial of GraphJSON a few days ago! I like it so far, kudos. Unrelated question: do you have some kind of notification setup for HN? I feel like I’ve seen you comment on many logging-related posts, so figured I’d ask =)
- flurly 5y ago
  Thanks! I do use f5bot, but in this case I was just perusing HN and figured I'd drop some links and a plug :)
javiermaestro 5y ago
Just a nit, for Facebook, Scuba is not the logging ingestion service. It's Scribe, which Scuba itself uses as the ingestion :) https://engineering.fb.com/2019/10/07/data-infrastructure/scribe/ https://engineering.fb.com/2019/10/07/data-infrastructure/sc... (I worked in Scribe)
- cduzz 5y ago
  It's a pity that scribe is both astonishingly great and totally abandoned.
  - javiermaestro 5y ago
    Another nit, the "abandoned" part is the one that was open sourced a loooong time ago. Since then, it underwent a ton of changes but yeah, it's within Facebook. So, not abandoned at all, but not available outside. The main issue (as usual in big companies) is the large amount of inter-dependencies with internal systems. Scribe as-is today doesn't make much sense outside Facebook. And yeah, it could be cleaned up, mock the internal services and OSS it. But that's a lot of work, both doing it and maintaining it. And having all those mocks, etc "wouldn't be Scribe as-is either"in terms of how e.g. it scales and so on. In any case, the storage (LogDevice, https://github.com/facebookarchive/LogDevice https://github.com/facebookarchive/LogDevice) which is a large part of the system was open sourced a while back and... sadly it went again into "not-maintained OSS". Finally, you can get a similar-ish system working with OSS tools (e.g. fluentd + Kafka) that will also scale quite well and which IMHO can also made to scale to Facebook-size levels. So, the incentive is there, but there are OSS alternatives already available :)
    - cduzz 5y ago
      Well, the open source part of it is totally abandoned. The internal improvements aren't interesting or available to anyone outside of facebook. The open source version is creaky and very difficult to drag forward to newer versions of thrift. So -- thanks Facebook for dropping weird artifacts like the aftermath of Roadside Picnic. Woe unto all who adopt these soon-to-be-orphaned technologies.

by flurly 5y ago

sockpuppet69 5y ago

bckr 5y ago

Thanks for the links! I was hoping TFA would break down their log-based observability strategy and go into things like trace ids, structured logs.... Instead I am disappointed to learn that engineering at Twitter still sounds.... suboptimal. This from the company that brought us the fail whale (I can't find the blog post now, but I recall Twitter's instability was due to a massive kludge that went unnoticed for years).

falafelite 5y ago

Just started my 2 week trial of GraphJSON a few days ago! I like it so far, kudos. Unrelated question: do you have some kind of notification setup for HN? I feel like I’ve seen you comment on many logging-related posts, so figured I’d ask =)

flurly 5y ago
Thanks! I do use f5bot, but in this case I was just perusing HN and figured I'd drop some links and a plug :)

javiermaestro 5y ago

Just a nit, for Facebook, Scuba is not the logging ingestion service. It's Scribe, which Scuba itself uses as the ingestion :) https://engineering.fb.com/2019/10/07/data-infrastructure/scribe/ https://engineering.fb.com/2019/10/07/data-infrastructure/sc... (I worked in Scribe)

cduzz 5y ago
It's a pity that scribe is both astonishingly great and totally abandoned.
- javiermaestro 5y ago
  Another nit, the "abandoned" part is the one that was open sourced a loooong time ago. Since then, it underwent a ton of changes but yeah, it's within Facebook. So, not abandoned at all, but not available outside. The main issue (as usual in big companies) is the large amount of inter-dependencies with internal systems. Scribe as-is today doesn't make much sense outside Facebook. And yeah, it could be cleaned up, mock the internal services and OSS it. But that's a lot of work, both doing it and maintaining it. And having all those mocks, etc "wouldn't be Scribe as-is either"in terms of how e.g. it scales and so on. In any case, the storage (LogDevice, https://github.com/facebookarchive/LogDevice https://github.com/facebookarchive/LogDevice) which is a large part of the system was open sourced a while back and... sadly it went again into "not-maintained OSS". Finally, you can get a similar-ish system working with OSS tools (e.g. fluentd + Kafka) that will also scale quite well and which IMHO can also made to scale to Facebook-size levels. So, the incentive is there, but there are OSS alternatives already available :)
  - cduzz 5y ago
    Well, the open source part of it is totally abandoned. The internal improvements aren't interesting or available to anyone outside of facebook. The open source version is creaky and very difficult to drag forward to newer versions of thrift. So -- thanks Facebook for dropping weird artifacts like the aftermath of Roadside Picnic. Woe unto all who adopt these soon-to-be-orphaned technologies.