30 ms·

Bigquery does not have anything close to real time query support like you want/need for observability data. And plus the cost is more like one engineer per quer

Bigquery does not have anything close to real time query support like you want/need for observability data. And plus the cost is more like one engineer per query. That shit is crazy expensive.

manigandham 6y ago
What do you mean by real-time? BQ can scan a petabyte in less than a minute, but partitioning and clustering will get most simple queries back in a few seconds. If you mean data freshness than BQ supports streaming ingest and will show those rows in the next query. SQL also lets you do a lot of aggregations in the database instead of pulling back raw metrics. And BQ has flat-rate pricing and discounts for large clients.
- buremba 6y ago
  Try to run a few petabyte-scale queries and you will be surprised when you see the invoice. If you switch to flat-rate pricing, don't expect the latency that you mentioned.
  - manigandham 6y ago
    You dont need to scan the whole dataset. Partitioning + clustering, especially for time-series, should be very efficient. BQ flat-rate pricing starts at 10k/month for a fixed number of processing slots. You can buy more slots to fit your query load. Anyways the point is that it's well within the financial means of Twitter, especially when compared to developing and operating this proprietary system.
    - buremba 6y ago
      If I'm not mistaken Twitter is mostly on GCP so they could just go with BigQuery instead of developing an in-house solution. We probably need to ask them in order to get a proper answer. :)
- rco8786 6y ago
  Less than a minute is not real time. Something like < 100ms for most queries would be real time. And then do that 10s of 1000s of times per minute, every minute, 24 hours a day. You can see how the query costs add up... I know it's easy to come on HN and play armchair architect but this is one area where you are just incorrect in your assumptions. Did Twitter need to build a(nother) in-house database? I don't know for sure. But I do know that BQ is not well suited to observability use cases.
  - manigandham 6y ago
    As I said, that's for scanning a petabyte. Selective queries would be much smaller and faster, costs can be handled with flat-rate pricing, and there's the BI Engine (in-memory cache) feature too. The details in the article (1.5PB logical data, 3x replication factor, only 2% of metrics ever read) and the fact that Twitter already runs infra in Google Cloud seem to align well with BQ in my experience (10+ years in adtech building even more complex backends). Please tell me what assumptions are incorrect so I can reconsider.
    - rco8786 6y ago
      My experience is ~2.5 years as a senior engineer on the Observability team at Twitter (from 2013-2015). I was part of the migration to Manhattan (mentioned in the post) from Cassandra, new alerting infra, query language design, among many other things. Your adtech experience should make you particularly sensitive to query latencies, so I find it interesting that you're glossing over that. Yes, only 2% of metrics ever read. That was the same back then too. The kicker is that you can't be sure which will be read, so the systems are built to be able to read any of the metrics with the same SLA. This is especially critical during an outage where engineers will need to quickly read metrics that in many cases were only written only a few seconds ago that they wouldn't otherwise read and are not configured as part of an ongoing alert. Additionally, the alerting infrastructure that runs on top of the TSDB is configured with 10k+ queries that run every minute. So even at 2% read, you're querying them over and over and over again because you always need the latest data plus whatever trailing data is needed to fulfill the alerting needs (trailing 10m, hour, day, month, etc). This also makes it a particularly hard caching problem. I can't speak to the state of GCP at Twitter. I know they had started to migrate some things but when I was there it was all colo. Could they use BQ? Sure it could probably be tooled with partitions/caching/etc to work I guess, but at the end of the day it's not what BQ is designed for which is data warehousing and BI. Could they have used something that wasn't a custom built in-house database? Yea almost certainly (No secret that there is some NIH going on at Twitter well before this) This happens alll the time on HN so I'm not here to call you out specifically...but it's really easy to read a company's blog post about their infrastructure decisions and immediately scoff and say "psh why didn't they just use X?"...as if you now have the same context that the engineering team has by reading a single blog post.
MoOmer 6y ago
Certainly depends on how you use it. If you plan accordingly and the data fits a compatible partitioning strategy, things work out well and costs can be manageable/predictable.

Bigquery does not have anything close to real time query support like you want/need for observability data. And plus the cost is more like one engineer per query. That shit is crazy expensive.

manigandham 6y ago

What do you mean by real-time? BQ can scan a petabyte in less than a minute, but partitioning and clustering will get most simple queries back in a few seconds. If you mean data freshness than BQ supports streaming ingest and will show those rows in the next query. SQL also lets you do a lot of aggregations in the database instead of pulling back raw metrics. And BQ has flat-rate pricing and discounts for large clients.

buremba 6y ago
Try to run a few petabyte-scale queries and you will be surprised when you see the invoice. If you switch to flat-rate pricing, don't expect the latency that you mentioned.
- manigandham 6y ago
  You dont need to scan the whole dataset. Partitioning + clustering, especially for time-series, should be very efficient. BQ flat-rate pricing starts at 10k/month for a fixed number of processing slots. You can buy more slots to fit your query load. Anyways the point is that it's well within the financial means of Twitter, especially when compared to developing and operating this proprietary system.
  - buremba 6y ago
    If I'm not mistaken Twitter is mostly on GCP so they could just go with BigQuery instead of developing an in-house solution. We probably need to ask them in order to get a proper answer. :)
rco8786 6y ago
Less than a minute is not real time. Something like < 100ms for most queries would be real time. And then do that 10s of 1000s of times per minute, every minute, 24 hours a day. You can see how the query costs add up... I know it's easy to come on HN and play armchair architect but this is one area where you are just incorrect in your assumptions. Did Twitter need to build a(nother) in-house database? I don't know for sure. But I do know that BQ is not well suited to observability use cases.
- manigandham 6y ago
  As I said, that's for scanning a petabyte. Selective queries would be much smaller and faster, costs can be handled with flat-rate pricing, and there's the BI Engine (in-memory cache) feature too. The details in the article (1.5PB logical data, 3x replication factor, only 2% of metrics ever read) and the fact that Twitter already runs infra in Google Cloud seem to align well with BQ in my experience (10+ years in adtech building even more complex backends). Please tell me what assumptions are incorrect so I can reconsider.
  - rco8786 6y ago
    My experience is ~2.5 years as a senior engineer on the Observability team at Twitter (from 2013-2015). I was part of the migration to Manhattan (mentioned in the post) from Cassandra, new alerting infra, query language design, among many other things. Your adtech experience should make you particularly sensitive to query latencies, so I find it interesting that you're glossing over that. Yes, only 2% of metrics ever read. That was the same back then too. The kicker is that you can't be sure which will be read, so the systems are built to be able to read any of the metrics with the same SLA. This is especially critical during an outage where engineers will need to quickly read metrics that in many cases were only written only a few seconds ago that they wouldn't otherwise read and are not configured as part of an ongoing alert. Additionally, the alerting infrastructure that runs on top of the TSDB is configured with 10k+ queries that run every minute. So even at 2% read, you're querying them over and over and over again because you always need the latest data plus whatever trailing data is needed to fulfill the alerting needs (trailing 10m, hour, day, month, etc). This also makes it a particularly hard caching problem. I can't speak to the state of GCP at Twitter. I know they had started to migrate some things but when I was there it was all colo. Could they use BQ? Sure it could probably be tooled with partitions/caching/etc to work I guess, but at the end of the day it's not what BQ is designed for which is data warehousing and BI. Could they have used something that wasn't a custom built in-house database? Yea almost certainly (No secret that there is some NIH going on at Twitter well before this) This happens alll the time on HN so I'm not here to call you out specifically...but it's really easy to read a company's blog post about their infrastructure decisions and immediately scoff and say "psh why didn't they just use X?"...as if you now have the same context that the engineering team has by reading a single blog post.

MoOmer 6y ago

Certainly depends on how you use it. If you plan accordingly and the data fits a compatible partitioning strategy, things work out well and costs can be manageable/predictable.