3 ms·
https://github.com/Logflare/logflare/tree/staging https://github.com/Logflare/logflare/tree/staging > Simply provide your BigQuery credentials and we stream lo
by simonw 4y ago
https://github.com/Logflare/logflare/tree/staging https://github.com/Logflare/logflare/tree/staging
> Simply provide your BigQuery credentials and we stream logs into your BigQuery table while automatically managing the schema
I didn't know BigQuery was capable of accepting streaming log data - in my mental model of the world it was the kind of database that you update using the occasional batch job, not from a streaming source of data.
Looks like that's the tabledata.insertAll method which has been around for quite a few years - though it's now called the "legacy streaming API" on https://cloud.google.com/bigquery/docs/streaming-data-into-bigquery https://cloud.google.com/bigquery/docs/streaming-data-into-b... which suggests using the more recent Storage Write API instead: https://cloud.google.com/bigquery/docs/write-api https://cloud.google.com/bigquery/docs/write-api
- martin_ 4y agoIt's pretty awesome & very cost effective! When I was at Twilio we streamed logstash/fluentd (forget which) -> kinesis -> BigQuery and it worked great - certainly better than the days we were trying to manage ES ourselves
- harpratap 4y agoWere people using it as an actual ES replacement for logs? fluentd -> GCP log sink -> BQ can get awful if you need to work with it on daily basis
- chasers 4y agoCurious how was it awful for you? Really have not regretted any part of using BigQuery.
- swyx 4y agois kinesis in the middle for more durability/being able to handle spikes? when and how does one make that sort of decision to insert kinesis into things? (esp since you were gonna get the data from aws -> gcp so you dont save anything staying within aws anyway)