5 ms·

At my day job, I build a lot of machine learning systems that require data to be fed in a time series manner[1]. Often this means building systems to analyze t

At my day job, I build a lot of machine learning systems that require data to be fed in a time series manner[1]. Often this means building systems to analyze terabytes of logs [semi]-realtime. All I have to say is - thank god! This is going to make my job a lot easier, and likely empower us to remove our current infrastructure setup. I know at one point we actually considered building our own time series database. Instead, we ended up utilizing a Kafka queue with an SQL based backend after we parsed and paired down the data, because it was the only one quick enough to do the queries. Should make a lot of the modeling I've worked on a bit easier[1]. [1] https://medium.com/capital-one-tech/batch-and-streaming-in-the-world-of-data-science-and-data-engineering-2cc029cdf554 https://medium.com/capital-one-tech/batch-and-streaming-in-t...

haggy 8y ago
I wouldn't jump the gun on this. I've been working within Amazon cloud for years and every year they make massive claims about new services at re:invent. Not saying this isn't going to be a good product, just saying it will probably take a while to be as useful as you're hoping.
- gnahckire 8y ago
  I agree w/ this. They make insane promises but, the promises don't like up to expectations. Kinesis analytics for example, can aggregate data across a time window (sliding window) from a stream (Kinesis). A huge issue that isn't document or stated is that when kinesis analytics restarts due to the process dies (being migrated, binpacked, etc.) the ENTIRE time window has to get re-aggregated. So your count drops to 0. Really unacceptable if you're using it to generate KPIs which you alert on. We ended up switching to a system which pushes the stream data into influx and running the aggregations there via queries. Dealing w/ AWS during this entire process was a huge pain.
  - kennu 8y ago
    Similar examples have been Fargate and Lambda, which have surprisingly long cold start times depending on runtimes and VPC configurations. IMHO it is a big part of AWS expertise to know these things and be able to choose the right services not just according to the marketing brochures, but according to how they really work for each use case. Having said that, I'm glad to now have learned about that Kinesis Analytics restart issue.
lykr0n 8y ago
Have you looked at Clickhouse for timeseries data? It's the one database I've found that can scale and can query in near-realtime. I've loaded a 100 Billion Rows in into a 5 shard database and can do full queries across the whole dataset in under 10 seconds. It also natively consumes multiple kafka topics.
- haggy 8y ago
  > I've loaded a 100 Billion Rows Have you done any load tests that would more closely mirror a production environment such as performing queries while clickhouse is handling a heavy insert load?
  - lykr0n 8y ago
    I'm working on developing benchmarking tools for internal testing, but both Yandex and CloudFlare use Clickhouse for realtime querying. I'm still in development phase for my product, but I'll make sure to post information & results when we launch here. https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/ https://blog.cloudflare.com/http-analytics-for-6m-requests-p... But I've spent a long time looking at the various solutions out there, and while ClickHouse is not perfect, I think it's the best multi-purpose database out there for large volumes of data. TimescaleDB is another one, but until they get sharding it's dead on arrival.
    - haggy 8y ago
      Very cool, I'll check this out!
      - lykr0n 8y ago
        It's a quirky piece of software and has limitations that need to be considered when standing up a production cluster- such as you cannot reshard it currently. If you have a 3 node cluster, it's messy and requires downtime to add another node.
        8y ago
mark_l_watson 8y ago
Keira, nice article. It will be a long time before we are allowed to use this new service in C1
mathattack 8y ago
Have you considered columnal databases like 1010data?
- 3rdAccount 8y ago
  I recall why this looks familiar. Their chief scientist put up a job posting on the subreddit for array languages. It sounds like they know how to use some pretty powerful tools. https://www.reddit.com/r/apljk/comments/42uf2f/not_exactly_a_k_job_ad_but_k3/ https://www.reddit.com/r/apljk/comments/42uf2f/not_exactly_a...

haggy 8y ago

I wouldn't jump the gun on this. I've been working within Amazon cloud for years and every year they make massive claims about new services at re:invent. Not saying this isn't going to be a good product, just saying it will probably take a while to be as useful as you're hoping.

gnahckire 8y ago
I agree w/ this. They make insane promises but, the promises don't like up to expectations. Kinesis analytics for example, can aggregate data across a time window (sliding window) from a stream (Kinesis). A huge issue that isn't document or stated is that when kinesis analytics restarts due to the process dies (being migrated, binpacked, etc.) the ENTIRE time window has to get re-aggregated. So your count drops to 0. Really unacceptable if you're using it to generate KPIs which you alert on. We ended up switching to a system which pushes the stream data into influx and running the aggregations there via queries. Dealing w/ AWS during this entire process was a huge pain.
- kennu 8y ago
  Similar examples have been Fargate and Lambda, which have surprisingly long cold start times depending on runtimes and VPC configurations. IMHO it is a big part of AWS expertise to know these things and be able to choose the right services not just according to the marketing brochures, but according to how they really work for each use case. Having said that, I'm glad to now have learned about that Kinesis Analytics restart issue.

lykr0n 8y ago

Have you looked at Clickhouse for timeseries data? It's the one database I've found that can scale and can query in near-realtime. I've loaded a 100 Billion Rows in into a 5 shard database and can do full queries across the whole dataset in under 10 seconds. It also natively consumes multiple kafka topics.

haggy 8y ago
> I've loaded a 100 Billion Rows Have you done any load tests that would more closely mirror a production environment such as performing queries while clickhouse is handling a heavy insert load?
- lykr0n 8y ago
  I'm working on developing benchmarking tools for internal testing, but both Yandex and CloudFlare use Clickhouse for realtime querying. I'm still in development phase for my product, but I'll make sure to post information & results when we launch here. https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/ https://blog.cloudflare.com/http-analytics-for-6m-requests-p... But I've spent a long time looking at the various solutions out there, and while ClickHouse is not perfect, I think it's the best multi-purpose database out there for large volumes of data. TimescaleDB is another one, but until they get sharding it's dead on arrival.
  - haggy 8y ago
    Very cool, I'll check this out!
    - lykr0n 8y ago
      It's a quirky piece of software and has limitations that need to be considered when standing up a production cluster- such as you cannot reshard it currently. If you have a 3 node cluster, it's messy and requires downtime to add another node.
      - 8y ago

mark_l_watson 8y ago

Keira, nice article. It will be a long time before we are allowed to use this new service in C1

mathattack 8y ago

Have you considered columnal databases like 1010data?

3rdAccount 8y ago
I recall why this looks familiar. Their chief scientist put up a job posting on the subreddit for array languages. It sounds like they know how to use some pretty powerful tools. https://www.reddit.com/r/apljk/comments/42uf2f/not_exactly_a_k_job_ad_but_k3/ https://www.reddit.com/r/apljk/comments/42uf2f/not_exactly_a...