2 ms·

Is this anything else than AWS Athena with extra steps? My team ingests petabytes of data each day in S3 that is then queriable in Athena, and it supports all

Is this anything else than AWS Athena with extra steps? My team ingests petabytes of data each day in S3 that is then queriable in Athena, and it supports all the same features that are mentioned here, such as interfacing with other types of datasources like an existing PostgreSQL database.

Topgamer7 4y ago
Probably not. Other than its slightly abstracted away from being solely ingrained in the AWS ecosystem. It shouldn't be hard to use another block level storage.
mfreed 4y ago
Do you use AWS Athena as the "database/serving layer" for customer-facing applications where performance matters? Our experience is that isn't a common use case for Athena, while it is the primary use case for TimescaleDB.
- iLoveOncall 4y ago
  No, exposing the data to our customers through Athena is the use case. So performance matters in the sense that querying data is our core offer, but we're talking about multiple seconds here, not milliseconds. I think the distinction makes sense then, thanks.
lph 4y ago
Seamless queries that span different storage layers is a pretty big deal. This is like having a table with new data in RDS and old data in Athena. The potential downside is unpredictable query performance. For example, suppose you have a query that calculates daily statistics from your time series data. The query takes 1 second to execute when you run it for yesterday's data, but 1 minute to run for a day six months in the past, because you've inadvertently shifted your work onto the slower storage layer.
rocmcd 4y ago
What does the response time look like for those queries? And what is the cost? I haven't heard great things about Athena, but I'd be curious to hear more about what works.
- iLoveOncall 4y ago
  I don't want to say too much, but the response time is not suitable to a real time application, queries will all take at least around a second to complete. But what users are querying explains why it takes a long time. Again, we're talking about a petabyte of data a day. Then in terms of cost it's around $0.5 per query maybe? The average is a very bad number because most queries will be something like $0.0001, but then some will be hundreds of dollars. But that's the only number I have off the top of my head.
gayyappan 4y ago
With the data tiering feature, data moves from EBS -> S3. The database is fully aware of all the metadata and the location of any data that it manages. So a user can query any data that lives on storage managed by the Timescale database. S3 is just another storage device for the db. Athena is not a database. Athena is query service for data lakes in S3 (e.g. load into S3 is not managed by Athena, it is not protected by any DB transactional semantics). Athena needs metadata info for the data in S3. (Timescale engineer)

Topgamer7 4y ago

Probably not. Other than its slightly abstracted away from being solely ingrained in the AWS ecosystem. It shouldn't be hard to use another block level storage.

mfreed 4y ago

Do you use AWS Athena as the "database/serving layer" for customer-facing applications where performance matters? Our experience is that isn't a common use case for Athena, while it is the primary use case for TimescaleDB.

iLoveOncall 4y ago
No, exposing the data to our customers through Athena is the use case. So performance matters in the sense that querying data is our core offer, but we're talking about multiple seconds here, not milliseconds. I think the distinction makes sense then, thanks.

lph 4y ago

Seamless queries that span different storage layers is a pretty big deal. This is like having a table with new data in RDS and old data in Athena. The potential downside is unpredictable query performance. For example, suppose you have a query that calculates daily statistics from your time series data. The query takes 1 second to execute when you run it for yesterday's data, but 1 minute to run for a day six months in the past, because you've inadvertently shifted your work onto the slower storage layer.

rocmcd 4y ago

What does the response time look like for those queries? And what is the cost? I haven't heard great things about Athena, but I'd be curious to hear more about what works.

iLoveOncall 4y ago
I don't want to say too much, but the response time is not suitable to a real time application, queries will all take at least around a second to complete. But what users are querying explains why it takes a long time. Again, we're talking about a petabyte of data a day. Then in terms of cost it's around $0.5 per query maybe? The average is a very bad number because most queries will be something like $0.0001, but then some will be hundreds of dollars. But that's the only number I have off the top of my head.

gayyappan 4y ago

With the data tiering feature, data moves from EBS -> S3. The database is fully aware of all the metadata and the location of any data that it manages. So a user can query any data that lives on storage managed by the Timescale database. S3 is just another storage device for the db. Athena is not a database. Athena is query service for data lakes in S3 (e.g. load into S3 is not managed by Athena, it is not protected by any DB transactional semantics). Athena needs metadata info for the data in S3. (Timescale engineer)