10 ms·
Amazon Timestream – Fast, scalable, fully managed time series database
- brian_herman__ 8y agoI wonder how this compares to KDB
- hellomichibye 8y agoSame for me:)
- nslocum 8y agoSince kdb is SO good with time series data, I would need a lot of tech details to even know if AWS is worth testing. It's a LOT of work to test a time series db at scale.
- SEJeff 8y agoThe core of Amazon's timeseries db likely doesn't fit into a CPU's L1 cache. It does with KDB :)
- cthalupa 8y ago>The core of Amazon's timeseries db likely doesn't fit into a CPU's L1 cache. It does with KDB :) Does it? https://kx.com/discover/in-memory-computing/ https://kx.com/discover/in-memory-computing/ seems to indicate that it takes up ~600 Kb (I'm not sure if this is bits or bytes, but even if it's bits, that turns into 75KB) L1 cache is per core. Skylake Xeons have a 64KB cache per core, 32KB for data and 32KB for instructions. Even with an even split there, you're not fitting 75KB (or 600KB) into the L1 cache. Bits would be a weird measurement to use when talking about memory utilization, so I'm pretty sure that it's 600 kilobytes. You're not anywhere close to fitting that into the L1 cache. L2 cache, sure. But you get the relatively spacious 1 megabyte for L2. I'm also not sure that the "core" fitting into the CPU cache is particularly meaningful for performance anyway. it doesn't say anything about how much outside of the core gets used, how big the working set size is for your workload, how much meaningful work is done on that working set of data, etc. If you're frequently using parts of the software that don't fit in the cache, or getting evicted from it for other code, or your working set of data doesn't fit in the cache and you're constantly going to main memory for the data you're working on, the "core" fitting in L1 cache (or L2 cache, which looks more realistic) is going to be basically meaningless.
- SEJeff 8y agoGah, I meant L2 cache, but was being entirely too smug. I remember a presentation a KX rep gave at our office a few jobs ago where this was one of their bullet points. I found it amusing, and a bit odd.
- cthalupa 8y agoGotcha! Definitely an interesting marketing point. I probably would have had the same reaction :)
- nimbius 8y agojesus christ six amazon articles in a day? AWS is undeniably the body of christ for HN but am i missing something? FSX, blockchain, timestream, Graviton, ground station, and cloudwatch... all of these articles are advertisements for mundane shit.
- meritt 8y agohttps://reinvent.awsevents.com/ https://reinvent.awsevents.com/ It happens every year during Google, Apple, Amazon, and Facebook events.
- nimbius 8y agothanks. Just glanced at the shop calendar and i guess the holidays are right around the corner too.
- danso 8y agoIt's AWS re:Invent day/week. I've never been, but I get the impression that it's like Apple's keynote, or Google I/O, in which big product announcements are made. On those days, you'll see multiple submissions about the respective conferences too.
- cheeze 8y agoThis is exactly what re:Invent is. Most teams dream of launching a new AWS product at re:Invent (and not missing their date and launching at a later time)
- danso 8y agoIt'd be an interesting blog post topic to look at how AWS:Invent (or I/O, or F8, etc) product threads on HN compare to actual product impact. I remember Rekognition getting decent discussion 2 years ago [0], but not along the angles or magnitude of how Rekognition is usually discussed in recent months. OTOH, other things I've been interested in as a data geek, I've barely heard of since reading about them on HN -- e.g. Athena [1] [0] https://news.ycombinator.com/item?id=13072956 https://news.ycombinator.com/item?id=13072956 [1] https://news.ycombinator.com/item?id=13072245 https://news.ycombinator.com/item?id=13072245
- addisonj 8y agoNice to see, this has felt like a gap in cloud offerings for a while... and the open source options have difficulties. From the little that was said, going to guess this uses something like Beringei (https://code.fb.com/core-data/beringei-a-high-performance-time-series-storage-engine/ https://code.fb.com/core-data/beringei-a-high-performance-ti...) under the hood
- probdist 8y agoSeems positioned to compete with Azure Data Explorer (MSFT's log/time series optimized service). I know Azure runs a lot of services on top of Data Explorer (previously called Kusto) I wonder if this is a true internal battle tested product or a me-too offering.
- adeal 8y agoI use Kusto daily and it is by far my favorite dev tool. It's incredibly fast and actually fun to use. I'm interested to compare it to Timeseries. Kusto is still the name of the query language and the desktop application (Kusto.Explorer), the service was just renamed to Azure Data Explorer.
- nkassis 8y agoI might be mistaken but wouldn't Data Explorer be more similar to AWS CloudWatch which has been around for a long time.
- probdist 8y agoAzure Data Explorer/ Kusto is more of a database that is optimized for the log use case than a service. There is a front end tool and a lot of the use-cases are around log management, but it is database you can do general SQL or KQL things with. Time series is one of the core use-cases for it also but it has less marketing around it.
- spullara 8y agoSounds more like AWS CloudWatch Log Insights launched yesterday.
- tjungblut 8y agoKusto is architecturally closer to Dremel (or BigQuery). It's a columnar compressed datastore with a nice query language. Not the most efficient way to store and query timeseries data though. Back then (internally) we actually had a lot of issues with ingesting and querying time series data at scale.
- axus 8y agoA place to put the timestamped data they download from yesterday's Amazon Ground Station.
- jopsen 8y agowhere is the docs?
- erikcw 8y agoSeems like this could be a great remote storage backend for Prometheus.
- nabadu87 8y agoOh yeah! Would that need a custom storage peovider in Prometheus?
- erikcw 8y agoYes [0]. I haven't had time to fully look into the details. But from looking at the existing integration -- it would be fantastically convenient to have something cloud native [1]. [0] https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations https://prometheus.io/docs/prometheus/latest/storage/#remote... [1] https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage https://prometheus.io/docs/operating/integrations/#remote-en...
- booleandilemma 8y agoThere are at least 7 Amazon-related stories on the HN front page right now, what’s going on?
- bruth 8y agoAWS re:Invent
- samstave 8y agoMerry christmas?
- deleted 8y ago[deleted]
- deleted 8y ago[deleted]
- dang 8y agohttps://hn.algolia.com/?sort=byDate&dateRange=all&type=comment&storyText=false&prefix=false&page=0&query=by:dang%20big%20tech%20conference https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
- temuze 8y agoHonest question: when dealing with time-series data, do you actually need every data point? Is that level of granularity really necessary? IMO, it makes way more sense to decide the aggregations you want ahead of time (e.g. "SELECT customer, sum(value) FROM purchases GROUP BY customer"). That way, you deal with substantially less data and everything becomes a whole lot simpler.
- bruth 8y agoReally depends on the use case. Working in healthcare, vital signs can be modeled as time series points, but are lower frequency than, say, metrics from servers. However we want to store every point so a spike is not missed. One could argue an unsustained spike is noise, but in the healthcare domain there may be a correlation with some external event (the purpose is surprised and their heart rate spikes).
- have_faith 8y agoThe clever thing to do in this scenario would be to keep every spike but delete all the data between similar data points after storing. So you get low granularity for identical/nearly-the-same data points and high granularity when something interesting happens. I don't have any experience with time-series data so maybe this is commonplace.
- lozenge 8y agoThat would be impossible to run any new analyses on. What some would do is record in blocks where every point after the earliest is stored as a delta. Then each block is more compressible as it contains a lot of 0s.
- deleted 8y ago[deleted]
- ilaksh 8y agoSure you can do if you're really sure that you won't need to group by something else later. You wouldn't want to store more granularity than necessary but you can't go back in time to get a data point you didn't store.
- samstave 8y agoSo what will this compare wrt boundary, signalfx, stackdriver, etc types of previous services... Ill have to go look into this, because if aws historic pricing for any large volume stream, quickly becomes untennable. Its very easy to have gobs and gobs of time series points... aws might make using this way too expensive for anything at relative scale for a small startup?
- spullara 8y agoIt appears that ingestion alone is more expensive than the commercial metric services. Might not matter for small scale.
- citilife 8y agoAt my day job, I build a lot of machine learning systems that require data to be fed in a time series manner[1]. Often this means building systems to analyze terabytes of logs [semi]-realtime. All I have to say is - thank god! This is going to make my job a lot easier, and likely empower us to remove our current infrastructure setup. I know at one point we actually considered building our own time series database. Instead, we ended up utilizing a Kafka queue with an SQL based backend after we parsed and paired down the data, because it was the only one quick enough to do the queries. Should make a lot of the modeling I've worked on a bit easier[1]. [1] https://medium.com/capital-one-tech/batch-and-streaming-in-the-world-of-data-science-and-data-engineering-2cc029cdf554 https://medium.com/capital-one-tech/batch-and-streaming-in-t...
- haggy 8y agoI wouldn't jump the gun on this. I've been working within Amazon cloud for years and every year they make massive claims about new services at re:invent. Not saying this isn't going to be a good product, just saying it will probably take a while to be as useful as you're hoping.
- gnahckire 8y agoI agree w/ this. They make insane promises but, the promises don't like up to expectations. Kinesis analytics for example, can aggregate data across a time window (sliding window) from a stream (Kinesis). A huge issue that isn't document or stated is that when kinesis analytics restarts due to the process dies (being migrated, binpacked, etc.) the ENTIRE time window has to get re-aggregated. So your count drops to 0. Really unacceptable if you're using it to generate KPIs which you alert on. We ended up switching to a system which pushes the stream data into influx and running the aggregations there via queries. Dealing w/ AWS during this entire process was a huge pain.
- kennu 8y agoSimilar examples have been Fargate and Lambda, which have surprisingly long cold start times depending on runtimes and VPC configurations. IMHO it is a big part of AWS expertise to know these things and be able to choose the right services not just according to the marketing brochures, but according to how they really work for each use case. Having said that, I'm glad to now have learned about that Kinesis Analytics restart issue.
- willlll 8y agoI'm actually impressed at how incredibly expensive they made this. $0.50 per million 1KB writes, which is 20x what aurora charges, since aurora allows 8KB writes. And Aurora is already expensive if you actually read/write to it.
- ravedave5 8y agoI get the feeling this is for important data (banking etc) so I have a feeling this is 200x cheaper than whatever else is available.
- tonyedgecombe 8y agoThat's not what is says in the release: "With Timestream, you can easily store and analyze log data for DevOps, sensor data for IoT applications, and industrial telemetry data for equipment maintenance."
- coder543 8y agoI really don't think it's 20x the cost of Aurora in general, considering that Aurora costs are not that simple, but I don't have time to run the numbers, so let's go with that for a moment. Do you really think Amazon would introduce a product that's 20x as expensive if they didn't know there was a market for it?
- barake 8y agoThe pricing lines up with CloudWatch Logs, 50 cents/GB, 3 cents/GB/month. Curious to see what the query language is for this, wonder if they're just exposing the backing store for CloudWatch as a service now.
- abalone 8y ago> $0.50 per million 1KB writes, which is 20x what aurora charges, since aurora allows 8KB writes. That's a weird comparison. 20x is only true if you write 8KB with every entry, and you haven't included the storage and instance savings. It's not hard to come up with suboptimal scenarios where this is more expensive, but that's missing the point. It's optimized for a specific kind of usage pattern.
- taf2 8y agoI couldn’t tell from the page is this SQL based similar to timescale or a more similar to influxdb?
- sciurus 8y agoIt's got to be a rough day for the team at https://www.influxdata.com/ https://www.influxdata.com/ . This could become serious competition for their InfluxCloud hosted offering.
- lozaning 8y agoAmazon is the best partner until you prove worth cannibalizing.
- dominotw 8y agoand for timescaledb, streamlio, two sigma, kdb+ , quasardb ( and perhaps pipelinedb)
- stingraycharles 8y agoQuasarDB employee chiming in - we’re not actually worried, our clients are typically operating at a scale that would make costs very prohibitive for this AWS product. Having said that, I can definitely see this be an interesting product for people doing less than 10k inserts per second.
- dominotw 8y ago> make costs very prohibitive for this AWS product. Competing with AWS on just cost sounds worrying to me.
- StreamBright 8y agoActually not really. We have migrated several customers to AWS and our experience with AWS services are kind of all over the place. S3 is super cheap and reliable and there is almost nothing that beats that but for example CloudWatch is extremely expensive for a large scale operation that is easy to beat with custom software (like Prometheus or something similar). I guess even Datadog would beat them (and they have much more advanced features and integrations). This means that there are multiple software vendors can exist in the same space even when Amazon has an offering in that space.
- dschuetz 8y agoAnother Amazon Ad?
- dang 8y agoMost big tech companies bundle their announcements on their annual conference day. It happens several times a year and these same complaints show up every time, but spamming the threads with them is excessive. https://hn.algolia.com/?sort=byDate&dateRange=all&type=comment&storyText=false&prefix=false&page=0&query=by:dang%20big%20tech%20cos https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
- inoiox 8y agoThere have been a lot of amazon links this week
- superkuh 8y agoA quick look at the Hacker News frontpage shows a bit of a problem, 1. Amazon Timestream (amazon.com) 3. Amazon Quantum Ledger Database (amazon.com) 8. Amazon FSx for Lustre (amazon.com) 13. AWS DynamoDB On-Demand (amazon.com) 14. Amazon's homegrown Graviton processor was very nearly an AMD Arm CPU (theregister.co.uk) 21. Building an Alexa-Powered Electric Blanket (shkspr.mobi) 30. Amazon FSx for Windows File Server (amazon.com)
- dang 8y agohttps://hn.algolia.com/?sort=byDate&dateRange=all&type=comment&storyText=false&prefix=false&page=0&query=by:dang%20big%20tech%20conference https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
- superkuh 8y agoAh, sorry. I should've known you'd already be on it. Thanks.
- vemv 8y agoWhether we are fans or not, keeping up with (and discussing) new AWS offerings is extremely relevant for practitioners.
- whoisjuan 8y agoAWS Re:Invent is happening in Las Vegas this week. AWS is launching new services all this week. People here are interested in cloud services, so it makes sense that people share and upvote those announcements. I don't see the problem that you're talking about.
- MagicPropmaker 8y agoWe had applications where we were tracking guests in a venue through various means. We tried a number of queuing systems to manage the flood of events, but they'd all fall over. I'll love to run my old "venue simulator" through this and see if it can stand up to actual guest load as they walk around, ride, purchase things.
- mLuby 8y agoI count 7 separate Amazon posts on the front of HN. Is this some conspiracy? #NotAmused #ShouldBeBundled
- twistedpair 8y agoThe conspiracy is called re:Invent
- coredog64 8y agoI'm wondering if this shares any technology with the CloudWatch metrics backend. They've been making improvements there all year, and most of them generally align with what's announced here. CloudWatch metrics are also very expensive for what you get, so that's another similarity to Timestream ;)
- sciurus 8y agoThis is not cheap for the "DevOps" use case. Imagine you have 1000 servers submitting data to 100 timeseries each minute. That's 100,000 writes a minute (unless they support batch writes across series) At $0.50 per million writes that's $72 a day or $26k a year. Now imagine you want to alert on that data. Say you have 100 monitors that each evaluate 1GB of data once a minute. At $10 per TB of data scanned, that's $1,440 a day or $525k a year!
- Alex3917 8y agoBetween RDS and these proprietary database products, AWS is now its own biggest competitor. That's potentially fine, but there is an inherent conflict of interest there and that needs to be properly managed, and while that may be happening it didn't come across in the keynote. The only way I can have trust in Amazon's proprietary products is if RDS continues to get less expensive every year, since that is effectively the BATNA to these new products. It's been a while now since the last RDS cost reductions, and unless we continue to see more of those it's hard to have confidence that Amazon will continue to treat their customers of these new proprietary services fairly over the long term.
- jaxxstorm 8y agoWell, that depends on what you consider cheap. Hiring someone to manage a time series system like graphite or prometheus is going to cost you a whole lot more than $26k a year
- deleted 8y ago[deleted]
- sciurus 8y agoYou're ignoring half of my example scenario. $26k writing the data, $525k for querying it just for alerting, plus whatever it costs to store and to query ad-hoc. That's over half a million dollars. Even if you hire someone for $250k, you can self-host your time series system for cheaper than that. Self-hosting isn't the only option though. For example, that hypothetical 1000 server scenario would cost $180k a year at list pricing on Datadog or SignalFX.
- Tehnix 8y agoQuite excited for this! We have currently been experimenting with using DynamoDB, and managing our own rollups of our incoming data (previously on an RDS, which is not a good choice for this kind of data). --- I've seen a lot of people complain about pricing, so I thought I'd share a little why we are excited about this: We have approximately 280 devices out, monitoring production lines, sending aggregated data every 5 seconds, via MQTT to AWS IoT. The average messages published that we see is around ~2 million a day (equipment is often turned off, when not producing). The packet size is very small, and highly compressable, each below 1KB, but let's just make it 1KB. We then currently funnel this data into Lambda, which processes it, and puts it into DynamoDB and handles rollups. The costs of that whole thing is approximately $20 a day (IoT, DynamoDB, Lambda and X-Ray), with Lambda+DynamoDB making up $17 of that cost. Finally, our users look at this data, live, on dashboards, usually looking at the last 8 hours of data for a specific device. Let's throw around that there will be 10,000 queries each day, looking at the data of the day (2GB/day / 280devices = 0.007142857 GB/device/day). --- Now, running the same numbers on the AWS Timestream pricing[0] (daily cost): - Writes: 2million * $0.5/million = $1 - Memory store: 2 GB * $0.036 = $0.072 - SSD store: (2GB * 7days) * $0.01 (GB/day) * 7days = $0.98 - Magnetic store: (2 GB * 30 days) * $0.03 (GB/month) = $1.8 - Query: 10,0000 queries * 0.007142857GB/device/day --> 71GB = free until day 14, where it'll cost $10, so $20 a month. Giving us: $1 + $0.072 + $0.98 + $1.8 + ($20/30) = $4.5/day. From these (very) quick calculations, this means we could lower our cost from ~$20/day to ~$4.5/day. And that's not even taking into account that it removes our need to create/maintain our own custom solution. I am probably missing some details, but it does look bright! [0] https://aws.amazon.com/timestream/pricing/ https://aws.amazon.com/timestream/pricing/
- deleted 8y ago[deleted]
- plasma 8y agoThe financial read cost of this database makes it practically unusable for customer facing dashboards, disappointing.
- brootstrap 8y agoBeen searching for years to a good alternative to postgres for storing gobs of weather timeseries data. So far we have been running postgres system for many years in production and have hired multiple contractors to implement a 'real timeseries solution'. All of which have been utter shit and complete failures. The AWS services are expensive as all hell. With a little bit of imagination we created a unique schema for timeseries data that doesnt require terabytes of space, and processes billions of data points a day, and has blazing fast queries into said data.
- DelaneyM 8y agoI once handled gobs of web tracking data in Cassandra, using Hadoop over Cassandra to build queryable rollups in MySQL and Pig for on the fly analysis Neither tech works on its own, but together (substituting Cassandra columns for hdfs) was magic for the specific data configuration & use case.
- jws 8y agoI moved a decade's worth of weather time series data from well indexed Sqlite to InfluxDB and was nothing but pleased. It ended up taking an order of magnitude less storage and so much faster to query that I didn't even bother to benchmark it. You can probably write a simple query to your Postgres database to cough out the text file to load InfluxDB to see how it works for you. Then it comes down to how easy it is to replace your query and insert functions… So it's all easy except for the hard part.
- porker 8y agoDid you try https://www.timescale.com/ https://www.timescale.com/?
- mharroun 8y agoThis is looking like a managed druid... that would be very nice to have.
- tjholowaychuk 8y agoAnyone know if this is what CloudWatch Insights uses? If so, it doesn't even come close to competing with Elasticsearch performance (with a tiny cluster), it seemed quite slow.