12 ms·
Amazon Aurora Now Available
- jpgvm 11y agoMentioned this on Twitter to the Aurora program manager but it would be awesome to see a PostgreSQL compatible frontend. Still though, awesome achievement to build a competitive database engine. :)
- nnx 11y agoOh yes, please. That would greatly simplify tooling and integration with RedShift (which already uses PostgreSQL frontend/protocol).
- georgewfraser 11y agoIf you want to synchronize your postgres db (and a lot of other things) to redshift, we do it as a service at fivetran.com
- stephenr 11y agoIsn't Aurora simply Amazon's private fork of MySQL, hosted for you?
- boulos 11y agoIt certainly seems to be a MySQL 5.6 fork (or at least backend ala InnoDB or similar), as the docs explicitly call out MySQL 5.6 compatibility. I feel like I wouldn't say "simply" though; that makes it seem like some guys at Amazon checked out the MySQL source, deleted some stuff and then released this.
- istvan__ 11y agoIt would have been extremely difficult because MySQL community edition is GPL and Amazon does not allow GPL code to be modified by the internal teams as far as I know. They might have changed their policy though...
- frankchn 11y agoUnlikely, because they already have a hosted MySQL 5.6 product in RDS. If anything, this is a MySQL 5.6 frontend + query processing engine with a new storage layer.
- reilly3000 11y agoThe already have that in RDS. This is a da add of their own design. They were pragmatic enough to use the query language that most devs already know and trust, but how they pull off reads, writes, and sharing at that scale must have fundamentally different internals than a single MySQL instance.
- olalonde 11y ago> but it would be awesome to see a PostgreSQL compatible frontend Why not just use PostgreSQL directly?
- jordanthoms 11y agoAurora has a bunch of nice features WRT automatic failover, redundancy, and automatically scaling storage as you push more data into the DB.
- josh2600 11y agoDoes Aurora do anything other than detect a heartbeat failure, shoot the node in the head and bring it up on another box WRT failover?
- awgupta 11y agoThat is the basic idea on such things, though distributed protocols become complex when you have to consider split-brain, multiple concurrent failures, etc which all can occur in large-scale events. Aurora leverages Amazon RDS Multi-AZ in this area. Their implementation has been battle-tested over many years and many many db instances.
- brightball 11y agoI'm really wondering if something very close will be possible soon with AWS EFS.
- mathnode 11y agoA Postgres to MySQL client mapping middleware is probably the holy grail? I think it's been done in the JDBC space.
- timdorr 11y agoYou can get this essentially via mysql_fdw: https://github.com/EnterpriseDB/mysql_fdw https://github.com/EnterpriseDB/mysql_fdw
- ahachete 11y agoThere's no such concept of Storage Engines/Plugins in PostgreSQL. So it looks more unlikely this could happen in short time.
- timdorr 11y agoYes, there are: http://www.postgresql.org/docs/9.4/static/extend-extensions.html http://www.postgresql.org/docs/9.4/static/extend-extensions.... https://wiki.postgresql.org/wiki/Foreign_data_wrappers https://wiki.postgresql.org/wiki/Foreign_data_wrappers
- jMyles 11y agoI've lost track of the characteristics of the players in this space. Do Rackspace or DigitalOcean have anything that even compares? I've always used Rackspace as a sort of default. They seem to employ good people, so they must be good - so the thinking goes. But what's the real breakdown between these three? And any others I might not know about?
- jpatokal 11y agoGoogle has CloudSQL (=MySQL) and BigQuery, which is SQL-ish. https://cloud.google.com/sql/ https://cloud.google.com/sql/ https://cloud.google.com/bigquery/ https://cloud.google.com/bigquery/
- johansch 11y agohttp://www.slideshare.net/bytebot/mysql-in-the-hosted-cloud http://www.slideshare.net/bytebot/mysql-in-the-hosted-cloud
- aquadrop 11y ago
- acconrad 11y agoDoes anyone know how this benchmarks against Postgres and MySQL? Would be curious to see how TPC-C runs on this platform but I can't find any studies of any independent parties benchmarking Aurora.
- boulos 11y agoIt only just came out of preview, so I'd expect someone independent to test it out in the coming weeks. Maybe / hopefully one of the Percona folks? Edit: typo (out of preview)
- awgupta 11y agoFor TPC-C like benchmarks, you can run: 1) CloudHarmony: https://github.com/cloudharmony/oltpbench https://github.com/cloudharmony/oltpbench 2) Percona: https://code.launchpad.net/~perconadev/perconatools/tpcc-mysql https://code.launchpad.net/~perconadev/perconatools/tpcc-mys... We've found it easier to load large datasets using CloudHarmony but have run both. I'd recommend reading through http://d0.awsstatic.com/product-marketing/Aurora/RDS_Aurora_Performance_Assessment_Benchmarking_v1-2.pdf http://d0.awsstatic.com/product-marketing/Aurora/RDS_Aurora_.... It is oriented towards SysBench, but will help you set up your clients to have enough network throughput to run a full test. Generally, we find the performance comparison improves on large instances, high throughput workloads, or when the data set does not fit in RAM.
- choppaface 11y agoIs 30GB of RAM the max for this service? That's rather paltry for very large joins, which seem plausible given they claim support to up to 64TB of data. Percona had recommended 144GB+ and moving to that from 64GB made our BI-type queries an order of magnitude faster. Obvi YMMV but in my experience one needs more RAM for joins before more disk for data. That said, the UI and system for managing replication looks pretty nice. I've never done MySQL admin directly but I do appreciate how much of a pain point this can be.
- nrb 11y agoFrom the article: "Instances are available in 5 sizes, with 2 to 32 vCPUs and 15.25 to 244 GiB of memory."
- boulos 11y agoNo; from the product page: "You can use Amazon RDS to scale your Amazon Aurora database instance up to 32 vCPUs and 244GiB Memory. You can also add up to 15 Amazon Aurora Replicas across three availability zones to further scale read capacity. Amazon Aurora automatically grows storage as needed, from 10GB up to 64TB."
- yeskia 11y agoSo keen on Australia to get a third availability zone so that we can use this - we were lucky enough to be invited to the preview programme but there was no point to us hosting on another continent. Very exciting!
- boulos 11y agoOfftopic a bit but I'm genuinely curious: Jeff, why did you post this at 10pm Pacific?
- goblok 11y agoBecause he feels like it.
- kolev 11y agoThat's pretty much his job and he's very passionate about it.
- jeffbarr 11y agoBecause that's when it was ready. Our launch process is fairly involved -- getting all of the code out to production servers across multiple regions, updating the console, pushing the docs, and so forth. The last three steps for a big release are push the press release, push the blog post, and push the social media.
- davidw 11y agoInteresting from a licensing point of view: the GPL does not require you to redistribute changes that are 'internal' to your organization. You are not redistributing the program itself in this case, you're just letting someone access it. For some people, this kind of end-run around the GPL is the poster child for the AGPL: https://en.wikipedia.org/wiki/Affero_General_Public_License https://en.wikipedia.org/wiki/Affero_General_Public_License
- Sanddancer 11y agoThere's a distinct possibility that it's not based on MySQL at all. From their prior discussions, this looks like a clean implementation that just uses its API, so even if MySQL were AGPLed, Amazon wouldn't need to release anything.
- Zikes 11y agoBut thanks to recent rulings [1], APIs are still copyrightable and fall under the same license. [1] http://arstechnica.com/tech-policy/2014/10/google-oracle-java-api-copyright-battle-lands-at-supreme-court/ http://arstechnica.com/tech-policy/2014/10/google-oracle-jav...
- sneak 11y agoI believe parent is using "API" in the "wire protocol" sense, not the "intra-process programming interface" sense. Those have not been deemed copyrightable as yet, because those don't necessarily require header files.
- mathnode 11y agoIt's just a storage engine, albeit a very good one (I was on the beta); storage engines act as a plugin to the mysql/mariadb server layer, and are typically just a separate binary.
- sneak 11y agoThe AGPL is fundamentally flawed. Imagine if Google were subject to it; the idea of free software is to have the option to modify that which you run yourself. Without a million spindles, Google's main products are useless to you. Services are different than software.
- iofj 11y agoI wonder what this is. Did they create a proper multitenant version of MySQL or are they simply running mysql without a container around it ? I'm guessing it's the latter (much easier) option, but ... Do you really get 5x savings from simply not having a container ?
- BillinghamJ 11y agoIt isn't MySQL at all. It just has a facade on the front compatible with MySQL's API.
- deleted 11y ago[deleted]
- mathnode 11y agoAmazon Aurora – New Cost-Effective MySQL-Compatible Database Engine for Amazon RDS * https://aws.amazon.com/blogs/aws/highly-scalable-mysql-compat-rds-db-engine/ https://aws.amazon.com/blogs/aws/highly-scalable-mysql-compa... It's an engine for MySQL. Aurora Database Architecture by Amazon Web Services: * https://www.youtube.com/watch?v=-TbRxwcux3c&list=WL&index=5 https://www.youtube.com/watch?v=-TbRxwcux3c&list=WL&index=5
- Mizza 11y agoCan anybody with any experience compare this vs RDS for me?
- giaour 11y agoAurora is an engine for RDS.
- zaargy 11y agoNo Cloudformation support yet? I don't see it in the docs.
- kolev 11y agoI'm highly disappointed by Amazon for recommending CloudFormation, but always neglecting to maintain it. Aurora has been in preview for a while, adding support for it in advance would be a great thing and coordinate the launches. As a person who fell in the trap to use the "best practice", I'm always months if not years behind in my CloudFormation stack as it takes months for CloudFormation engineers to add, let's say, a single attribute. So, based on my experience, give it a quarter before you see it in CloudFormation! I keep bugging Jeff Barr, but it seems that he doesn't have much control over the process. The whole development process at Amazon seems broken. Everything is developed in a silo and once GA, then other teams start integrating it, unless the two products cannot operate in isolation. P.S. I keep checking this URL [0] several times a day and I curse several times a day, because it almost never changes! [0] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/ReleaseHistory.html https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
- codeonfire 11y agoAmazon is involved in a lawsuit over CloudFormation and U.S. Patent No. 8,271,974. Perhaps they are hesitant to invest until that is cleared up.
- kolev 11y agoThey still update it, just very slowly, but I wasn't aware of this other patent troll. I guess Terraform and Fugue are in danger as well.
- imaginenore 11y agoAt these prices I would rather install MySQL on Hetzner myself at a fraction of a cost.
- agopaul 11y agoI think Aurora doesn't compare to any other host-it-yourself solution, being managed and 'replicated' across multiple datacenters. Plus, Hetzner, really?
- notwedtm 11y agoEvery time I see a comment like this, I get the feeling that the poster doesn't understand how "the cloud" works. Everybody knows that renting a dedicated server (or purchasing your own servers) is generally a more cost effective approach. Cloud computing is used to accomplish different goals when used appropriately. Sure you could install MySQL on a box at Hetzner, but what happens when it goes down? Do you feel like maintaining the box for security and updates? What about log rotation? What if you need to migrate to a different geographical region, or need to increase capacity? Smaller sites might be okay with managing those on their own, but as you grow, being able to offload that work to Amazon (who tend to know what they are doing more than the average joe) is a benefit when you have other business-logic issues to deal with. So yes, naively getting a box at Hetzner or any other dedicated provider would be less of an upfront monetary cost, but in the long run, the advantages aren't as clear cut.
- waps 11y agoReally ? And what if Amazon loses data ? [1] But all data saved to S3 was safe, right ? No. [2] The key to decent safety is to 1) have an online backup at a different provider you can switch to quickly 2) have regular offline backups 3) actually reinstall from those offline backups on a regular basis Amazon falls flat on it's face with (1). If you use Amazon's tools you cannot have that, period. I would argue that securing against amazon financial failure is one of the important objectives of (2), so you cannot use glacier for that. This means that using just amazon (or even having your primary and online backup on amazon) is a bad idea. Given that you can't really use those amazon products anyway, you may as well save the money and go for hetzner and ovh or something like that. Famous customers like Netflix moved away from Amazon lock in [3] and for anything that you want to last, you should too. We all know that most people's backup strategy tends to be "X" will never fail. X in the worst case is their hard drive. For larger businesses X usually is something like EMC, and we've all seen it fail (usually, to be fair, not EMC's fault). You are replacing X with "Amazon" and that's it. [1] http://www.businessinsider.com.au/amazon-lost-data-2011-4 http://www.businessinsider.com.au/amazon-lost-data-2011-4 [2] http://www.quora.com/Has-Amazon-S3-ever-lost-data-permanently http://www.quora.com/Has-Amazon-S3-ever-lost-data-permanentl... [3] http://venturebeat.com/2014/06/19/adrian-cockcroft-public-cloud/ http://venturebeat.com/2014/06/19/adrian-cockcroft-public-cl...
- elktea 11y agoIs this based on Galera?
- falcolas 11y agoThat's my question as well - and an important one. Using Galera cluster with MySQL imposes several performance and usage constraints an end user needs to be aware of. And if it's not Galera, how did Amazon work around the constraints of multiple writers in an ACID database, and what constraints does it impose?
- mathnode 11y agoIt's linked in the article, here is more links: https://news.ycombinator.com/item?id=9960276 https://news.ycombinator.com/item?id=9960276
- falcolas 11y agoIt definitely looks like Galera (which is what powers both MySQL and MariaDB cluster implementations; it's a storage engine built atop InnoDB), but it's hard to say without more information. They mention a quorum write with automatic recovery across three nodes, but doesn't mention the method used - two phased commit, checking commits against pending transactions, etc. It's a very complex thing to implement, and unless they have made leaps beyond what Galera has done, for some workloads it will be fast, but for others it will perform far worse than a standard MySQL instance. Of course, I guess it could also be a cluster built upon NDB, but the lack of memory constraints on the size of the data makes that less likely.
- awgupta 11y agoAurora isn't implemented based on either Galera or NDB.
- falcolas 11y agoSince it sounds like you have the information on what it is based upon (if only the principles which were used to address distributed ACID consistency), it would be good to get this information dissiminated - it's hard to trust that it will "just work" when we have so many examples of distributed ACID not working well.
- anotherangrydev 11y ago>they verified that each Amazon Aurora instance is able to deliver on our performance target of up to 100,000 writes and 500,000 reads per second This bit caught my attention, does an "Amazon Aurora instance" means one computing instance? Or do they refer to something like your allocated share of the overall Aurora platform? Because if they are able to achieve that performance per-machine, I'm truly amazed. Their larger instance appears to be "32 vCPUs and 244GiB Memory", that sounds credible to be able to sustain that throughput, particularly if your whole data fits in RAM, but barely. Would be nice to see R/W performance on the smaller instances.
- jeffbarr 11y agoYou can read the blog post that I wrote last year when we announced Aurora to learn more about how it works: https://aws.amazon.com/blogs/aws/highly-scalable-mysql-compat-rds-db-engine/ https://aws.amazon.com/blogs/aws/highly-scalable-mysql-compa...
- o_____________o 11y agoDo you have any plans for Geo functionality? We're working on a massive project that Aurora sounds perfect for, aside from the apparent lack of that feature.
- devNoise 11y agoIt seems like MySQL 5.7 will start to have better GIS support. Though that is still a development release at this point. Hopefully if Amazon tries to maintain compatibility, native GIS support may eventually arrive.
- awgupta 11y agoit is a per-instance number. For those SysBench numbers, we are running an r3.8xlarge instance type.
- waivej 11y agoAt that pricing, I'm curious if it makes sense to put blobs in the database rather than S3. Any thoughts?
- cornellwright 11y agoThat's a nice pricing, particularly paying what you actually use but it's still more than 3x the cost of just storing it in S3.
- jewel 11y agoAt $100/TB/mo? I don't think so. S3 is $30/TB/mo. Unless you have some compelling reason to have the blobs in the database, like indexing them, you'd be better off keeping them somewhere else. I guess if you won't have many blobs, it may be worthwhile to save yourself engineering time of supporting both S3 and Aurora. In an ideal world, there'd just be one permanent data store with a built-in caching layer, for databases, blobs, and log files.
- toomuchtodo 11y ago> I guess if you won't have many blobs, it may be worthwhile to save yourself engineering time of supporting both S3 and Aurora. In an ideal world, there'd just be one permanent data store with a built-in caching layer, for databases, blobs, and log files. We're getting close to that. Write once your data to S3 for persistent storage, then load into Elastic Search. Mostly there.
- meritt 11y agoI'd avoid it. It's expensive and MySQL is going to store all those blobs off-page anyway. S3 gives you nice perks like versions and you can just store the s3_name+version as metadata in your DB instead. Also if you want to do any heavy processing of your blobs outside of the DB, S3 tends to be a lot better for parallel access (e.g. hadoop jobs) Obviously don't know your exact use case, but that's what we do.
- wahnfrieden 11y ago> Due to Amazon Aurora’s unique storage architecture, replication lag is extremely low, typically between 10 ms and 20 ms. I think in the original announcement, this number was lower, but this is still great. In this range of latency, the most compelling thing relevant to my needs is the possibility of using read replicas for serving read-only API calls, with minimal risk of having a write API call immediately followed by a read API call serving stale data. It's possible to orchestrate a prevention for that with regular latency-prone MySQL replication, but it carries tremendous complexity depending on the application. Would be very interested if anyone else has explored this idea further with Aurora.
- felixgallo 11y agoThe AWS pages are very heavy on marketing and light on system characteristics. It would be pretty nice to know if the characteristics of Aurora are different than RDS; for example, RDS table size and index rebuilding were a big problem for a client, and I suspect that Aurora would do better, but have no reason to believe it.
- jdc0589 11y agoAurora is not an RDS alternative. It is a new database engine available to use on the RDS platform.
- felixgallo 11y agohopefully everyone else understood what I said.
- awgupta 11y agoWe've made a number of improvements relative to MySQL, for example with large numbers of tables and the results set cache. There are some improvements on large tables and schema changes, but quite a bit more to be done in those areas. You can contact our PM team at aurora[dash]pm[at]amazon[dot]com to give us the issues that matter most. Feedback is how we prioritize!
- dylanz 11y agoWe've been experiencing huge problems with our multi-AZ and encrypted MySQL RDS instance this morning (it looks like a hardware issue). We're contacting support, but are considering taking our entire application down and migrating to Aurora. The timing on this is too ironic.