12 ms·
Amazon Launches Managed Cassandra Service
- heipei 7y agoThey could have offered a managed ScyllaDB service and saved themselves and their customers a lot of money while staying CQL / Cassandra compatible. Or what am I missing here?
- enitihas 7y agoLicense, ScyllaDB is AGPL 3.0. Also maintainability I am assuming. ScyllaDB is written by a team of people experts in writing high performance low level code, and I am not sure it is easy for others to fork it easily. Cassandra being written in java and being a much more normal application (no DPDK), might be easier to fork.
- _msw_ 7y agoDisclosure: I work for AWS, but this is my personal opinion. We have lots of developers that write code that's similar to how ScyllaDB is built. Lots of DPDK. There are many other practical reasons why a fork should never be the first choice when building a service that you have to sustain for customers basically forever. Especially if there's active, high velocity development going on.
- pritambarhate 7y agoScylla is licensed under AGPL. Most of the cloud services companies try to stay away from AGPL. However, I think the real reason is not even that. I suspect underneath it uses modified version of DynamoDB and it's just wire compatible with Cassandra. That's how they are offering pay as you go pricing and auto-scaling. Considering the kind of resources Apache Cassandra requires to run, I don't think they can offer this kind of pricing. This is the same company that charges $0.2 per hour to run a K8s cluster.
- _msw_ 7y agoDisclosure: I work for AWS The new Amazon Managed Apache Cassandra Service does use the open source Apache Cassandra code. The team is excited about working with the development community on Cassandra. See more at https://aws.amazon.com/blogs/opensource/contributing-cassandra-community/ https://aws.amazon.com/blogs/opensource/contributing-cassand...
- AtlasBarfed 7y agoDid you guys swap out the storage guts like Rocksandra did? Can we access the sstables of our tables? Can we do triggers and UDF? Can we access the timestamps at a CELL LEVEL like cassandra proper (VERY important for online/downtimeless data migration)
- jjirsa 7y agoI don't work for AWS, but I do run Cassandra at scale for a living, and this line: > Considering the kind of resources Apache Cassandra requires to run Makes no sense. There is plenty of talent at AWS to run Cassandra proper on AWS for less money than they're currently charging for Dynamo. This was true before this announcement - it's how people like Instaclustr stay in business (containerized cassandra as a service on EC2).
- deleted 7y ago[deleted]
- jjirsa 7y agoAGPL and immature.
- AtlasBarfed 7y agoI would like a scylladb option as well, but it isn't exactly feature parity with cass 3.11.X, to say nothing of 4.X It started like gangbusters, but I get the feeling the dev money ran out because the feature rollout has completely stalled.
- PeterCorless 7y agoAside: We have Scylla Cloud. It runs on AWS. Our next major release includes LWT and CDC, so any major gaps will be closed soon. And yes, we're looking at Cassandra 4.0 closely. In some ways, we will implement features similarly to Cassandra but subtly and importantly different. Example: We do both local and global secondary indexes (Cassandra only has local). Also, our CQL LIKE function is closer in implementation to SQL LIKE than to Cassandra's. And so on. btw: ScyllaDB just got $25M in new funding in September 2019. We're doing fine financially.
- cnlwsu 7y ago> what am I missing here? Mostly that your believing Scylla marketing literature and paid benchmarks over what people actually use and experience. Theres thousands of large Cassandra deployments and a hand full of Scylla ones and it's not because people didn't do their research. If they did to ScyllaDB though it will probably kill the product, with Cassandra this is more of a positive for the community.
- stas12 7y agoThat's a strong accusation. Do you have a reference to any 'unpaid benchmarks', so we can judge ourselves?
- privateSFacct 7y agoI don't think many enteprises are eager to have AGPL floating around in their repo's or has that changed? I don't think google allowed checkins of AGPL to their repo but maybe this has all changed.
- jedberg 7y agoPoor Datastax. Another company killed by AWS.
- _msw_ 7y agoDisclosure: I work for AWS, but this is my personal opinion. I don't know of any company "killed" by AWS, and I don't think that Datastax will be either.
- jedberg 7y agoHeh you don’t know about them because they don’t exist anymore. I know at least three founders who had to shut down after AWS launched a feature at re:invent. One went on to start another company that was acquired by Facebook in an 8 figure deal.
- _msw_ 7y agoWere they in angel / seed rounds? Which AWS features?
- rjurney 7y agoAWS contributes very little to open source and takes all the SaaS profits from the project founders.
- bklyn11201 7y agoWe begged Datastax to provide a managed service in AWS. We used Datastax for consulting and for their repackaging of Cassandra with admin tools. After perpetual pain with Cassandra management, and little belief that Datastax would enter the market for managed services, we eventually decided to stop everything and rewrite to Dynamo.
- AtlasBarfed 7y agoDatastax was undeniably a beneficial force behind earlier cassandra, and I initially thought the Apache board pseudo-ejecting them from the project governance was foolish, but I do think they were starting to mess with cassandra a bit too much. They can probably support their product line atop this just fine. They really make their money on integrating cassandra and a bunch of other big data frameworks in one neat little package.
- jedberg 7y agoAt reddit, we used Cassandra, and it was a huge pain to manage. At Netflix we used it, and had a whole team of engineers that built tools just to manage Cassandra. If this service had existed then, it would have made life so much easier!
- _cerv 7y agoI just left a company that used an in-house cloud object storage product for their cloud DVR service. It was an amazing piece of tech that was hamstrung by some of the most irritating "features" of Cassandra. We spent more time cleaning up than anything else. I am looking forward to seeing how people use this at scale.
- rob-olmos 7y agoCould you provide a bit more specifics on what was being cleaned up please? Thanks
- takeda 7y agoI find it a bit weird, that company that created a DynamoDB is now supporting a data store that supposed to be an open source DynamoDB. Does that mean Cassandra is better than DynamoDB since people would still prefer to use Cassandra in AWS?
- y4mi 7y agoOnly if it were completely compatible. As they aren't, even new projects will often not use DynamoDB to avoid vendor lock-in
- Spivak 7y agoNaah, I just think it means that people will pay for a hosted Cassandra instance that wouldn’t migrate to DynamoDB. Honestly having a SaaS and the hosting instances of your competing offerings is probably a good strategy.
- enitihas 7y agoI think Amazon realises that it is better for them to disrupt themselves, rather than someone else.
- alexnewman 7y agoI'd bet money this is actually dynamodb and will be insanely expensive.
- giaour 7y agoPricing is available at https://aws.amazon.com/mcs/pricing/ https://aws.amazon.com/mcs/pricing/ MCS has a very similar pricing model to on-demand mode DynamoDB (https://aws.amazon.com/dynamodb/pricing/on-demand/ https://aws.amazon.com/dynamodb/pricing/on-demand/) but is ~15% more expensive on all line items.
- alexnewman 7y agoWow super psyched I got triple downvoted when I dared to say something mean against one of the great deathstar companies. I've noticed if I ever dare to say anything mean against amazon or google I get instantly downvoted. It is more reliable than a bot. Anyway, dynamodb is insanely expensive compared to postgresql and I'd recommend only using it when you need to expose a datastore directly to the user as it's attribute based access control is easy to use. Postgresql serverless is now a thing as well. Anyway, if you truely want a WAN datastore, exposed to web clients, with attribute based access control, run dynamo.
- giaour 7y agoExcited to see a managed open source offering with the same pricing paradigm as pay-per-request DynamoDB. I frequently encounter projects for which a Dynamo-style storage layer would be a natural fit, but often use RDS Postgres because I don't want to lock projects into the AWS ecosystem or sign teams up for operating their own Cassandra or Riak clusters.
- scarface74 7y agoSo instead of using the best solution, you chose to use a non optimal solution for fear of “lock in”? Were they not using any other AWS services? If not, why pay more for a cloud provider than a colo and not get any of the benefits of it?
- giaour 7y agoI would disagree with that characterization and would instead say that DynamoDB's lack of portability between vendors made it a non-optimal solution. The services I work on need to survive specific vendors going out of business.
- scarface74 7y agoWhat’s more likely to happen your company going out of business or AWS? Is your company better capitalized than Amazon? Would you also refuse to use Windows just in case Microsoft went out of business?
- stephenr 7y agoAmazon going out of business isn’t the only reason you shouldn’t be happy to be tied to them. If you want decent “high availability” and failover when shit hits the fan, that means multi vendor. Who else offers dynamodb? If amazon were to increase the price of dynamo db 10x would you still just keep using it, because you’re already using it, and eat the 10x cost increase? The amount of comments that essentially boil down to “amazon is not going anywhere, you’re stupid for worrying about lock in” is both staggering and depressing.
- didip 7y agoIt's about freakin time. Dynamo's lack of query language has always been a pain point for me. Congrats AWS team!
- pritambarhate 7y agoMaster stroke! Especially the pricing and autoscaling. Till they brought autoscaling and pay as you go pricing for DynamoDB, Google cloud data store was superior product (at least on paper) as you didn't need to think about preprovisioning the capacity. It is supposed to just scale. They have brought the same model to Cassandra. So no vendor lock in! Also one of the reasons small projects stayed away from Cassandra is the requirement of an hefty cluster to get decent performance out of it. That too is taken care by AWS and made Cassandra much more accessible. Now only thing that needs to be seen is how good this AWS product is. Especially for first year many AWS services get very average reviews.
- enitihas 7y agoThis seems to be very good for people who don't want to be locked in to DDB. The prices are very similar to DynamoDB(slightly higher), and the model is pay as you go. Prices: Managed Cassandra: Write request: units $1.45 per million write request units Read request: units $0.29 per million read request units Storage: $0.30 per GB-month Dynamo DB On Demand: Write request: units $1.25 per million write request units Read request: units $0.25 per million read request units Storage: $0.25 per GB-month
- deleted 7y ago[deleted]
- paragraft 7y agoTangentially: I don't understand why they have storage costs so high for DDB still (hasn't changed since 2013) when Aurora's charging $0.10 GB-month (with 6 way redundancy at that, vs DDB's 3x)...
- adzm 7y agoIs there any reason to actually use Cassandra instead of Scylla?
- deleted 7y ago[deleted]
- jjirsa 7y agoBiased as a cassandra committer, but: - Feature sets aren't the same yet - History / maturity - License - Actual savings dont tend to match proposed savings - Development isn't driven by a startup that AWS can kill with an announcement like this.
- manigandham 7y agoThe biggest feature still missing in Scylla is Lightweight Transactions. Otherwise no, use ScyllaDB instead for much better performance, automatic tuning, minimal maintenance, and better feature implementations like truly scalable global secondary indexes and materialized views.
- rbranson 7y agoMy guess is that this is on-demand DynamoDB wrapped with a Cassandra-compatible frontend. Just based on the pricing and stated characteristics. It would be really hard to provide on-demand capacity for compute and storage using anything remotely similar to off-the-shelf Cassandra. The pricing looks like DynamoDB on-demand + a bit extra to cover the cost of operating the frontend. EDIT: confirmed by an AWS employee here https://twitter.com/_msw_/status/1201924979647905792 https://twitter.com/_msw_/status/1201924979647905792
- bklyn11201 7y agoThis explanation does make a lot of sense. But why not just add a Cassandra interface to DynamoDB the way they added a Postgres interface to Aurora RDS?
- enitihas 7y agoAurora RDS is a meta product name though, you can't use it without specifying the engine, MySQL or PostgreSQL. What you are saying would be more like adding a postgres frontend to mysql.
- scarface74 7y agoThey didn’t add a Postgres front end to RDS. They took the Postgres open source code and (grossly simplified) changed the storage layer and added more AWS specific features. Aurora/MySQL and Aurora/Postgres or more or less AWS specific forks of the respective projects.
- _msw_ 7y agoDisclosure: I work for AWS Off-the-shelf Cassandra has a fair amount of flexibility in its architecture. Open Source Cassandra is the thing that is in the front end. And the team is excited about the idea of collaborating with the development community around some of the generally useful abstractions needed to build this managed database experience. More at https://aws.amazon.com/blogs/opensource/contributing-cassandra-community/ https://aws.amazon.com/blogs/opensource/contributing-cassand...
- tschellenbach 7y agoFor the first 3 years of Stream we used Cassandra. Afterwards we switched to a custom RocksDB + Raft solution. (somewhat outdated stackshare interview: https://stackshare.io/stream/stream-and-go-news-feeds-for-over-300-million-end-users https://stackshare.io/stream/stream-and-go-news-feeds-for-ov...) The difference is massive. Cassandra was hard to manage and after many years of our team using it still had random spikes. RocksDB+Raft has been extremely solid, doesn't require any maintenance, load times are flat, zero spikes. Cassandra was awesome, but it definitely has some issues. That's also why companies like ScyllaDB see space in that market. I wonder if AWS's cassandra implementation is better than regular cassandra.
- jjirsa 7y ago> Cassandra was hard to manage and after many years of our team using it still had random spikes A lot of that is getting better over time. A non-trivial percentage of contributions are coming from companies that are only looking to improve their operational story, not adding new features. Most of the talks at NGCC (Next generation cassandra conference) focused on operational improvements - it's something we care a lot about (myself especially).
- AtlasBarfed 7y agoWere your spikes all p999 impacts from GC? I'm hoping rocksandra will solve a lot of that.
- gregwebs 7y agoThose that don't want to roll their own RocksDB + Raft can consider using TiKV, which uses the RocksDB + Raft architecture. https://tikv.org/ https://tikv.org/
- techie128 7y agoSomething awesome is coming up for Cassandra 4.0
- bsg75 7y agoCan you add anything more specific? Something related to the storage engine?
- antoncohen 7y agoOne of the killer features of Cassandra is cross-region replication with location aware consistency for queries. It looks like in this preview of Manages Cassandra Service they are only supporting single regions clusters. I do hope they support multi-region clusters in the future. It would also be nice if they documented any differences from Apache Cassandra, like if Amazon MCS improves secondary indexes so they can be used in more cases.
- AtlasBarfed 7y agoI personally don't trust dynamodb's mat views/secondary indexes since they never explain how they maintain index coherency behind their black box. If they do have better tech for that, I also hope they can solve the mat view problems in cassandra 3.x
- AtlasBarfed 7y agoSo many questions - per-cell timestamps (this is IMPORTANT for online data migration with no downtime)? - can you choose compaction strategy? - access to sstables for rapid data loads/custom backups? - triggers? - UDT? - how will upgrades work? - are they using rocksandra or other techniques? - how about a scylladb option?
- xref 7y agoI’m curious if this plays out like their MongoDB debacle where Amazon burned all their goodwill, only extracting value from the community and returning nothing
- cnlwsu 7y agoThe contributors for Cassandra are from a group of companies that use it, not a single company that is trying to be profitable from it. I think this is a positive for the community, even if they don't contribute back since it would provide an easy getting started platform.
- techie128 7y agoASF != Mongo
- thekozmo 7y agoKudos for AWS for the ability to launch so many services, many of them are competing with each other or complimentary. At first I (disclosure, ScyllaDB co-founder) called their serverless a bluff but I gave it a try and it's nice to create a table without waiting for any server. That said, I know personally that Dynamo scales really slow so they wouldn't catch up the speed. It also say 1 digit ms latency but state the plan to improve jvm overhead. Another funny thing is (regardless of tech) that AWS folks have wonderful people talking about open source but this very solution isn't open at all. It's impossible to figure out what's Cassandra and what's Dynamo. How about a diagram? Lastly, it's pricey (for 1M iops you'll pay $3.5M/year!!) and doesn't have counters, UDT, materialized views and plenty other features. If you got down here, hey, give ScyllaDB a try, it's OSS (AGPL) and as a service on AWS and has features that neither C* or MCS has - workload prioritization for instance. Dor
- marktangotango 7y agoJust curious about AGPL. Do you feel that AGPL gives you a hedge against a cloud service deploying your project at scale for profit that the project would otherwise miss out on? I ask this as someone who has been sitting on some code for 10 years for precisely this reason!
- PeterCorless 7y agoA: The "one additional feature" - Section 2(d) reads as follows: " If the Program as you received it is intended to interact with users through a computer network and if, in the version you received, any user interacting with the Program was given the opportunity to request transmission to that user of the Program's complete source code, you must not remove that facility from your modified version of the Program or work based on the Program, and must offer an equivalent opportunity for all users interacting with your Program through a computer network to request immediate transmission by HTTP of the complete source code of your modified version or other derivative work." "This enables the original author... to download the source and receive the benefit of any modifications to its original work. So if a cloud vendor (or anyone) decided to take your software and use it, if they made modifications to it you are entitled to get access to those modifications. For many people this is a barrier to entry -- they want the ability to modify it in however they want (which they can still do privately under AGPL). But if they offer it in public, they have to share their mods back to the community. At least, that is my layperson's understanding of it. I am not an intellectual property lawyer and do not pretend to be. Source: http://www.affero.org/oagf.html http://www.affero.org/oagf.html
- rjurney 7y agoAmazon taking from open source without giving back.
- alexbilbie 7y agohttps://aws.amazon.com/blogs/opensource/contributing-cassandra-community/ https://aws.amazon.com/blogs/opensource/contributing-cassand...
- rjurney 7y agoRight. Total bullshit PR you're supposed to be smarter than.
- tus88 7y agoCassandra hasn't changed their license like MongoDB yet?
- PeterCorless 7y agoScyllaDB's take on the new announcement: https://www.scylladb.com/2019/12/04/managed-cassandra-on-aws-our-take/ https://www.scylladb.com/2019/12/04/managed-cassandra-on-aws...
- rjurney 7y agoIf AWS runs everyone's open source for them as a metered service and people don't run their own software... then they can't edit that software and make contributions back into open source. The open source model breaks, no new projects except from the likes of Amazon. AWS would have to step up open source contributions to match users' and they in fact do the opposite: they are extreme leeches compared to Google and Microsoft.
- vroyer 7y agoI'm the author of Elassandra (Elasticsearch plugin for Apache Cassandra) currently under Apache 2. I think i'm going to release the next major version of Elassandra under AGPL like ScyllaDB...