12 ms·
Scaling Riak at Kiip
- gsibble 14y agoGreat talk! Looking at Riak vs MongoDB right now for a production system in fact. The data isn't K/V though and we need rich queries so I'm not sure what our solution will be unfortunately.
- bfrog 14y agoWhy not just use postgresql?
- WALoeIII 14y agoAgreed, start with a known entity. Take the one special table that is killing you and move it to a special database (Riak/Cassandra etc) suited for that task only once you understand exactly what you need.
- tptacek 14y agoI think Kiip agrees with you completely too.
- jasonmccay 14y agoI find it a better course of action to work on properly understanding your data problem and then evaluate the options available instead of just sticking with the stand-by solution ("Well, we've always done x!"). Just like with anything in technology ... there is going to be pain as you escalate the level of complexity and what you are trying to accomplish.
- mitchellh 14y agoI disagree, based on experience. On the surface, MongoDB did solve our data problem, and we thought we sufficiently researched the technology that it would work, even attending multiple MongoDB conferences and talking to 10gen employees. The issue was that underlying architectural decisions in MongoDB ended up biting us and limiting us rather significantly. It could be argued that this is because MongoDB is a rather new piece of tech (I disagree, I think MongoDB is fundamentally flawed, but it doesn't matter in this argument). Because of our experience, going with the standby IS the best choice, until you _need_ something else. Riak was a change necessitated by its fast growth such that horizontally scaling was necessary when you're in an environment such as EC2. Ignoring MongoDB specifically, the IDEA of MongoDB simply isn't correct here, a Dynamo-style K/V store is the correct option, and Riak happens to be a fantastic one.
- taligent 14y agoBecause Postresql is not a NoSQL database despite the recent lipstick additions. I love the database but it is not optimal if you have a fluid schema like many use cases have today.
- deleted 14y ago[deleted]
- lwat 14y agoI recently asked if anyone could point me to an (open source) app on github or wherever that has a 'fluid schema' on a NoSQL system and nobody was able to show me one. Do you have a concrete example I can look at please?
- lwat 14y agoNothing? For something as common as you say I'd have thought there would be many examples out there. I don't get how this can be touted so often when there's no concrete use cases for this 'feature'.
- jasonmccay 14y agoI think that you would find MongoDB to work nicely for you ... especially with the need for rich queries (which assumes, on my part, that you have more complex data needs, documents, etc.). With all due respect to the Kiip engineering team, this wasn't a strong case for using Riak over MongoDB ... but rather the general pain that a engineering team feels when horizontally scaling in the cloud.
- tptacek 14y agoIt's been a riak-y day for me, so I watched this avidly. You're no doubt wondering what the issues were; they come 20 minutes in, and were (paraphrased): * At scale in production, adding a new node took days to complete all the handoffs; they recommend adding new nodes as soon as it's looking like you need them, rather than waiting until you're redlining. * 2i is slow, especially in EC2; a straight KV "get" is milliseconds-denominated; 2i index queries were taking multiple seconds. Use 2i, they say, but in background processes. * Javascript MapReduce is slow; this is well known. They confirm Erlang MR was adequate. * As the LevelDB keyspace grows, there's a stepping function in latency; 5ms, then 15ms, then 25ms; the solution is to add nodes. (LevelDB is Google's KV store, a new option for Riak, required if you're using secondary indexes). * Riak Control didn't work for them over low-latency connections. * Once, a Chef misconfiguration left the whole cluster flapping on and off, which corrupted the cluster; they recovered with Basho support. Be careful about adding and removing nodes rapidly. * Similarly, flapping a single node caused the cluster to get into a state where it wouldn't converge again; the cluster worked but no nodes could be added until they (presumably?) restarted it.
- j2labs 14y agoVery useful data. Do you have anything more comprehensive about your experience?
- tptacek 14y agoI don't work at Kiip. :)
- j2labs 14y agoI thought you meant to say you had additional personal experience, but perhaps you've just been reading about Riak today. Thanks anyway.
- tptacek 14y agoI've had a 3-node hardware Riak cluster lying around for about 4 months now, but only just today cut over from Postgres to it. I can talk to you about how shiny Riak is on day 1, but as this presentation shows, the day 1 behavior of a Riak cluster can be a bit of a trap.
- Scramblejams 14y agoSomething they (understandably) didn't address in the video was fault tolerance on the Postgres side. What do people like these days for that?
- mitchellh 14y agoI'll address that here: We use streaming replication with a hot standby for fault tolerance on the PostgreSQL side. For backups in addition to the streaming replication we do WAL shipping to S3 every 15 minutes or 16 MB, and we do a full base backup to S3 daily during our low-traffic time of the day, which uploads around 30 GB to S3 daily (but LZO-compressed, so actually down to around 3 GB).
- Scramblejams 14y agoThanks, Mitchell. Is your hot standby in a different zone (e.g. one in the West, one in the East) so you can handle one of their big outages?
- mitchellh 14y agoYes
- platform 14y agoin the last couple of month reading this site, I see more negative MongoDB reviews than most other 'cool tech' stuff. On one hand MongoDB by carrot-or-stick was pushed into environments with high-write needs (OLTP kind of systems). On another hand having true secondary indexes and semistructured data makes Mongo a 'closest to RDBMS' choice. So things like global write lock, indexes must fit into memory, auto-sharding questions, single-thread map-reduce -- all are pretty significant limitations for an OLTP data store. I wish MongoDB does not get discouraged and instead steps back, reviews academic foundations of the system and pick a couple of use cases and optimise their builds for them (similar to datawarehouse vs oltp kind of offerings)
- reality_hacker 14y agoAnything concrete about why cassandra is bad?..
- gregburd 14y agoOne key difference is that Riak will rebalance data across nodes as they are added or removed automatically, Cassandra will not. You have to manually adjust the partitioning of data, balancing it by hand.
- minaguib 14y agoThe glossing-over cassandra (because of Digg history etc..) was the point in the talk when I started raising eyebrows. Engineers evaluating systems should let the systems speak for themselves. We have a fairly small cassandra cluster in production, serving over 50x the volume they mentioned in their talk, with good latency (real-time bidding) and not-too-painful operational footprint.