4 ms·
Pretty interesting once you read past the marketing push. I mostly like the approach, but there are a lot of questions/issues that spring to mind (not that som
by jmull 3y ago
Pretty interesting once you read past the marketing push.
I mostly like the approach, but there are a lot of questions/issues that spring to mind (not that some of them don't already have answers, but I didn't read everything). I'll list some of them:
* I'm pretty sure restrictive schemas are a feature not a bug, but I suppose you can add your own in your ETL "microbatch streaming" implementation (if I'm reading this right, this is where you transform the events/data that have been recorded to the indexed form your app wants to query). So you could, e.g., filter out any data with invalid schema, and/or record and error about the invalid data, etc. A pain, though, for it to be a separate thing to implement.
* I'm not that excited to have my data source and objects/entities be Java.
* The Rama business model and sustainability story seem like big question marks that would have to have strong, long-lasting answers/guarantees before anyone should invest too much in this. This is pretty different and sits at a fundamental level of abstraction. If you built on this for years (or decades) and then something happened you could be in serious trouble.
* Hosting/deployment/resources-needed is unclear (to me, anyway)
* Quibble on "Data models are restrictive": common databases are pretty flexible these days, supporting different models well.
* I'm thinking a lot of apps won't get too much value from keeping their events around forever, so that becomes a kind of anchor around the neck, a cost that apps using Rama have to pay whether they really want it or not. I have questions about how that scales over time. E.g., say my has depot has 20B events and I want to add an index to a p-state or a new value to an enum... do I need to ETL 20 billion events to do routine changes/additions? And obviously schema changes get a lot more complicated than that. I get that you could have granular pstates but then I start worrying about the distributed nature of this. I guess you would generally do migrations by creating new pstates with the new structure, take as much time as you need to populate them, then cut over as gradually as you need, and then retire the old pstates on whatever timeline you want.... But that's a lot of work you want to avoid doing routinely, I'd think.
I'm starting to think of more things, but I better stop (my build finished long ago!)
- nathanmarz 3y ago* By "restrictive schemas" I mean being forced to represent your data storage in non-optimal ways – like not being able to have nested objects in a first-class way. Schemas themselves are extremely important, and they should be as tight as possible. * Rama's JVM-based, so the entire ecosystem is available to you. You can represent data as primitive types, Java objects, Protobuf, Clojure records, etc. * You deploy and manage your own Rama clusters. The number of nodes / instance types depends on the app, but Rama doesn't use more resources than traditional architectures combining multiple tools. * Some databases support multiple very specific data models (e.g. Redis). I don't consider that flexible compared to Rama, which allows for arbitrary combinations of arbitrarily sized data structures of arbitrary partitioning. * Depots (the "event sourcing" part of Rama) can be optionally trimmed. So you can configure it to only keep the last 2M entries per partition, for example. Some applications need this, while others don't. * If you're adding a new PState, it's up to you how far back in the depot to start. For example, you could say "start from events appended after a specific timestamp" or "start from 10M records ago on each partition". * We have a first-class PState migrations feature coming very soon. These migrations are lazy, so there's no downime. Basically you can specify a migration function at any level of your PStates, and the migration functions are run on read. In the background, it migrates iterates over the PState to migrate every value on disk (throttled so as not to use too many resources).