3 ms·

Using Datomic Pro at work for a data warehouse and soon for another project for <REDACTED>. I'm working on a migration toolkit for Datomic. (Migrating schemas,

Using Datomic Pro at work for a data warehouse and soon for another project for <REDACTED>. I'm working on a migration toolkit for Datomic. (Migrating schemas, not between database kinds.) Ask me questions about the API, my experience with ops, or anything else if you like.

dustingetz 13y ago
Can you ramble about how datomic is from ops perspective for a moment if you will? thanks
- coolsunglasses 13y ago
  Just to clarify, I don't know what problems sgrove ran into, I've asked him in IRC and will elaborate if he tells me, but at present, I dunno. So here's what I have encountered: Datomic's primary limitation is at the transactor level. Storage is a non-issue, especially if you're using DynamoDB. Even with PostgreSQL write-throughput was ridic because of the way they're using the database. Transactor fail-over I'm not sure I'd take too seriously, but I don't think they'll drop any writes on the floor either. You shouldn't attempt to run a datomic transactor on a cheapo VPS. Try to put together at least 4 gb of RAM. Which is the same as what's currently standard for consumer laptops. No excuses. Deploying Datomic itself is straight-forward. The config is super simple and all you need as a dependency is Java. I'm actually a little puzzled that people want "Datomic as a Service". Datomic and PostgreSQL were the simplest parts of my stack to deploy. You want beefy peers as well so as to make maximum use of the peer limitations on the license. This isn't as big of a deal as people think because you're not structuring your app the way Rails and Django apps do on Heroku. You don't do 1 beefy database + 5,000 terrible VPSes. Not only is that a management overhead (automation or not), but you're better off vertically scaling a couple peers first before running a shitload of dumb VPSes running your service. The running assumption is that you should use DynamoDB, but PostgreSQL was seriously efficient for our purposes. Indexing adds write overhead, but given the current limitations on what you can change in the schema online (nothing), you want to plan ahead here. If you fail to anticipate something in your schema, you'll have to use a migration toolkit such as I plan to release soon. The componentized nature of Datomic makes vertical scaling fairly pleasant as you can more precisely target what you're upgrading/improving. You probably want to include a pass-through querying API in your peers accessing Datomic so that you don't have to run the REST service. This'll enable access to the data from a long-tail of servers that don't necessarily need to be that close to the data and are performing dumb queries. This solves the "but my webapp needs 100 crappy VPSes to perform!" problem. Any kind of heavy-duty aggregation/analytical workloads against a Datomic dataset should be performed on a big peer with memcached. Use roll-ups for pete's sake! You can use a bi-temporal timeline with analytics data but I haven't fully explored the implications beyond Datomic making querying along that dimension nicer. The Doc Brown array store presented at the Clojure meetup last night is a very interesting option for people doing multi-dimensional slicing of analytics data. I'd personally use Datomic as a stand-in for any SQL database doing an OLTP workload, where previously I would've used PostgreSQL. Especially if history or reproducible results are important. That having been said, it appears to be adaptable to workloads I wouldn't have expected (analytics).
rbolkey 13y ago
I'm interested in what and how quick the development/testing cycle is. How quickly can you spin up, populate, and destroy a test instance for integration testing? How do you iterate on feature development against it?
- coolsunglasses 13y ago
  Super fast, better than any SQL database or MongoDB. Datomic (free and pro) has an "in memory" version that I used when developing my Clojure application. Makes firing off test-cases super simple and nice. I used fixtures (edn) for the schema and for example data as well. I would reload/dump/interact with the in-memory database in my REPL directly. Quite nice. You can run a test instance (free/dev, which uses H2), that's no big deal to setup either, same as running a local MongoDB or PostgreSQL instance. Ease-of-configuration is on par with MongoDB but with sane defaults. It "just works". I use environment variables for configuration, so the default Datomic database is just an in-memory instance, but I can point it to a localhost free instance easily. Whether I use a free instance or an in-memory instance depends on whether or not I need the full transaction log. Typically no, but the migration toolkit I'm working on requires a persistent database so the defaults there are different for testing/development. I'm seriously in love with a database that has a embedded and "real world" modes of operation with parity in their functionality (except for the log).
coolsunglasses 13y ago
I posted my migration toolkit Brambling to HN. You can find it here: https://github.com/bitemyapp/brambling/ https://github.com/bitemyapp/brambling/

dustingetz 13y ago

Can you ramble about how datomic is from ops perspective for a moment if you will? thanks

coolsunglasses 13y ago
Just to clarify, I don't know what problems sgrove ran into, I've asked him in IRC and will elaborate if he tells me, but at present, I dunno. So here's what I have encountered: Datomic's primary limitation is at the transactor level. Storage is a non-issue, especially if you're using DynamoDB. Even with PostgreSQL write-throughput was ridic because of the way they're using the database. Transactor fail-over I'm not sure I'd take too seriously, but I don't think they'll drop any writes on the floor either. You shouldn't attempt to run a datomic transactor on a cheapo VPS. Try to put together at least 4 gb of RAM. Which is the same as what's currently standard for consumer laptops. No excuses. Deploying Datomic itself is straight-forward. The config is super simple and all you need as a dependency is Java. I'm actually a little puzzled that people want "Datomic as a Service". Datomic and PostgreSQL were the simplest parts of my stack to deploy. You want beefy peers as well so as to make maximum use of the peer limitations on the license. This isn't as big of a deal as people think because you're not structuring your app the way Rails and Django apps do on Heroku. You don't do 1 beefy database + 5,000 terrible VPSes. Not only is that a management overhead (automation or not), but you're better off vertically scaling a couple peers first before running a shitload of dumb VPSes running your service. The running assumption is that you should use DynamoDB, but PostgreSQL was seriously efficient for our purposes. Indexing adds write overhead, but given the current limitations on what you can change in the schema online (nothing), you want to plan ahead here. If you fail to anticipate something in your schema, you'll have to use a migration toolkit such as I plan to release soon. The componentized nature of Datomic makes vertical scaling fairly pleasant as you can more precisely target what you're upgrading/improving. You probably want to include a pass-through querying API in your peers accessing Datomic so that you don't have to run the REST service. This'll enable access to the data from a long-tail of servers that don't necessarily need to be that close to the data and are performing dumb queries. This solves the "but my webapp needs 100 crappy VPSes to perform!" problem. Any kind of heavy-duty aggregation/analytical workloads against a Datomic dataset should be performed on a big peer with memcached. Use roll-ups for pete's sake! You can use a bi-temporal timeline with analytics data but I haven't fully explored the implications beyond Datomic making querying along that dimension nicer. The Doc Brown array store presented at the Clojure meetup last night is a very interesting option for people doing multi-dimensional slicing of analytics data. I'd personally use Datomic as a stand-in for any SQL database doing an OLTP workload, where previously I would've used PostgreSQL. Especially if history or reproducible results are important. That having been said, it appears to be adaptable to workloads I wouldn't have expected (analytics).

rbolkey 13y ago

I'm interested in what and how quick the development/testing cycle is. How quickly can you spin up, populate, and destroy a test instance for integration testing? How do you iterate on feature development against it?

coolsunglasses 13y ago
Super fast, better than any SQL database or MongoDB. Datomic (free and pro) has an "in memory" version that I used when developing my Clojure application. Makes firing off test-cases super simple and nice. I used fixtures (edn) for the schema and for example data as well. I would reload/dump/interact with the in-memory database in my REPL directly. Quite nice. You can run a test instance (free/dev, which uses H2), that's no big deal to setup either, same as running a local MongoDB or PostgreSQL instance. Ease-of-configuration is on par with MongoDB but with sane defaults. It "just works". I use environment variables for configuration, so the default Datomic database is just an in-memory instance, but I can point it to a localhost free instance easily. Whether I use a free instance or an in-memory instance depends on whether or not I need the full transaction log. Typically no, but the migration toolkit I'm working on requires a persistent database so the defaults there are different for testing/development. I'm seriously in love with a database that has a embedded and "real world" modes of operation with parity in their functionality (except for the log).

coolsunglasses 13y ago

I posted my migration toolkit Brambling to HN. You can find it here: https://github.com/bitemyapp/brambling/ https://github.com/bitemyapp/brambling/