6 ms·

I think most people don't know that Datomic can be seen as a layer between your code/date and nearly any backend (including SQL: for example you can use Postgre

I think most people don't know that Datomic can be seen as a layer between your code/date and nearly any backend (including SQL: for example you can use PostgreSQL "behind" Datomic). "CRA" (Create Read Append) is the future for 99% of all the application out there IMNSHO. I know most people don't believe it but you're simply not coming back once you understand how easy it is to "recreate the state" using such a tool. In addition to CRA, the fact that queries are forever cacheable is huge. The (lifelong) problem of "cache invalidation" simply doesn't exist anymore. I'm not saying that Clojure is going to replace Java/C# or that Datomic is going to be used everywhere. But the concepts at the heart of these (like, say, lockless concurrency in Clojure and the "grow-only" property of Datomic) are what the future's gonna be made of.

bretthoerner 14y ago
I mostly agree with your sentiment, but, > I think most people don't know that Datomic can be seen as a layer between your code/date and nearly any backend (including SQL: for example you can use PostgreSQL "behind" Datomic). While this is true, it doesn't matter. That Datomic is backed with Postgres doesn't matter to me unless I can query it with all the same functionality as I do Posgtres. Which I can't. It's just a dumb store (like all the backends), so this is an meaningless feature except to people running operations.
est 14y ago
> "CRA" (Create Read Append) is the future for 99% of all the application out there IMNSHO How about counters e.g. number of upvotes ? It's typical upsert operations but needs powerful aggression for sorting.
- grayrest 14y ago
  Nathan Marz has been promoting this as the Lambda Architecture, has a couple presentations/blog posts about the idea and is writing a book. While other people probably have similar ideas, I'm unaware of anybody else attempting to teach them in a similarly cohesive manner. I don't know what I'm doing but I'm working my way through understanding this area so I'll work through the queries as an exercise. For your counter, you'd record the individual votes as tuples of (timestamp, voter, vote, subject). These get dumped into a distributed data store. From there, they get batch processed into the equivalent of a database view. The cascalog query for your 'posts' view would be something like: (defn post-view [post-source vote-source output] (?<- output [?timestamp ?postid ?upvotes ?downvotes ?title ?body] (post-source ?timestamp ?postid ?title ?body) (vote-source _ _ ?vote ?postid) (> ?vote 0 :> !up) (c/!count !up :> ?upvotes) (< ?vote 0 :> !down) (c/!count !down :> ?downvotes))) Produces a set of (timestamp, id, timestamp, upvotes, downvotes, title, body) tuples. You can then do another pass on it to run your sort function: (defn post-score [now-ms time-ms upvotes downvotes] (let [mins-since (div (- now-ms time-ms) (* 60 1000)) (div (- upvotes downvotes) (+ mins-since 1))])) (defn post-rank [post-view-source output] (?<- output [?postid ?votes ?title] (post-view-source ?timestamp ?postid ?upvotes ?downvotes ?title _) (- ?upvotes ?downvotes :> ?votes) (post-score (System/currentTimeMillis) ?timestamp ?upvotes ?downvotes :> ?score) (:sort ?score) (:reverse true)) This would give (id, vote total, title) tuples that you can send to your templating engine (assuming url is based entirely on id) to make the front page. The neat thing about the approach is that you can change everything later. If, for example, you wanted to do weighted upvotes/downvotes, you could produce a scoring function for the up/down votes like the one used here for scoring posts and aggregate the votes using sum instead of count.
  - est 14y ago
    That's cool thanks! But for a counter of millions, count vote action one by one seems inefficient. I think we still need some sort of `update` mechanism in a pure CRA db?
    - grayrest 14y ago
      It's not particularly efficient but that's not a design goal of the system. The tradeoff is robustness–can't lose or corrupt data as long as the master data set is safe–and flexibility–generate whatever views you like on the data whenever you choose. The design is Twitter's analytics system which was running this sort of thing over a 27TB raw dataset using Hadoop so apparently it scales if you throw more hardware at it. There's a second layer using Storm (not written up in the book yet, so I don't know the details) that handles all data newer than the most recent batch run and you somehow merge that new data with the old data (also not written up yet). I don't need to have this sort of system implemented immediately so I'm content to sit around and wait for new book chapters rather than try to muddle through.
    - Tuna-Fish 14y ago
      If you don't count vote action one by one, how do you prevent multiple votes by same user?

bretthoerner 14y ago

I mostly agree with your sentiment, but, > I think most people don't know that Datomic can be seen as a layer between your code/date and nearly any backend (including SQL: for example you can use PostgreSQL "behind" Datomic). While this is true, it doesn't matter. That Datomic is backed with Postgres doesn't matter to me unless I can query it with all the same functionality as I do Posgtres. Which I can't. It's just a dumb store (like all the backends), so this is an meaningless feature except to people running operations.

est 14y ago

> "CRA" (Create Read Append) is the future for 99% of all the application out there IMNSHO How about counters e.g. number of upvotes ? It's typical upsert operations but needs powerful aggression for sorting.

grayrest 14y ago
Nathan Marz has been promoting this as the Lambda Architecture, has a couple presentations/blog posts about the idea and is writing a book. While other people probably have similar ideas, I'm unaware of anybody else attempting to teach them in a similarly cohesive manner. I don't know what I'm doing but I'm working my way through understanding this area so I'll work through the queries as an exercise. For your counter, you'd record the individual votes as tuples of (timestamp, voter, vote, subject). These get dumped into a distributed data store. From there, they get batch processed into the equivalent of a database view. The cascalog query for your 'posts' view would be something like: (defn post-view [post-source vote-source output] (?<- output [?timestamp ?postid ?upvotes ?downvotes ?title ?body] (post-source ?timestamp ?postid ?title ?body) (vote-source _ _ ?vote ?postid) (> ?vote 0 :> !up) (c/!count !up :> ?upvotes) (< ?vote 0 :> !down) (c/!count !down :> ?downvotes))) Produces a set of (timestamp, id, timestamp, upvotes, downvotes, title, body) tuples. You can then do another pass on it to run your sort function: (defn post-score [now-ms time-ms upvotes downvotes] (let [mins-since (div (- now-ms time-ms) (* 60 1000)) (div (- upvotes downvotes) (+ mins-since 1))])) (defn post-rank [post-view-source output] (?<- output [?postid ?votes ?title] (post-view-source ?timestamp ?postid ?upvotes ?downvotes ?title _) (- ?upvotes ?downvotes :> ?votes) (post-score (System/currentTimeMillis) ?timestamp ?upvotes ?downvotes :> ?score) (:sort ?score) (:reverse true)) This would give (id, vote total, title) tuples that you can send to your templating engine (assuming url is based entirely on id) to make the front page. The neat thing about the approach is that you can change everything later. If, for example, you wanted to do weighted upvotes/downvotes, you could produce a scoring function for the up/down votes like the one used here for scoring posts and aggregate the votes using sum instead of count.
- est 14y ago
  That's cool thanks! But for a counter of millions, count vote action one by one seems inefficient. I think we still need some sort of `update` mechanism in a pure CRA db?
  - grayrest 14y ago
    It's not particularly efficient but that's not a design goal of the system. The tradeoff is robustness–can't lose or corrupt data as long as the master data set is safe–and flexibility–generate whatever views you like on the data whenever you choose. The design is Twitter's analytics system which was running this sort of thing over a 27TB raw dataset using Hadoop so apparently it scales if you throw more hardware at it. There's a second layer using Storm (not written up in the book yet, so I don't know the details) that handles all data newer than the most recent batch run and you somehow merge that new data with the old data (also not written up yet). I don't need to have this sort of system implemented immediately so I'm content to sit around and wait for new book chapters rather than try to muddle through.
  - Tuna-Fish 14y ago
    If you don't count vote action one by one, how do you prevent multiple votes by same user?