4 ms·

How Prismatic deals with data storage and aggregation

69 points by snippyhollow 15y ago

dmix 15y ago
Will Prismatic be releasing any clojure libraries as open source?
- aria 15y ago
  Yes. We don't have a definite date, but soon.
  - tborg 15y ago
    I can't wait. I've been working with a bunch of XML files (TEI corpora, actually), and using document stores to build out indexes. One of the things that I've found is that what I want to do with reduce functions (build complex dictionaries, for instance with the key being a word stem and value being another dict with key=word form and value=list of locations or related words or whatever) is not the kind of thing Mongo/Couch DB reduce functions are designed for (i.e. scalars). I learned what a 'reduce' is from learning Clojure (which was my intro to functional programming), not from databases. And so I got really used to the idea of reducing into nested dictionaries, because Clojure is good at that. Which meant that when I came to using MapReduce in DB environments (particularly CouchDB, but also a lil' Hadoop), I was a little disappointed. Which is to say that a solid Clojuric abstraction layer that emphasizes a bucket approach on top of a variety of different stores sounds like my dream library.
willf 15y ago
Just a small typo: the 'associative' link links to the Wikipedia article on communitivity.
- aria 15y ago
  fixed.thanks for noticing.
cleancode1 15y ago
Are you going to release a public API?
frisco 15y ago
While cool, and interesting, this strikes me as the exact type of overengineering tech-heavy startups kept being told to stop doing. I get the argument that it'll "give you a platform to build on" and help you "move fast later," but how much did you spend on engineering a storage library that doesn't solve a critical business need? $10k? $20k? How many people spent hours and hours on building another Clojure-based abstraction layer over key-value stores? I also don't buy the argument that it'll help you be really "scalable". Once your datasets no longer fit in memory, it's all custom. Hadoop is a great solution if you have infinite money. Data locality matters. All those different services may expose a consistent-enough interface that you can build a common abstraction over them; their latency and reliability properties, failure modes, consistency, etc are not homogeneous, and at scale you can't pretend that they are. If your data fits in memory today (and that means 144g - 192g per machine), then why are you worrying about this? You'll need to rewrite a huge part of your infrastructure to scale from 100k users to 10M users anyway. TL;DR: solve the problem first and abstract out the framework later. Also "scalability" is incredibly expensive (and unnecessary) to engineer for a priori.
- aria 15y ago
  I'm not quite sure you got the gist of the post. The library isn't about scalability. It's about having a consistent way to describe how data sources accumulate/aggregate new values to do something useful with them. We don't have a batch MapReduce problem and so Hadoop isn't really an option and, having used it in the past, not sure it's even a great choice. Also, most of the data we're working with comes from content being shared and not number of users, so for our case in particular, dealing with a lot of data is an issue from day one. Lastly, " solve the problem first and abstract out the framework later" I think is one of the core misconceptions in computer science and engineering. Abstraction lets you build faster if done correctly. Again, if we only needed a simple key-value store and only had to deal with managing user data/events, we have built this kind of library out. Many of the reasons we built this library is because of the kinds of data aggregation we encounter in relevance ranking and machine learning.
  - frisco 15y ago
aidenn0 15y ago
Are the merge operations performed within a transaction for the backends that support them, or is atomicity of merges not desired?
davekale 15y ago
Looking forward to the libraries' release!
salimmadjd 15y ago
This is your second post, and you've already become one of my favorites. Have a nlp question for you. I noticed you're doing: {"Prismatic" {"runs" 1},"runs" {"on" 1}, "on" {"coffee" 1}} Why do you care to have a count of each word that follows and not assume its always one. Why not just analyze a simple bi-gram as key and an int to keep the count as value?

69 points by snippyhollow 15y ago

dmix 15y ago

Will Prismatic be releasing any clojure libraries as open source?

aria 15y ago
Yes. We don't have a definite date, but soon.
- tborg 15y ago
  I can't wait. I've been working with a bunch of XML files (TEI corpora, actually), and using document stores to build out indexes. One of the things that I've found is that what I want to do with reduce functions (build complex dictionaries, for instance with the key being a word stem and value being another dict with key=word form and value=list of locations or related words or whatever) is not the kind of thing Mongo/Couch DB reduce functions are designed for (i.e. scalars). I learned what a 'reduce' is from learning Clojure (which was my intro to functional programming), not from databases. And so I got really used to the idea of reducing into nested dictionaries, because Clojure is good at that. Which meant that when I came to using MapReduce in DB environments (particularly CouchDB, but also a lil' Hadoop), I was a little disappointed. Which is to say that a solid Clojuric abstraction layer that emphasizes a bucket approach on top of a variety of different stores sounds like my dream library.

willf 15y ago

Just a small typo: the 'associative' link links to the Wikipedia article on communitivity.

aria 15y ago
fixed.thanks for noticing.

cleancode1 15y ago

Are you going to release a public API?

frisco 15y ago

While cool, and interesting, this strikes me as the exact type of overengineering tech-heavy startups kept being told to stop doing. I get the argument that it'll "give you a platform to build on" and help you "move fast later," but how much did you spend on engineering a storage library that doesn't solve a critical business need? $10k? $20k? How many people spent hours and hours on building another Clojure-based abstraction layer over key-value stores? I also don't buy the argument that it'll help you be really "scalable". Once your datasets no longer fit in memory, it's all custom. Hadoop is a great solution if you have infinite money. Data locality matters. All those different services may expose a consistent-enough interface that you can build a common abstraction over them; their latency and reliability properties, failure modes, consistency, etc are not homogeneous, and at scale you can't pretend that they are. If your data fits in memory today (and that means 144g - 192g per machine), then why are you worrying about this? You'll need to rewrite a huge part of your infrastructure to scale from 100k users to 10M users anyway. TL;DR: solve the problem first and abstract out the framework later. Also "scalability" is incredibly expensive (and unnecessary) to engineer for a priori.

aria 15y ago
I'm not quite sure you got the gist of the post. The library isn't about scalability. It's about having a consistent way to describe how data sources accumulate/aggregate new values to do something useful with them. We don't have a batch MapReduce problem and so Hadoop isn't really an option and, having used it in the past, not sure it's even a great choice. Also, most of the data we're working with comes from content being shared and not number of users, so for our case in particular, dealing with a lot of data is an issue from day one. Lastly, " solve the problem first and abstract out the framework later" I think is one of the core misconceptions in computer science and engineering. Abstraction lets you build faster if done correctly. Again, if we only needed a simple key-value store and only had to deal with managing user data/events, we have built this kind of library out. Many of the reasons we built this library is because of the kinds of data aggregation we encounter in relevance ranking and machine learning.
- frisco 15y ago

aidenn0 15y ago

Are the merge operations performed within a transaction for the backends that support them, or is atomicity of merges not desired?

davekale 15y ago

Looking forward to the libraries' release!

salimmadjd 15y ago

This is your second post, and you've already become one of my favorites. Have a nlp question for you. I noticed you're doing: {"Prismatic" {"runs" 1},"runs" {"on" 1}, "on" {"coffee" 1}} Why do you care to have a count of each word that follows and not assume its always one. Why not just analyze a simple bi-gram as key and an int to keep the count as value?