5 ms·

Software Engineering at Prismatic

87 points by fogus 15y ago

plam 15y ago
where's the rss feed to subscribe to?
- aria 15y ago
  I'll put that up right now. Sorry it's a very new blog. Here's a direct link http://blog.getprismatic.com/blog/rss.xml http://blog.getprismatic.com/blog/rss.xml
  - spariev 15y ago
    Any chance of getting back contents of Bradford's old blog http://measuringmeasures.com/ http://measuringmeasures.com/ ? There were a few brilliant articles on becoming data scientist, learning statistics etc. It would be much appreciated.
    - gtani 15y ago
      that was a good one: http://web.archive.org/web/20101102120728/http://measuringmeasures.com/blog/2010/3/12/learning-about-machine-learning-2nd-ed.html http://web.archive.org/web/20101102120728/http://measuringme...
  - siavosh 15y ago
    Out of curiosity, is clojure used to do the front facing web components of prismatic as well or only the data/computation work?
    - dustingetz 15y ago
      this is an excellent question. i personally know two clojure teams, both use rails for their front-facing website - i believe they feel clojure's offerings aren't mature enough yet. sorry, i can't elaborate because I don't have any personal experience.
davekale 15y ago
Great post. Very thought-provoking.
sorenmacbeth 15y ago
nerd crushing so hard this team and their work.
programnature 15y ago
Very much looking forward to the rest of the posts, and the release of these projects :)
mattrepl 15y ago
The CEO, bradfordcross, is amazing at distilling good engineering to principles. Software engineering for hackers.
jshen 15y ago
"While we make heavy use of the core of Clojure, we don't use its concurrency primitives (atoms, refs, STM, etc.) because a function like pmap doesn't have enough fine grained control for our needs. We opt instead to build our own concurrency abstractions in Clojure on top of the outstanding java.util.concurrent package." I wrote a web crawler in clojure at at&t and our team came to this same conclusion. First we tried to implement a custom atom, but finally using java.util.concurrent directly. I'm not sure what this says about the clojure concurrency features, but I've found myself using them sparingly even when doing a lot of concurrency.
- eternalban 15y ago
  Rich Hickey is borderline genius in my book, and I really enjoy his philosophical approach to CS. Clojure is a fine language, but I frankly never bought the "concurrency issues" as a reasonable motivation for Clojure (over Java). Fact remains that STMs (as originally developed by the Scala team in (yes) Java), "outstanding" [OP] concurrency, and idempotent/functional programming remains /well within reach/ of the seasoned developer in Java. Also, unless I misunderstand their architecture (and it is single node ..), then the fact remains that in a distributed system, the "semantics" of interaction are not bound by the programming language behind box x, and are entirely a matter of the (remote) API you expose. STMs are a concern of a shared memory space -- once you go distributed, it is really borderline snakeoil to talk about the benefits of "functional" languages in creating remote functions.
  - jshen 15y ago
    Our system was both distributed and highly concurrent in the same memory space. We started off using atoms, but we needed visibility and control over the queue feeding the thread for the atom which lead us to using java.util.concurrent directly. Our use case was fairly odd though.
- swannodette 15y ago
  For people outside the community this may seem like a surprise. But Rich Hickey has been saying for a long time now that Clojure explicitly does not try to emulate many of the tools found in java.util.concurrent for the very reason that they are quite good.
noelwelsh 15y ago
I'm most interested to know about the Flop library. This has always seemed a weak point of the JVM ecosystem. While there are many numeric libraries none seems dominant and speed suffers next to BLAS. Of course programming and using such a library in Java is also painful. This is a domain where functions really are by definition the right abstraction!
davegauer 15y ago
"We tend to build our own libraries simply because much of the functionality of open-source code is locked up into monolithic frameworks and not easily re-usable." Wow, I couldn't agree more with this concept of library vs. framework. While frameworks provide all kinds of up-front time savings, I've never used one that didn't end up getting in my way and require all kinds of deeply placed hacks to work how I (or my company/client) wanted. I sometimes feel out of the loop because I'm not using all of the popular frameworks everyone else is talking about at the moment. So it's refreshing to hear I'm not alone in the opinion that time is saved in the long run by writing one's own small, modular, easily understood and reused libraries if none are already available. I do feel as though small, self-contained code bases are becoming easier to find with the rise in popularity of sites such as GitHub and Bitbucket. So perhaps the trend towards less monolithic software can become more mainstream.
- gruseom 15y ago
  I had the same reaction. Fully agree with both you and the OP.

87 points by fogus 15y ago

plam 15y ago

where's the rss feed to subscribe to?

aria 15y ago
I'll put that up right now. Sorry it's a very new blog. Here's a direct link http://blog.getprismatic.com/blog/rss.xml http://blog.getprismatic.com/blog/rss.xml
- spariev 15y ago
  Any chance of getting back contents of Bradford's old blog http://measuringmeasures.com/ http://measuringmeasures.com/ ? There were a few brilliant articles on becoming data scientist, learning statistics etc. It would be much appreciated.
  - gtani 15y ago
    that was a good one: http://web.archive.org/web/20101102120728/http://measuringmeasures.com/blog/2010/3/12/learning-about-machine-learning-2nd-ed.html http://web.archive.org/web/20101102120728/http://measuringme...
- siavosh 15y ago
  Out of curiosity, is clojure used to do the front facing web components of prismatic as well or only the data/computation work?
  - dustingetz 15y ago
    this is an excellent question. i personally know two clojure teams, both use rails for their front-facing website - i believe they feel clojure's offerings aren't mature enough yet. sorry, i can't elaborate because I don't have any personal experience.

davekale 15y ago

Great post. Very thought-provoking.

sorenmacbeth 15y ago

nerd crushing so hard this team and their work.

programnature 15y ago

Very much looking forward to the rest of the posts, and the release of these projects :)

mattrepl 15y ago

The CEO, bradfordcross, is amazing at distilling good engineering to principles. Software engineering for hackers.

jshen 15y ago

"While we make heavy use of the core of Clojure, we don't use its concurrency primitives (atoms, refs, STM, etc.) because a function like pmap doesn't have enough fine grained control for our needs. We opt instead to build our own concurrency abstractions in Clojure on top of the outstanding java.util.concurrent package." I wrote a web crawler in clojure at at&t and our team came to this same conclusion. First we tried to implement a custom atom, but finally using java.util.concurrent directly. I'm not sure what this says about the clojure concurrency features, but I've found myself using them sparingly even when doing a lot of concurrency.

eternalban 15y ago
Rich Hickey is borderline genius in my book, and I really enjoy his philosophical approach to CS. Clojure is a fine language, but I frankly never bought the "concurrency issues" as a reasonable motivation for Clojure (over Java). Fact remains that STMs (as originally developed by the Scala team in (yes) Java), "outstanding" [OP] concurrency, and idempotent/functional programming remains /well within reach/ of the seasoned developer in Java. Also, unless I misunderstand their architecture (and it is single node ..), then the fact remains that in a distributed system, the "semantics" of interaction are not bound by the programming language behind box x, and are entirely a matter of the (remote) API you expose. STMs are a concern of a shared memory space -- once you go distributed, it is really borderline snakeoil to talk about the benefits of "functional" languages in creating remote functions.
- jshen 15y ago
  Our system was both distributed and highly concurrent in the same memory space. We started off using atoms, but we needed visibility and control over the queue feeding the thread for the atom which lead us to using java.util.concurrent directly. Our use case was fairly odd though.
swannodette 15y ago
For people outside the community this may seem like a surprise. But Rich Hickey has been saying for a long time now that Clojure explicitly does not try to emulate many of the tools found in java.util.concurrent for the very reason that they are quite good.

noelwelsh 15y ago

I'm most interested to know about the Flop library. This has always seemed a weak point of the JVM ecosystem. While there are many numeric libraries none seems dominant and speed suffers next to BLAS. Of course programming and using such a library in Java is also painful. This is a domain where functions really are by definition the right abstraction!

davegauer 15y ago

"We tend to build our own libraries simply because much of the functionality of open-source code is locked up into monolithic frameworks and not easily re-usable." Wow, I couldn't agree more with this concept of library vs. framework. While frameworks provide all kinds of up-front time savings, I've never used one that didn't end up getting in my way and require all kinds of deeply placed hacks to work how I (or my company/client) wanted. I sometimes feel out of the loop because I'm not using all of the popular frameworks everyone else is talking about at the moment. So it's refreshing to hear I'm not alone in the opinion that time is saved in the long run by writing one's own small, modular, easily understood and reused libraries if none are already available. I do feel as though small, self-contained code bases are becoming easier to find with the rise in popularity of sites such as GitHub and Bitbucket. So perhaps the trend towards less monolithic software can become more mainstream.

gruseom 15y ago
I had the same reaction. Fully agree with both you and the OP.