6 ms·

I won't commit HN suicide by questioning his choice of Clojure (edit: the downvotes and only positive replies are interesting, though), but I will focus on the

I won't commit HN suicide by questioning his choice of Clojure (edit: the downvotes and only positive replies are interesting, though), but I will focus on the original impetus: it's interesting to me that a few problems are explained in Python, including solutions of how to fix them, but the author doesn't want to use the solutions. Honestly, it sounds like he wrote off virtualenv without trying it, because he'd have far less complaints about porting and dependencies if he had. As for the GIL, the author didn't even consider multiple processes -- he explained them away as a solution for some workloads. A Web app is one giant workload where this model makes sense: multiprocessing is the approach you should be taking with a Web application. Tie a request to one core, and spawn enough WSGI applications for the number of cores you have (and then some). That's an elegant solution to this problem, since a request doesn't need to fart around with other requests in most cases. If your app can't handle multiple copies of itself running, how do you expect it to scale? This reads a little bit like a waffling, like a person who considers a completely different environment and rewrite of the entire stack a solution to some thorns in Python. I'm all for picking an alternative Web stack, but: > I had written a database loader to import Apple’s Enterprise Partner Feed (EPF) and a web crawler in Python and next up was the web interface. > It all seemed like a smooth sailing but in the back of my head I was beginning to have doubts about my decisions. > Why we are choosing Clojure as our main programming language This being the third blog post for an as-yet-unreleased product, I'd say worry about delivering a product instead of justifying a rewrite of your work thus far on your blog. If I were considering funding your startup, this blog post would be a fairly bad sign to me. At any rate, a computer language is just a tool to implement an idea, and focusing strongly on the language of choice is busywork itself.

technomancy 16y ago
Starting a "Why we choose X" post with weak token "here's why W, Y, and Z are inappropriate" blurbs can't be chalked up to anything but flamebait.
- reinhardt 16y ago
  Or just fanboyism and shiny-new-silver-bullet-ism.
deleted 16y ago
[deleted]
gislik 16y ago
Thanks for some very spot on points. I do mention multiple processes in the blog: "Fortunately not all problems require concurrent solutions as they are either IO-bound or can be scaled by forking multiple processes." And this can be a solution to many of the scaling needs. Some of the challenges I'm facing are however more CPU oriented and it gives me a warm feeling to have a concurrency model like STM to lean on. I totally agree with you on the point regarding worrying about delivering a product instead of rewiting the work. That's why the services already written will remain in place and I'll there fore be running Python alongside the Clojure/JVM stack.
- jedsmith 16y ago
  > I'll there fore be running Python alongside the Clojure/JVM stack. Yikes, really? That sounds even worse than a clean break. You don't shed your perceived problems with Python and instead get to manage two codebases and stacks. Your operations team will absolutely love you in the future. Based upon a casual read of what your service does, I'm really stressing to think of a CPU-bound situation that can't be pooled into a multiprocessing pool. All of the heavy work I can think of your service doing -- particularly crawling -- is going to end up network or I/O-bound, isn't it? I'm coming up empty and working with a theory. Perhaps you can share what your CPU-bound process is?
  - gislik 16y ago
    The Python services probably end up getting rewritten at some point but I want to focus on delivering a product. Right now the Python services can easily be deployed as the don't have that many dependencies on other libs. All further development will be done in Clojure though. Of course most things can bee pooled into a multiprocessing pool. It's just a matter of the pain you have to endure while doing so. For some tasks it's a straight forward process while for other tasks it can be pretty painful. The crawler is IO-bound but most of the collaborative filtering algorithms are CPU-bound and having nice concurrency constructions in the language is a bonus.
  - dkarl 16y ago
    It's wise of them to keep the old solution around while they're experimenting with a new technology. At some point they'll have an urgent need to deploy something that they can't figure out how to do on the Java stack, and they'll be able to fall back on their Python expertise to get it done by the deadline.
bad_user 16y ago
Also related to the GIL -- the "processing" module in Python takes care of any sharing scenarios you might have and it is mostly API-compatible with the threading API in Python. And if that's too heavy, there's always stuff like GEvent. Asynchronous I/O and multiprocessing solves 80% of all problems you may have. I do wish that Python / Ruby would be GIL-free, but for my startup I choose these platforms because they've got no parallel when it comes to rapid-development of web applications, because no matter how cool this and that language is, nothing beats the productivity gained by robust, battle-tested web frameworks (not to mention other stuff, like NLTK or SciPy). As an early-adopter myself I might try out a couple of projects in Clojure, but I wouldn't bet my business on it unless I saw a clear need that Clojure satisfies and that dwarfs all other disadvantages, like platform immaturity.
- F_J_H 16y ago
  Good points. After all, "shipping is a feature".
- swannodette 16y ago
  Having built websites in Ruby and Python, I've found Clojure to hold its own in terms of productivity. Clojure's library ecosystem is healthy and growing and you can tap into the battle-tested Java frameworks with ease. Saying that Asynchronous I/O and multiprocessing solves 80% of the problems is hand-waving away the complexity such approaches might add to a project. Clojure simply gives you more and better tools then Ruby and Python currently do for dealing with concurrency. You might not bet your business on Clojure, but certainly other people have for a couple years now and they don't seem to take issue with it. EDIT: removed the defensive bit. My experience, as this thread shows, is that people get touchy about the limitations of the GIL.
  - bad_user 16y ago
    Stating an opinion is not getting defensive. You can do whatever floats your boat. EDIT: RE: Don't take me wrong, I sometimes hate the limitations that the GIL brings. I'm continually looking at alternative implementations, like Pypy or Rubinius or JRuby, or other languages, like Haskell or Clojure, but then I end up doing a lot of yak shaving and nothing gets done.
ch0wn 16y ago
As web applications usually don't do much number crunching, the GIT is even more meaningless in that decision. Threading to avoid IO-blocking doesn't trigger the GIL at all and that is what you have to deal with in web apps.
- ch0wn 16y ago
  I meant to write GIL, obviously. Errors like this happen, when you type "git" about 200 times a day.
technomancy 16y ago
Relevant: Why I hate Advocacy: http://www.perl.com/pub/2000/12/advocacy.html http://www.perl.com/pub/2000/12/advocacy.html
nupark 16y ago
> As for the GIL, the author didn't even consider multiple processes ... I really hate this argument for its disingenuity. Some things (including message passing) are naturally fastest with shared memory. Multiple processes are not an equivalent substitute. Clojure, for instance, leverages shared memory to implement its excellent, lightweight STM and related high-level threading constructs, and shared memory is what allows Clojure to easily implement cheap, MVCC functional data structures. Multiple processes aren't an answer, here. To implement the same high-level constructs efficiently requires re-introducing shared memory through more complex, less efficient mechanisms, and often leaves the problem of sharing access to higher-level constructs (objects, instead of data) unsolved. Multiple processes are a poor work-around for a lack of support for concurrency, not a solution. They make sense if you're sandboxing, but make little sense for implementing a high-performance concurrent server. To pre-empt the erlang discussion -- erlang's message passing model does scale, but Erlang's runtime does require support for real shared memory concurrency in order to maximize single-machine performance.
- nl 16y ago
  > As for the GIL, the author didn't even consider multiple processes ... I really hate this argument for its disingenuity. Some things (including message passing) are naturally fastest with shared memory. Multiple processes are not an equivalent substitute. This isn't a disingenuous answer at all, since it goes on to point out that A Web app is one giant workload where this model makes sense: multiprocessing is the approach you should be taking with a Web application. As someone with a strong Java background, I have to continually force myself to be aware of my bias against concurrency-via-multiple-processes. If you have a stateless application (and if at all possible, statelessness is a good thing [1]), then multiple processes scale vertically reasonably well (on the same machine), but having a multiple-process architecture makes scaling horizontally (across machines) natural and easy. It's true that shared access to higher-level constructs in a multiple process environment isn't solved, but (a) shared access to data is an anti-pattern [2], and (b) Memcache plus some kind of serialization works pretty well when you do need shared data. [1] Yes, applications exist where statelessness doesn't make sense. In this case, though the author was originally developing on AppEngine, so I doubt there is much state in the app. [2] I'm not saying you should never have shared data. I am saying you should avoid it as much as possible.

technomancy 16y ago

Starting a "Why we choose X" post with weak token "here's why W, Y, and Z are inappropriate" blurbs can't be chalked up to anything but flamebait.

reinhardt 16y ago
Or just fanboyism and shiny-new-silver-bullet-ism.

deleted 16y ago

[deleted]

gislik 16y ago

Thanks for some very spot on points. I do mention multiple processes in the blog: "Fortunately not all problems require concurrent solutions as they are either IO-bound or can be scaled by forking multiple processes." And this can be a solution to many of the scaling needs. Some of the challenges I'm facing are however more CPU oriented and it gives me a warm feeling to have a concurrency model like STM to lean on. I totally agree with you on the point regarding worrying about delivering a product instead of rewiting the work. That's why the services already written will remain in place and I'll there fore be running Python alongside the Clojure/JVM stack.

jedsmith 16y ago
> I'll there fore be running Python alongside the Clojure/JVM stack. Yikes, really? That sounds even worse than a clean break. You don't shed your perceived problems with Python and instead get to manage two codebases and stacks. Your operations team will absolutely love you in the future. Based upon a casual read of what your service does, I'm really stressing to think of a CPU-bound situation that can't be pooled into a multiprocessing pool. All of the heavy work I can think of your service doing -- particularly crawling -- is going to end up network or I/O-bound, isn't it? I'm coming up empty and working with a theory. Perhaps you can share what your CPU-bound process is?
- gislik 16y ago
  The Python services probably end up getting rewritten at some point but I want to focus on delivering a product. Right now the Python services can easily be deployed as the don't have that many dependencies on other libs. All further development will be done in Clojure though. Of course most things can bee pooled into a multiprocessing pool. It's just a matter of the pain you have to endure while doing so. For some tasks it's a straight forward process while for other tasks it can be pretty painful. The crawler is IO-bound but most of the collaborative filtering algorithms are CPU-bound and having nice concurrency constructions in the language is a bonus.
- dkarl 16y ago
  It's wise of them to keep the old solution around while they're experimenting with a new technology. At some point they'll have an urgent need to deploy something that they can't figure out how to do on the Java stack, and they'll be able to fall back on their Python expertise to get it done by the deadline.

bad_user 16y ago

Also related to the GIL -- the "processing" module in Python takes care of any sharing scenarios you might have and it is mostly API-compatible with the threading API in Python. And if that's too heavy, there's always stuff like GEvent. Asynchronous I/O and multiprocessing solves 80% of all problems you may have. I do wish that Python / Ruby would be GIL-free, but for my startup I choose these platforms because they've got no parallel when it comes to rapid-development of web applications, because no matter how cool this and that language is, nothing beats the productivity gained by robust, battle-tested web frameworks (not to mention other stuff, like NLTK or SciPy). As an early-adopter myself I might try out a couple of projects in Clojure, but I wouldn't bet my business on it unless I saw a clear need that Clojure satisfies and that dwarfs all other disadvantages, like platform immaturity.

F_J_H 16y ago
Good points. After all, "shipping is a feature".
swannodette 16y ago
Having built websites in Ruby and Python, I've found Clojure to hold its own in terms of productivity. Clojure's library ecosystem is healthy and growing and you can tap into the battle-tested Java frameworks with ease. Saying that Asynchronous I/O and multiprocessing solves 80% of the problems is hand-waving away the complexity such approaches might add to a project. Clojure simply gives you more and better tools then Ruby and Python currently do for dealing with concurrency. You might not bet your business on Clojure, but certainly other people have for a couple years now and they don't seem to take issue with it. EDIT: removed the defensive bit. My experience, as this thread shows, is that people get touchy about the limitations of the GIL.
- bad_user 16y ago
  Stating an opinion is not getting defensive. You can do whatever floats your boat. EDIT: RE: Don't take me wrong, I sometimes hate the limitations that the GIL brings. I'm continually looking at alternative implementations, like Pypy or Rubinius or JRuby, or other languages, like Haskell or Clojure, but then I end up doing a lot of yak shaving and nothing gets done.

ch0wn 16y ago

As web applications usually don't do much number crunching, the GIT is even more meaningless in that decision. Threading to avoid IO-blocking doesn't trigger the GIL at all and that is what you have to deal with in web apps.

ch0wn 16y ago
I meant to write GIL, obviously. Errors like this happen, when you type "git" about 200 times a day.

technomancy 16y ago

Relevant: Why I hate Advocacy: http://www.perl.com/pub/2000/12/advocacy.html http://www.perl.com/pub/2000/12/advocacy.html

nupark 16y ago

> As for the GIL, the author didn't even consider multiple processes ... I really hate this argument for its disingenuity. Some things (including message passing) are naturally fastest with shared memory. Multiple processes are not an equivalent substitute. Clojure, for instance, leverages shared memory to implement its excellent, lightweight STM and related high-level threading constructs, and shared memory is what allows Clojure to easily implement cheap, MVCC functional data structures. Multiple processes aren't an answer, here. To implement the same high-level constructs efficiently requires re-introducing shared memory through more complex, less efficient mechanisms, and often leaves the problem of sharing access to higher-level constructs (objects, instead of data) unsolved. Multiple processes are a poor work-around for a lack of support for concurrency, not a solution. They make sense if you're sandboxing, but make little sense for implementing a high-performance concurrent server. To pre-empt the erlang discussion -- erlang's message passing model does scale, but Erlang's runtime does require support for real shared memory concurrency in order to maximize single-machine performance.

nl 16y ago
> As for the GIL, the author didn't even consider multiple processes ... I really hate this argument for its disingenuity. Some things (including message passing) are naturally fastest with shared memory. Multiple processes are not an equivalent substitute. This isn't a disingenuous answer at all, since it goes on to point out that A Web app is one giant workload where this model makes sense: multiprocessing is the approach you should be taking with a Web application. As someone with a strong Java background, I have to continually force myself to be aware of my bias against concurrency-via-multiple-processes. If you have a stateless application (and if at all possible, statelessness is a good thing [1]), then multiple processes scale vertically reasonably well (on the same machine), but having a multiple-process architecture makes scaling horizontally (across machines) natural and easy. It's true that shared access to higher-level constructs in a multiple process environment isn't solved, but (a) shared access to data is an anti-pattern [2], and (b) Memcache plus some kind of serialization works pretty well when you do need shared data. [1] Yes, applications exist where statelessness doesn't make sense. In this case, though the author was originally developing on AppEngine, so I doubt there is much state in the app. [2] I'm not saying you should never have shared data. I am saying you should avoid it as much as possible.