4 ms·

Thanks for some very spot on points. I do mention multiple processes in the blog: "Fortunately not all problems require concurrent solutions as they are either

by gislik 16y ago

Thanks for some very spot on points. I do mention multiple processes in the blog: "Fortunately not all problems require concurrent solutions as they are either IO-bound or can be scaled by forking multiple processes." And this can be a solution to many of the scaling needs. Some of the challenges I'm facing are however more CPU oriented and it gives me a warm feeling to have a concurrency model like STM to lean on. I totally agree with you on the point regarding worrying about delivering a product instead of rewiting the work. That's why the services already written will remain in place and I'll there fore be running Python alongside the Clojure/JVM stack.

jedsmith 16y ago
> I'll there fore be running Python alongside the Clojure/JVM stack. Yikes, really? That sounds even worse than a clean break. You don't shed your perceived problems with Python and instead get to manage two codebases and stacks. Your operations team will absolutely love you in the future. Based upon a casual read of what your service does, I'm really stressing to think of a CPU-bound situation that can't be pooled into a multiprocessing pool. All of the heavy work I can think of your service doing -- particularly crawling -- is going to end up network or I/O-bound, isn't it? I'm coming up empty and working with a theory. Perhaps you can share what your CPU-bound process is?
- gislik 16y ago
  The Python services probably end up getting rewritten at some point but I want to focus on delivering a product. Right now the Python services can easily be deployed as the don't have that many dependencies on other libs. All further development will be done in Clojure though. Of course most things can bee pooled into a multiprocessing pool. It's just a matter of the pain you have to endure while doing so. For some tasks it's a straight forward process while for other tasks it can be pretty painful. The crawler is IO-bound but most of the collaborative filtering algorithms are CPU-bound and having nice concurrency constructions in the language is a bonus.
- dkarl 16y ago
  It's wise of them to keep the old solution around while they're experimenting with a new technology. At some point they'll have an urgent need to deploy something that they can't figure out how to do on the Java stack, and they'll be able to fall back on their Python expertise to get it done by the deadline.

by gislik 16y ago

jedsmith 16y ago

> I'll there fore be running Python alongside the Clojure/JVM stack. Yikes, really? That sounds even worse than a clean break. You don't shed your perceived problems with Python and instead get to manage two codebases and stacks. Your operations team will absolutely love you in the future. Based upon a casual read of what your service does, I'm really stressing to think of a CPU-bound situation that can't be pooled into a multiprocessing pool. All of the heavy work I can think of your service doing -- particularly crawling -- is going to end up network or I/O-bound, isn't it? I'm coming up empty and working with a theory. Perhaps you can share what your CPU-bound process is?

gislik 16y ago
The Python services probably end up getting rewritten at some point but I want to focus on delivering a product. Right now the Python services can easily be deployed as the don't have that many dependencies on other libs. All further development will be done in Clojure though. Of course most things can bee pooled into a multiprocessing pool. It's just a matter of the pain you have to endure while doing so. For some tasks it's a straight forward process while for other tasks it can be pretty painful. The crawler is IO-bound but most of the collaborative filtering algorithms are CPU-bound and having nice concurrency constructions in the language is a bonus.
dkarl 16y ago
It's wise of them to keep the old solution around while they're experimenting with a new technology. At some point they'll have an urgent need to deploy something that they can't figure out how to do on the Java stack, and they'll be able to fall back on their Python expertise to get it done by the deadline.