6 ms·

Solr is just so easy to get up and running, so powerful with so many options for scaling, I don't see the benefit of "pure Python" here. (To be fair, despite li

Solr is just so easy to get up and running, so powerful with so many options for scaling, I don't see the benefit of "pure Python" here. (To be fair, despite living and breathing Python most days, I've never thought "pure Python" is a selling point.)

ra 13y ago
Solr, Sphinx and ElasticSearch are all reliable search-at-scale options for those of us without petabyte problems. Whoosh is fine for very small datasets (megabytes) and low loads, just watch out for file permission issues.
- Torn 13y ago
  ElasticSearch is actually designed to scale to petabytes and beyond: http://elasticsearch.com/product/ http://elasticsearch.com/product/ ("petabytes of data with ease")
nknighthb 13y ago
Having had nothing but extremely negative experiences with production JVM deployments, anything being not-JVM-based is a huge selling point for me. Furthermore, "pure python" itself means that I can run it in any Python implementation I want, including pypy, without worrying about any C modules.
- koolkao 13y ago
  what kinds of JVM issues did you run into in deployment? for personal projects I have always used C++, Python, RoR, MEAN, etc. but at work where my role is non-dev, our software dev team likes Java and the heavy weighted GWT... frankly shutting me out. I'd love to know how to convince them to switch out of Java.
  - nknighthb 13y ago
    Memory issues out the ass. Which are obviously all in my head because there are no memory leaks in Java. I must be doing it wrong. Am I sure I know how to multiply? Of course you shouldn't be using version 1.2.3.4.5, our code only works with 1.2.3.4.4, everybody knows that. Well, except for that code, which of course you have to run on 1.2.3.4.4c. And this third-party daemon over here, that only runs on an obscure 6-year-old JVM and you have to put it in /var/deadchicken/ and insert three squirrels before executing. You won't convince them, they've already jumped off the cliff. It's all perfectly normal to them. If you want to do development, find a company that hasn't been contaminated yet.
    - robrenaud 13y ago
      I don't actually write Java professionally. But it sounds like your problem isn't Java/JVM, but rather really shitty code. If it breaks when you change minor versions of the JVM, the code is broken.
      - nknighthb 13y ago
        Just from my own personal experience, you've just indicted code from three well-known SV companies, at least two more non-SV fortune-500s, a mess of B2B vendors, the Apache foundation, and a few minor, independent open-source projects. I've heard stories from friends in a few other companies about both internal and third-party code. At some point you reach the conclusion that the problem with an ecosystem is more fundamental than a few lazy developers. But even if not, your statement isn't actionable. If all the code in a certain set happens to be really shitty, the only rational choice is to not use that code. Any other argument ends up being one only of semantics.
        jsmeaton 13y ago
        I think I've deployed only one java web app (puppet-db) that doesn't have huge memory leak issues. With the lack of process recycling in tomcat to mitigate "bad code", I cringe every time I have to deploy another one.
        ris 13y ago
- swasheck 13y ago
  All of the troubles that I've had have been relative to compiling the JVM Python code and linking it correctly to my Java binary at compile time. Once I've gotten that working, I've found Lucene to be quite simple - and faster than Whoosh!
randlet 13y ago
Pure Python is definitely a selling point for those of us who regularly deploy to multiple platforms. Knowing you can 'pip install pure-python-lib' on any platform and have it work every single time without regards to what c compilers and c libraries you have available is a major boon.
- davb 13y ago
  Or, more importantly (in my case), very easily bundle pure Python code with your application. I understand the arguments against it, but in any project that might produce deliverables that will be in play for more than a few weeks the first thing I do is create a "lib" folder where any third party dependencies live and get imported from. With Python this is usually very easy. I've maintained my fair share of legacy code and more often than not one of the biggest issues is tracking down ancient libraries which work with my code. There are very few packages I trust to maintain fairly stable interfaces and be available years into the future (things like Apache HTTPd, PostgreSQL and memcached); everything else lives in the project lib folder. I can "hg clone" my app at any time and get the exact version of the third party packages I need. Every so often I update the packages and test the integration.
  - lotyrin 13y ago
    This is why good dependency management systems support both specifying compatible semver ranges but also pinning/locking specific releases. You get nice one-step install/update/add/remove for your deps with the tool, but a fresh checkout can get the last known working versions and deploys can be guaranteed to use the same versions as were run in CI, etc. Looking now, it seems that pip can finally do this with pip-compile.
- zzzeek 13y ago
  Solr and its backend Lucene are written in Java. so it runs on any platform arguably even more easily than Python.
  - randlet 13y ago
    Understood but I was addressing whether 'pure Python' is a selling feature which it undeniably is. edit: I don't mean to say that it is a selling feature for everyone (i.e. people who aren't already using Python) but for those who are already using Python it definitely is.
SEJeff 13y ago
Solr is pretty good stuff, but I've had a bit more luck clustering and scaling elasticsearch horizontally with many hundreds of terabytes of data. From what I've read however, the newer solr versions took a lot of ideas from Elasticsearch and they are somewhat comparable at the moment.
jMyles 13y ago
Not the most popular, but test suite veracity is my favorite "pure Python" selling point.
- mattdeboard 13y ago
  You'll have to explain this. Most tests we write in Python are redundant with static typing.

ra 13y ago

Solr, Sphinx and ElasticSearch are all reliable search-at-scale options for those of us without petabyte problems. Whoosh is fine for very small datasets (megabytes) and low loads, just watch out for file permission issues.

Torn 13y ago
ElasticSearch is actually designed to scale to petabytes and beyond: http://elasticsearch.com/product/ http://elasticsearch.com/product/ ("petabytes of data with ease")

nknighthb 13y ago

Having had nothing but extremely negative experiences with production JVM deployments, anything being not-JVM-based is a huge selling point for me. Furthermore, "pure python" itself means that I can run it in any Python implementation I want, including pypy, without worrying about any C modules.

koolkao 13y ago
what kinds of JVM issues did you run into in deployment? for personal projects I have always used C++, Python, RoR, MEAN, etc. but at work where my role is non-dev, our software dev team likes Java and the heavy weighted GWT... frankly shutting me out. I'd love to know how to convince them to switch out of Java.
- nknighthb 13y ago
  Memory issues out the ass. Which are obviously all in my head because there are no memory leaks in Java. I must be doing it wrong. Am I sure I know how to multiply? Of course you shouldn't be using version 1.2.3.4.5, our code only works with 1.2.3.4.4, everybody knows that. Well, except for that code, which of course you have to run on 1.2.3.4.4c. And this third-party daemon over here, that only runs on an obscure 6-year-old JVM and you have to put it in /var/deadchicken/ and insert three squirrels before executing. You won't convince them, they've already jumped off the cliff. It's all perfectly normal to them. If you want to do development, find a company that hasn't been contaminated yet.
  - robrenaud 13y ago
    I don't actually write Java professionally. But it sounds like your problem isn't Java/JVM, but rather really shitty code. If it breaks when you change minor versions of the JVM, the code is broken.
    - nknighthb 13y ago
      Just from my own personal experience, you've just indicted code from three well-known SV companies, at least two more non-SV fortune-500s, a mess of B2B vendors, the Apache foundation, and a few minor, independent open-source projects. I've heard stories from friends in a few other companies about both internal and third-party code. At some point you reach the conclusion that the problem with an ecosystem is more fundamental than a few lazy developers. But even if not, your statement isn't actionable. If all the code in a certain set happens to be really shitty, the only rational choice is to not use that code. Any other argument ends up being one only of semantics.
      - jsmeaton 13y ago
        I think I've deployed only one java web app (puppet-db) that doesn't have huge memory leak issues. With the lack of process recycling in tomcat to mitigate "bad code", I cringe every time I have to deploy another one.
      - ris 13y ago
swasheck 13y ago
All of the troubles that I've had have been relative to compiling the JVM Python code and linking it correctly to my Java binary at compile time. Once I've gotten that working, I've found Lucene to be quite simple - and faster than Whoosh!

randlet 13y ago

Pure Python is definitely a selling point for those of us who regularly deploy to multiple platforms. Knowing you can 'pip install pure-python-lib' on any platform and have it work every single time without regards to what c compilers and c libraries you have available is a major boon.

davb 13y ago
Or, more importantly (in my case), very easily bundle pure Python code with your application. I understand the arguments against it, but in any project that might produce deliverables that will be in play for more than a few weeks the first thing I do is create a "lib" folder where any third party dependencies live and get imported from. With Python this is usually very easy. I've maintained my fair share of legacy code and more often than not one of the biggest issues is tracking down ancient libraries which work with my code. There are very few packages I trust to maintain fairly stable interfaces and be available years into the future (things like Apache HTTPd, PostgreSQL and memcached); everything else lives in the project lib folder. I can "hg clone" my app at any time and get the exact version of the third party packages I need. Every so often I update the packages and test the integration.
- lotyrin 13y ago
  This is why good dependency management systems support both specifying compatible semver ranges but also pinning/locking specific releases. You get nice one-step install/update/add/remove for your deps with the tool, but a fresh checkout can get the last known working versions and deploys can be guaranteed to use the same versions as were run in CI, etc. Looking now, it seems that pip can finally do this with pip-compile.
zzzeek 13y ago
Solr and its backend Lucene are written in Java. so it runs on any platform arguably even more easily than Python.
- randlet 13y ago
  Understood but I was addressing whether 'pure Python' is a selling feature which it undeniably is. edit: I don't mean to say that it is a selling feature for everyone (i.e. people who aren't already using Python) but for those who are already using Python it definitely is.

SEJeff 13y ago

Solr is pretty good stuff, but I've had a bit more luck clustering and scaling elasticsearch horizontally with many hundreds of terabytes of data. From what I've read however, the newer solr versions took a lot of ideas from Elasticsearch and they are somewhat comparable at the moment.

jMyles 13y ago

Not the most popular, but test suite veracity is my favorite "pure Python" selling point.

mattdeboard 13y ago
You'll have to explain this. Most tests we write in Python are redundant with static typing.