6 ms·

Lucene: The Good Parts

93 points by SkyRocknRoll 11y ago

ddorian43 11y ago
I still don't understand why lucene can't use the original document for aggregations ? I undestand that it will be slower if the field isn't indexed, but it should be doable like rdbms do.
- boomzilla 11y ago
  It can, as long as you store all the fields that will be needed for aggregation. Stored, non-indexed field retrieval can be slow as it might involve a lot of random seeks, so it might be slower than relation dbs. What use case do you have in mind?
  - ddorian43 11y ago
    I mean in the case of _source field that elasticsearc uses. But seems that is separate. My usecase is I want to do querying+filtering on all fields of the document. In this case you have to: 1.store the field or document so you can get back the value (ex: when querying the document) 2.index the field for filtering 3.separately index the field for aggregation(doc_values) While in rdbms you only need(1).
latchkey 11y ago
Here is a bit of interesting history that I bet few people know about. Doug Cutting worked on VTwin at Apple long before he wrote Lucene. This was a codename for Apple's search technology they were building into the OS. I knew about it because at the time I was building computer labs for my college and bought a lot of Apple hardware (about a million dollars worth) and my rep added me to the private beta. Many years later, when I saw the Lucene project being worked on and saw that Doug was behind it, I immediately reached out to him and asked him if he was interested in joining the Jakarta group, as I thought this would be a great addition to our growing community. Needless to say, the rest is history, but I feel like if nobody had reached out to Doug, he may not have gotten as much exposure and may not have been motivated to start the rest of the amazing projects that he's worked on, including Hadoop. =) [1] https://www.google.com/search?q=apple+vtwin+codename https://www.google.com/search?q=apple+vtwin+codename
prasanthv 11y ago
Great article! Never even knew about Lucene.
derefr 11y ago
> SQL was not then, and is still not now, a very good blob or document storage system. What does that even mean? SQL is a wire protocol for relational queries. Document/blob/key-value stores can be made to speak SQL. Most of them just choose not to implement SQL support for some weird reason.
- Tenhundfeld 11y ago
  I think you're being overly pedantic – or maybe deliberately obtuse. I can't tell. In discussions of "NoSQL" technologies, it's common to use SQL as a shorthand for traditional, relational databases which generally support SQL. We're talking about databases like MySQL, Oracle, Postgres, MS SQL Server, etc. Sure, Andrew could have been more precise with this language, but from the context, do you honestly think he's using SQL to mean specifically Structured Query Language, the 4G language developed in the 70's?
  - altcognito 11y ago
    Is it "overly" pedantic when making broad criticism of a given technology to at least be using the word that identifies said technology? RDBS is really common way to identify this class of products.
    - Tenhundfeld 11y ago
      You know what. I think you're right. It's not overly pedantic to say that the article shouldn't use SQL to mean RDBS – or RDBMS, as I see it more commonly. My comment was specifically in response to your approach, not your overall point: > What does that even mean? SQL is a wire protocol for relational queries. First, if we're being pedantic, I disagree with your usage of "wire protocol" to describe the SQL language, but I get what you're saying. What I was trying to comment on was your feigned confusion. It was perfectly clear to me what the author meant by "SQL" in the context of the article, and I think you understood too. If you had said something like, "SQL is a language and bad term to describe Relational Database Management Systems (RDBMS), which is what the author really meant," I wouldn't have commented. I probably would have upvoted actually, because I agree. Anyway, not a big deal, I don't think we're in conflict with the substance of your point, just how you chose to express it in this one example. Have a great day. :)
pixelmonkey 11y ago
I was just informed today that this article, "Lucene: The Good Parts", which I wrote a few months ago, was also just published in Hacker Monthly for this month's print/digital issue (June 2015): http://hackermonthly.com/issue-61.html http://hackermonthly.com/issue-61.html
bracewel 11y ago
sigh HTTPS mixed content... really?

93 points by SkyRocknRoll 11y ago

I still don't understand why lucene can't use the original document for aggregations ? I undestand that it will be slower if the field isn't indexed, but it should be doable like rdbms do.

boomzilla 11y ago
It can, as long as you store all the fields that will be needed for aggregation. Stored, non-indexed field retrieval can be slow as it might involve a lot of random seeks, so it might be slower than relation dbs. What use case do you have in mind?
- ddorian43 11y ago
  I mean in the case of _source field that elasticsearc uses. But seems that is separate. My usecase is I want to do querying+filtering on all fields of the document. In this case you have to: 1.store the field or document so you can get back the value (ex: when querying the document) 2.index the field for filtering 3.separately index the field for aggregation(doc_values) While in rdbms you only need(1).

latchkey 11y ago

Here is a bit of interesting history that I bet few people know about. Doug Cutting worked on VTwin at Apple long before he wrote Lucene. This was a codename for Apple's search technology they were building into the OS. I knew about it because at the time I was building computer labs for my college and bought a lot of Apple hardware (about a million dollars worth) and my rep added me to the private beta. Many years later, when I saw the Lucene project being worked on and saw that Doug was behind it, I immediately reached out to him and asked him if he was interested in joining the Jakarta group, as I thought this would be a great addition to our growing community. Needless to say, the rest is history, but I feel like if nobody had reached out to Doug, he may not have gotten as much exposure and may not have been motivated to start the rest of the amazing projects that he's worked on, including Hadoop. =) [1] https://www.google.com/search?q=apple+vtwin+codename https://www.google.com/search?q=apple+vtwin+codename

prasanthv 11y ago

Great article! Never even knew about Lucene.

derefr 11y ago

> SQL was not then, and is still not now, a very good blob or document storage system. What does that even mean? SQL is a wire protocol for relational queries. Document/blob/key-value stores can be made to speak SQL. Most of them just choose not to implement SQL support for some weird reason.

Tenhundfeld 11y ago
I think you're being overly pedantic – or maybe deliberately obtuse. I can't tell. In discussions of "NoSQL" technologies, it's common to use SQL as a shorthand for traditional, relational databases which generally support SQL. We're talking about databases like MySQL, Oracle, Postgres, MS SQL Server, etc. Sure, Andrew could have been more precise with this language, but from the context, do you honestly think he's using SQL to mean specifically Structured Query Language, the 4G language developed in the 70's?
- altcognito 11y ago
  Is it "overly" pedantic when making broad criticism of a given technology to at least be using the word that identifies said technology? RDBS is really common way to identify this class of products.
  - Tenhundfeld 11y ago
    You know what. I think you're right. It's not overly pedantic to say that the article shouldn't use SQL to mean RDBS – or RDBMS, as I see it more commonly. My comment was specifically in response to your approach, not your overall point: > What does that even mean? SQL is a wire protocol for relational queries. First, if we're being pedantic, I disagree with your usage of "wire protocol" to describe the SQL language, but I get what you're saying. What I was trying to comment on was your feigned confusion. It was perfectly clear to me what the author meant by "SQL" in the context of the article, and I think you understood too. If you had said something like, "SQL is a language and bad term to describe Relational Database Management Systems (RDBMS), which is what the author really meant," I wouldn't have commented. I probably would have upvoted actually, because I agree. Anyway, not a big deal, I don't think we're in conflict with the substance of your point, just how you chose to express it in this one example. Have a great day. :)

pixelmonkey 11y ago

I was just informed today that this article, "Lucene: The Good Parts", which I wrote a few months ago, was also just published in Hacker Monthly for this month's print/digital issue (June 2015): http://hackermonthly.com/issue-61.html http://hackermonthly.com/issue-61.html

bracewel 11y ago

sigh HTTPS mixed content... really?