5 ms·

I agree with what you're saying but I think its important to point out that there is a Developer experience in most mainstream languages and then there is the O

I agree with what you're saying but I think its important to point out that there is a Developer experience in most mainstream languages and then there is the Operations experience of deploying, configuring, maintaining, scaling, and paying for, various RDBMS. Developers grow up crashing their programs. Crashing your DBMS (or similiarly messing it up) is not as friendly or immediate in feedback/consequence, so people shy away from it. The end result is that many developers are influenced to use code more and databases less.

0db532a0 8y ago
Elasticsearch quite conveniently makes it so difficult and fucking annoying to do data analytics with their pile of turd custom query un-language shambles that it leaves anyone stuck with it rightly reinventing oblong wheels with diamond bearings lubricated with iron filings in code. Aah, but dev-mis-ops say that it’s easy to deploy and maintain on AWS. It scales too! Well, at least the whole set-up keeps everyone employed.
- KennyCason 8y ago
  Could you elaborate more on your specific ES problems? Having worked with Elasticsearch extensively for many years, and do not share this sentiment. Many of the ES queries fundamentally aren't meant to have a 1-to-1 equivalent in SQL as ES's goal is to specialize as a distributed search index, and not a general relative database. If your point is that ES is being more frequently used as an alternative to just writing/scaling/using a SQL service to it's full potential, then I can somewhat understand the sentiment.
  - 0db532a0 8y ago
    I completely agree with the idea that ES might be good as a distributed search index. I really wish people would leave it at that. The company where I last worked had a main product which would send in readings every second. The only interface to these readings was Elasticsearch. dev-mis-ops were convinced that this was a good idea. Even the front end would send its queries to Elasticsearch to display statistics. My task was to look at the average energy usage over one year for the seven days in the week divided into fifteen-minute intervals. This is literally just a SELECT MOD GROUP BY in SQL. I am not even talking about more advanced SQL features here like windows, transactions, joins and so on. I tried for ages looking through terrible documentation trying to get it to work, at which point I thought I’d have one of the resident ES experts take a bit of his own medicine. It took about five attempts of back and forth over two days to get it right. This wasn’t the only query we had problems with. At some point, I just asked them to give me a CSV interface so I could do it in AWK. You can’t even download CSVs of data sets without some external tool! They were still convinced that ES was fit for purpose.
- AznHisoka 8y ago
  What version of elasticsearch are you using? Even in version 1.x, aggregation queries are so intuitive and pleasant to work with, imo. Now the actual libraries written in Ruby and other languages may be clunky. In fact, I am not sure why people need all those fancy BI tools. 99% of the time, Elasticsearch aggregations gives me the data I need.
  - 0db532a0 8y ago
    Challenge, with context in my other comment at the same depth: Suppose you have a table which includes power usage readings for all devices, sent every five seconds. Something like this: (id, timestamp, power) Write me an ES query which gets the average power usage for device id=x over the last year over the seven days in the week, each divided over fifteen-minute intervals. To clarify: The groups will be the product of the days of the week with the ninety-six fifteen-minute intervals in a day, making 7*96 unique groups. You will not be able to do this in any simple way.
    - AznHisoka 8y ago
      This is an indexing problem. Add 2 extra fields. For each timestamp, you also need to index the 15-minute interval (ie 7:13:05 maps to 7:00-7:15) and the day of the week. After that, do a terms aggregation on day of week with a subaggregation on 15 min interval. If you can’t reindex all your data, then of course it will be hard. But once the data is indexed and mapped the right way, its easy.
      - 0db532a0 8y ago
        So you couldn’t write the query. It just isn’t practical to create an index for every query which comes to mind during research, and even to create the index, you have to use a terrible, undocumented embedded query language.
        AznHisoka 8y ago
        Not create a new index - reindex the existing documents I have personally managed 10+ TB clusters and reindexed multiple times when I need to add new fields. The key is to take some time to clearly understand what analytics you want to conduct, then add the fields you need afterwards.

0db532a0 8y ago

Elasticsearch quite conveniently makes it so difficult and fucking annoying to do data analytics with their pile of turd custom query un-language shambles that it leaves anyone stuck with it rightly reinventing oblong wheels with diamond bearings lubricated with iron filings in code. Aah, but dev-mis-ops say that it’s easy to deploy and maintain on AWS. It scales too! Well, at least the whole set-up keeps everyone employed.

KennyCason 8y ago
Could you elaborate more on your specific ES problems? Having worked with Elasticsearch extensively for many years, and do not share this sentiment. Many of the ES queries fundamentally aren't meant to have a 1-to-1 equivalent in SQL as ES's goal is to specialize as a distributed search index, and not a general relative database. If your point is that ES is being more frequently used as an alternative to just writing/scaling/using a SQL service to it's full potential, then I can somewhat understand the sentiment.
- 0db532a0 8y ago
  I completely agree with the idea that ES might be good as a distributed search index. I really wish people would leave it at that. The company where I last worked had a main product which would send in readings every second. The only interface to these readings was Elasticsearch. dev-mis-ops were convinced that this was a good idea. Even the front end would send its queries to Elasticsearch to display statistics. My task was to look at the average energy usage over one year for the seven days in the week divided into fifteen-minute intervals. This is literally just a SELECT MOD GROUP BY in SQL. I am not even talking about more advanced SQL features here like windows, transactions, joins and so on. I tried for ages looking through terrible documentation trying to get it to work, at which point I thought I’d have one of the resident ES experts take a bit of his own medicine. It took about five attempts of back and forth over two days to get it right. This wasn’t the only query we had problems with. At some point, I just asked them to give me a CSV interface so I could do it in AWK. You can’t even download CSVs of data sets without some external tool! They were still convinced that ES was fit for purpose.
AznHisoka 8y ago
What version of elasticsearch are you using? Even in version 1.x, aggregation queries are so intuitive and pleasant to work with, imo. Now the actual libraries written in Ruby and other languages may be clunky. In fact, I am not sure why people need all those fancy BI tools. 99% of the time, Elasticsearch aggregations gives me the data I need.
- 0db532a0 8y ago
  Challenge, with context in my other comment at the same depth: Suppose you have a table which includes power usage readings for all devices, sent every five seconds. Something like this: (id, timestamp, power) Write me an ES query which gets the average power usage for device id=x over the last year over the seven days in the week, each divided over fifteen-minute intervals. To clarify: The groups will be the product of the days of the week with the ninety-six fifteen-minute intervals in a day, making 7*96 unique groups. You will not be able to do this in any simple way.
  - AznHisoka 8y ago
    This is an indexing problem. Add 2 extra fields. For each timestamp, you also need to index the 15-minute interval (ie 7:13:05 maps to 7:00-7:15) and the day of the week. After that, do a terms aggregation on day of week with a subaggregation on 15 min interval. If you can’t reindex all your data, then of course it will be hard. But once the data is indexed and mapped the right way, its easy.
    - 0db532a0 8y ago
      So you couldn’t write the query. It just isn’t practical to create an index for every query which comes to mind during research, and even to create the index, you have to use a terrible, undocumented embedded query language.
      - AznHisoka 8y ago
        Not create a new index - reindex the existing documents I have personally managed 10+ TB clusters and reindexed multiple times when I need to add new fields. The key is to take some time to clearly understand what analytics you want to conduct, then add the fields you need afterwards.