6 ms·

I've said it on here before, but the way in which Elasticsearch used to lock away critical security functionality (like TLS support and RBAC) behind a paid subs

by lol768 6y ago

I've said it on here before, but the way in which Elasticsearch used to lock away critical security functionality (like TLS support and RBAC) behind a paid subscription whilst making just enough functionality available for free such that users could shoot their foot off is disgusting. This only ever changed after Open Distro for Elasticsearch came onto the scene and forced Elastic's hand. I entirely agree the vendors are (partially) to blame here.

bmcahren 6y ago
This is why we are refactoring our database to be able to migrate to Amazon documentdb from MongoDB. Encryption at rest.... Pay up!
- tyre 6y ago
  Curious, why do you use Mongo? Does it give you something that a JSONB column in Postgres wouldn’t?
  - scandox 6y ago
    Maybe they have 10 years of code sitting on it? It's what a colleague calls sound historical reasons.
    - deleted 6y ago
      [deleted]
  - BossingAround 6y ago
    Surprisingly, I feel Mongo has been kind of surging back into developers' minds. I thought Mongo was utterly dead, and I also don't know why would one use Mongo instead of JSONB. The last I heard (3-4 years ago?), there were some fundamental problems with Mongo. That said, I've incidentally heard a lot about Mongo in the last half a year. Might be my bubble. Might be MongoDB actually maturing and getting really good. I hope it's the latter.
    - lallysingh 6y ago
      I mostly hear complaints about email spam from their salespeople.
  - StriverGuy 6y ago
    If you have ever used Mongo Atlas, you would understand why people use mongo. The administration ease that the platform provides is unmatched by any other DBaaS I am aware of.
  - cpursley 6y ago
    Yes, hipster cred. For everyone else, JUP ("Just Use Postgres") Snark aside, Postgres should hold you over for quite some time. With Postgres you probably don't need Redis, Elastic Search, Mongo or Kafka.
  - hamandcheese 6y ago
    I use jsonb heavily. While it is amazing, I definitely wouldn’t rely on it as a general purpose replacement for NoSQL/schemaless data storage. An example of an issue I am dealing with currently: while you can create a gin index to speed up containment queries, Postgres doesn’t keep any statistics about jsonb columns. This means the query planner will sometimes do stupid things, like using the index even for very non-selective overlap conditions, which is a lot slower than just doing a sequential scan. Less of an issue for me but worth considering: the size of the gin index in my use case seems to be about 5x bigger than the size of the unindexed data. I was surprised by the size increase. I only use the containment operator so I could make a smaller/faster index using the jsonb_path_ops operator class. This is on my todo list :) Like all non-btree indexes in Postgres, the index is unordered. That means sorting by values in the jsonb column will always be slow. This doesn’t matter for selective queries, but exacerbates my already slow non-selective queries that return large result sets. That said, if your queries are selective, jsonb + gin indexes are surprisingly performant (in the 0.5-10ms range for small result sets). My use case is a mix of structured relational data with jsonb for user-defined values (which of course they want to use for querying/sorting and I was dumb enough to say “sure, why not?”) In terms of the magnitude of data, there’s roughly 10 million rows. Each team using this service has the query scoped to about 500k-1 million records, and then additional filters (on the jsonb column) will scope that down to anywhere between 60k-0 results.
    - acidbaseextract 6y ago
      Thanks for the detailed response. I'm curious, is the NoSQL store a canonical store of its own data? If not, how do you replicate from postgres to the secondary store? I ask because where I work we sync postgres to a secondary store for search, but the way it's done in a piecemeal, application-specific way gives me the heebie jeebies. It almost certainly will result in that secondary store drifting. Unfortunately we can't use something like zombodb [1] as we're on amazon RDS. It seems like you know your stuff, and seeing non-deterministic consistency irritates the heck out of me! [1] https://github.com/zombodb/zombodb https://github.com/zombodb/zombodb
  - pavel_odintsov 6y ago
    Thank you for raising this question. Well, for my things I use MongoDB because of very convenient integration with programming languages (Go, C++ via Mongo C++) and zero hassle with schema But I'll be happy to replace it by something else, my load is extremely small and only single requirement is to have DB as network daemon, not as embedded storage as it will be used by 2 applications (main daemon an API). RethinkDB was really nice candidate for it but it's not alive anymore: https://rethinkdb.com/blog/rethinkdb-shutdown/ https://rethinkdb.com/blog/rethinkdb-shutdown/
  - lmm 6y ago
    A JSONB column in Postgres doesn't have: - Proper replication in its first-class, default configuration - Automatically managed cluster membership - Seamless automated failover - First-class async client libraries in many languages - A non-awful query language Focusing on the JSON is beside the point (though it is a convenience). Wake me up when you've got a properly distributed database.
    - andoriyu 6y ago
      Are you trolling? Also, wtf is properly distributed database?
- mushi 6y ago
  You’re switching from a free database to a paid one because what?
BossingAround 6y ago
Well, it's just another attempt at monetizing the product. Nowadays, companies and developers expect everything to be OSS (and I love it) yet it's incredibly expensive to develop SW (and very few people do OSS just because of passion--I tried and failed miserably). Locking RBAC and TLS behind a paid subscription is a sure way to force companies with security teams to pay for it (or not to use it). This particular lesson comes relatively cheap--dropping your data is not the worst an attacker could do with it. Hopefully, more people will research what I'd call "security 101"...
- pavel_lishin 6y ago
  > Well, it's just another attempt at monetizing the product. Don't make excuses for their shitty business practices.
  - gameswithgo 6y ago
    Don't tell people what to do.
  - cookiecaper 6y ago
    The thing is that no one cares. It's a cultural issue. If Elastic makes it "harder" to get off the starting line by setting secure defaults, they'll get their lunch eaten by a fork called EasyElastic that people perceive as "easier". This probably isn't an excuse for Elastic anymore, but it's how Elastic was born ("easier than Solr!" -- they didn't invent full-text search, after all) and it's how whatever supplants Elastic will be born. Why is MySQL dominant over Postgres even though "referential integrity" didn't come to the game until v5? Why Docker over jails or OpenVZ? etc. People adopt technology because it's fashionable and it becomes fashionable in part because it's perceived as easy to use. Security and ease-of-use are not quite true opposites, but there's definitely some intrinsic tension. We have a lot of hapless practitioners in the space and the root problems here won't go away until we get some standards. Better solutions usually lose because crappy stuff focuses more effort on marketing and an "easy" onboarding process, where better stuff focuses on the operational complications of the real world.
- 6y ago

by lol768 6y ago

bmcahren 6y ago

This is why we are refactoring our database to be able to migrate to Amazon documentdb from MongoDB. Encryption at rest.... Pay up!

tyre 6y ago
Curious, why do you use Mongo? Does it give you something that a JSONB column in Postgres wouldn’t?
- scandox 6y ago
  Maybe they have 10 years of code sitting on it? It's what a colleague calls sound historical reasons.
  - deleted 6y ago
    [deleted]
- BossingAround 6y ago
  Surprisingly, I feel Mongo has been kind of surging back into developers' minds. I thought Mongo was utterly dead, and I also don't know why would one use Mongo instead of JSONB. The last I heard (3-4 years ago?), there were some fundamental problems with Mongo. That said, I've incidentally heard a lot about Mongo in the last half a year. Might be my bubble. Might be MongoDB actually maturing and getting really good. I hope it's the latter.
  - lallysingh 6y ago
    I mostly hear complaints about email spam from their salespeople.
- StriverGuy 6y ago
  If you have ever used Mongo Atlas, you would understand why people use mongo. The administration ease that the platform provides is unmatched by any other DBaaS I am aware of.
- cpursley 6y ago
  Yes, hipster cred. For everyone else, JUP ("Just Use Postgres") Snark aside, Postgres should hold you over for quite some time. With Postgres you probably don't need Redis, Elastic Search, Mongo or Kafka.
- hamandcheese 6y ago
  I use jsonb heavily. While it is amazing, I definitely wouldn’t rely on it as a general purpose replacement for NoSQL/schemaless data storage. An example of an issue I am dealing with currently: while you can create a gin index to speed up containment queries, Postgres doesn’t keep any statistics about jsonb columns. This means the query planner will sometimes do stupid things, like using the index even for very non-selective overlap conditions, which is a lot slower than just doing a sequential scan. Less of an issue for me but worth considering: the size of the gin index in my use case seems to be about 5x bigger than the size of the unindexed data. I was surprised by the size increase. I only use the containment operator so I could make a smaller/faster index using the jsonb_path_ops operator class. This is on my todo list :) Like all non-btree indexes in Postgres, the index is unordered. That means sorting by values in the jsonb column will always be slow. This doesn’t matter for selective queries, but exacerbates my already slow non-selective queries that return large result sets. That said, if your queries are selective, jsonb + gin indexes are surprisingly performant (in the 0.5-10ms range for small result sets). My use case is a mix of structured relational data with jsonb for user-defined values (which of course they want to use for querying/sorting and I was dumb enough to say “sure, why not?”) In terms of the magnitude of data, there’s roughly 10 million rows. Each team using this service has the query scoped to about 500k-1 million records, and then additional filters (on the jsonb column) will scope that down to anywhere between 60k-0 results.
  - acidbaseextract 6y ago
    Thanks for the detailed response. I'm curious, is the NoSQL store a canonical store of its own data? If not, how do you replicate from postgres to the secondary store? I ask because where I work we sync postgres to a secondary store for search, but the way it's done in a piecemeal, application-specific way gives me the heebie jeebies. It almost certainly will result in that secondary store drifting. Unfortunately we can't use something like zombodb [1] as we're on amazon RDS. It seems like you know your stuff, and seeing non-deterministic consistency irritates the heck out of me! [1] https://github.com/zombodb/zombodb https://github.com/zombodb/zombodb
- pavel_odintsov 6y ago
  Thank you for raising this question. Well, for my things I use MongoDB because of very convenient integration with programming languages (Go, C++ via Mongo C++) and zero hassle with schema But I'll be happy to replace it by something else, my load is extremely small and only single requirement is to have DB as network daemon, not as embedded storage as it will be used by 2 applications (main daemon an API). RethinkDB was really nice candidate for it but it's not alive anymore: https://rethinkdb.com/blog/rethinkdb-shutdown/ https://rethinkdb.com/blog/rethinkdb-shutdown/
- lmm 6y ago
  A JSONB column in Postgres doesn't have: - Proper replication in its first-class, default configuration - Automatically managed cluster membership - Seamless automated failover - First-class async client libraries in many languages - A non-awful query language Focusing on the JSON is beside the point (though it is a convenience). Wake me up when you've got a properly distributed database.
  - andoriyu 6y ago
    Are you trolling? Also, wtf is properly distributed database?
mushi 6y ago
You’re switching from a free database to a paid one because what?

BossingAround 6y ago

Well, it's just another attempt at monetizing the product. Nowadays, companies and developers expect everything to be OSS (and I love it) yet it's incredibly expensive to develop SW (and very few people do OSS just because of passion--I tried and failed miserably). Locking RBAC and TLS behind a paid subscription is a sure way to force companies with security teams to pay for it (or not to use it). This particular lesson comes relatively cheap--dropping your data is not the worst an attacker could do with it. Hopefully, more people will research what I'd call "security 101"...

pavel_lishin 6y ago
> Well, it's just another attempt at monetizing the product. Don't make excuses for their shitty business practices.
- gameswithgo 6y ago
  Don't tell people what to do.
- cookiecaper 6y ago
  The thing is that no one cares. It's a cultural issue. If Elastic makes it "harder" to get off the starting line by setting secure defaults, they'll get their lunch eaten by a fork called EasyElastic that people perceive as "easier". This probably isn't an excuse for Elastic anymore, but it's how Elastic was born ("easier than Solr!" -- they didn't invent full-text search, after all) and it's how whatever supplants Elastic will be born. Why is MySQL dominant over Postgres even though "referential integrity" didn't come to the game until v5? Why Docker over jails or OpenVZ? etc. People adopt technology because it's fashionable and it becomes fashionable in part because it's perceived as easy to use. Security and ease-of-use are not quite true opposites, but there's definitely some intrinsic tension. We have a lot of hapless practitioners in the space and the root problems here won't go away until we get some standards. Better solutions usually lose because crappy stuff focuses more effort on marketing and an "easy" onboarding process, where better stuff focuses on the operational complications of the real world.
6y ago