8 ms·

I also want to point their Node.js transactions API is wrong and looks like they have no idea how promises or async code work in JS. In mongo, you have a `with

by inglor 6y ago

I also want to point their Node.js transactions API is wrong and looks like they have no idea how promises or async code work in JS. In mongo, you have a `withTransaction(fn)` helper that passes a session parameter. Mongo can call this function mutliple times with the same session object. This means that if you have an async function with reference to a session and a transaction gets retried - you very often get "part of one attempt + some parts of another" committed. We had to write a ton of logic around their poor implementation and I was shocked to see the code underneath. It was just such a stark contrast to products that I worked with before that generally "just worked" like postgres, elasticsearch or redis. Even tools people joke about a lot like mysql never gave me this sort of data corruption. Edit: I was kind of angry when writing this so I didn't provide a source and I'm a bit surprised this go so many upvotes without a source (I guess this community is more trusting than I assumed :] ). Anyway for good measure and to behave the way I'd like others to when making such accusations here is where they pass the same session object to the transacton https://github.com/mongodb/node-mongodb-native/blob/e5b762c6d53afa967f24c26a1d1b6c921757c9c9/lib/sessions.js#L376 https://github.com/mongodb/node-mongodb-native/blob/e5b762c6... (follow from withTransaction in that file) - I can add examples of code easily introducing the above mentioned bug if people are interested.

inglor 6y ago
If you work for Mongo and are reading this. Please just fix it. I don't need to win and I don't care about being "right". I just don't want to be called to the office on a weekend anymore for this sort of BS. Production incidents with MongoDB last year: 15 Production instances with redis, elasticsearch and mysql combined last year: 2 (and with much less severity) Edit: just to add: I didn't pick Mongo, I was just the engineer called to clean that mess. I created enough of my own messes to not resent the person who made that shot for it. We are constantly on the verge of rewriting the MongoDB stuff since a database that small (~250GB) should really not have these many issues (In previous workplaces I ran ~10TB PostgreSQL deployments with much more complicated schemas and queries with far fewer issues). It's also expensive and support at Mongo Atlas hasn't been great (we should probably self host but I am not used to small databases being so problematic)
- brianwawok 6y ago
  This is why most of us don’t use mongo in production. It’s just not worth it. Postgres is a tank and supports Json when you really need it.
  - hetspookjee 6y ago
    The Guardian posted quite a nice blog in 2018 about the switch to Postgres from MongoDB. Especially interesting because they intended to use Postgres as replacement document storage: Here's the link https://www.theguardian.com/info/2018/nov/30/bye-bye-mongo-hello-postgres https://www.theguardian.com/info/2018/nov/30/bye-bye-mongo-h...
    - guanzo 6y ago
      > Automatically generating database indexes on application startup is probably a bad idea. aw crap. oh well it probably doesn't matter for my small-ish application.
  - Quekid5 6y ago
    I was actually amazed that a big CMS/E-commerce vendor proudly proclaimed in a sales meeting that they were on MongoDB. I suppose salespeople probably aren't into the nitty-gritty, but their tech people should have warned them about this. Maybe they were just trying to pull our collective leg, but I suppose that why I was at that meeting. It was obviously an instant 'No'.
    - leviathant 6y ago
      There aren't a lot of CMS/Ecommerce vendors that sit on MongoDB, so maybe we were in a meeting together! Even if we weren't - as a sales engineer on a large CMS/ECommerce platform with merchants running $150M+ in annual revenue, with an average client retention of seven years, and two decades of agency experience behind the decisions around building that platform, if you instantly said no just because of MongoDB, maybe you don't know as much about MongoDB as you think you do. I came from a SQL background myself, and had reservations based on all the things I'd read about MongoDB as we decided to build a platform after doing things bespoke for two decades, but time has proven our architecture choices out. It's easy to be proud of something that works well.
jfkebwjsbx 6y ago
> Even tools people joke about a lot like mysql never gave me this sort of data corruption. People rightfully joked about MySQL when they had the non-ACID engine. Same for MongoDB. A database that loses data when properly used is a joke. Yes, there are use cases out there for fast non-guaranteed writes. No, 99% of companies don’t have them.
- Something1234 6y ago
  Can you name a use case for a fast non guaranteed write?
  - Jweb_Guru 6y ago
    Sure. Data that people don't care about enough to be worried about losing--for example, time series data from an unimportant remote sensor. Should this data be recorded at all? Maybe not, but if should then a best-effort recording may be fine. It may even be all that's possible.
    - mbreese 6y ago
      I wouldn’t go as far as to say an “unimportant” remote sensor... but I think you’re correct in spirit. I could think of an instance where you’d like to log data, but the occasional datapoint being missing wouldn’t be terrible. Maybe something like a temperature monitor — you’d like to have a record of the temperature by the minute, but if a few records dropped out, you’d be able to guess the missing values from context. Something like the data monitoring equivalent of UDP vs TCP.
  - throwaway744678 6y ago
    Analytics: you don't want to slow down your app, and you don't care if you lose a few records in the process.
    - zbentley 6y ago
      Importantly, in analytics workloads, it is very important to know roughly how many writes aren't making it. Otherwise your analytics system sucks.
vorticalbox 6y ago
> Mongo can call this function multiple times with the same session object. isn't that the point? you can use a session to do multi actions within that session.
- inglor 6y ago
  If you have code that looks like this: withTransaction(async session => { await Promise.all([someOp(sesson), someOtherOp(session)]); }); Mongo may retry running it (calling the function again) if a "TransaientTransactionError" is raised (the transaction is retried from the client side rather than at the cluster). However, when the driver calls your function again it doesn't invalidate the `session` object - so previous calls to the same function can make updates to the database. Let's say `someOp` does something that causes the transaction to retry and `someOtherOp` is doing something non-mongo-related in the meantime (like pulling a value from redis). Now `someOtherOp` reached the mongo part of its code and it is executing it happily with the same session object (so operations succeed although they really shouldn't) The point of transactions like you said is to perform multiple operations atomically and for them to happen "exactly once or not at all". With Mongo in practice it is very easy to get "Once and some leftovers from a previous attempt".
  - IgorPartola 6y ago
    Sorry, I haven’t had my coffee yet. If I am reading this correctly, either someOp() or someOtherOp() may execute first, no? And if you introduce an external database, why do you expect Mongo to handle that rollback? Say someOtherOp() increments a Redis value by 1. If that part executed first since both are asynchronous here, what would a Mongo session have to do with it? What exactly would invalidating that session object do here? And what would the session object do after it was invalidated?
    - deleted 6y ago
      [deleted]
    - waheoo 6y ago
      It sounds like the old session object is reused and becomes live again or something.
hodgesrm 6y ago
> Even tools people joke about a lot like mysql never gave me this sort of data corruption. That's about a decade out of date at this point. MySQL/InnoDB is the standard table engine and corruption is exceedingly rare. As of 2014, when I last directly worked on MySQL prod systems, there was no practical difference from PostgreSQL in terms of transactional guarantees. That includes APIs like JDBC which we used for billions of transactions.
- morelisp 6y ago
  MySQL still has no transactional DDL (and I think still even autocommits if you try). This is a major difference from Postgres which I believe supports everything short of dropping tables.
  - takeda 6y ago
    I wouldn't use that particular thing against MySQL. DDL normally supposed to be always outside of a transaction, it's just PostgreSQL feature that you can use them inside and be able to rollback. BTW I'm convinced you also can drop table within a transaction in PostgreSQL.
    - morelisp 6y ago
      No, MySQL stands out here. Postgres, SQL Server, DB2, and Firebird all give at least some way to do some major DDL transactionally. Usability varies (e.g. Oracle supports a very specific kind of change that is not its normal DDL statements), but it's at least possible. https://wiki.postgresql.org/wiki/Transactional_DDL_in_PostgreSQL:_A_Competitive_Analysis#Oracle https://wiki.postgresql.org/wiki/Transactional_DDL_in_Postgr... That MySQL autocommits is also even worse than just "doesn't support it."
    - dragonwriter 6y ago
      > DDL normally supposed to be always outside of a transaction A basic element of the relational model is that metadata is stored as relational data and that the same guarantees that apply to manipulating main data in the database apply to manipulating the schema metadata. It's true that many real relational databases compromise on this element in various ways at times, but it is absolutely not the case that DDL “is supposed to be” non-transactional.
lossolo 6y ago
When I was evaluating MongoDb couple of years ago (around the time they were switching to WiredTiger engine), I've found memory leak in their Node.js client on day one, I've submitted a ticket on their Jira and the same time I had a look at other issues they had there. I saw there memory leak after memory leak, memory corruption everywhere, data disappearing without any reason, segfaults etc. After that MongoDB was dropped as a candidate for a DB in project I was working on, we went with Postgres and never regretted it.
takeda 6y ago
MySQL is less of a joke than MongoDB is. They similarly started by someone who didn't know anything about databases and learned about them on the go. Actually both of them started as much faster alternatives to other databases, both ended up having complete rewrite of its engine written by someone from outside that knew their stuff. MySQL ISAM then MyISAM and then InnoDB (written by an outsider). Similarly MongoDB got a WiredTiger written. The thing is that MySQL is older so it went through all of it earlier, but it still suffers from poor decisions from the past. This is contrasting with PostgreSQL, where correctness and reliability was #1 from the beginning. It started as an awfully slow database, but performance for improved and we now have correct, reliable and fast database.
- berns 6y ago
  MySQL is no joke, nothing is perfect and Postgresql is not 100% reliable. Remember: Transaction ID wraparound: https://twitter.com/bcantrill/status/1110647418008133632 https://twitter.com/bcantrill/status/1110647418008133632 Incorrect use of fsync: https://news.ycombinator.com/item?id=19119991 https://news.ycombinator.com/item?id=19119991
  - goatinaboat 6y ago
    MySQL is no joke If you were around back in the day you will remember the MySQL team claiming that no one needed transactions or referential integrity, that you should just do it yourself in the application...
    - twic 6y ago
      Does MySQL support check constraints yet?
      - dnissley 6y ago
        It does finally! In 8.0.16+
    - beatrobot 6y ago
hintymad 6y ago
Just curious, what was the reason that your team decided to work around the problem instead of migrating away from MongoDB?
- capableweb 6y ago
  Not the author but done similar things (patching something rather than migrating away from it). Usually it's way more work to migrate away than just patching it again to fit your use-case. Once you find yourself having to patch it too often, you start thinking about migrating away. Then the research slowly begins ad-hoc until it hits "seems we need to migrate away now, otherwise we're spending too much time working around something / fixing their broken shit", that's when you sit down and decide to migrate away from it. Also would depend on how long time you think the application will be around. You're building an MVP to evaluate something? Just hack together whatever will work (then throw away). You're maintain software for a library/archive that will most likely stick around for a long time, even if they say it's just temporary? Do decisions that will help in the future, always.
- inglor 6y ago
  We have a complicated system and migration is ~3 months we won't be shipping features. We have a roadmap we need to meet and so far we have been trying to spill money on it rather than developers (paying mongo atlas) and adding features incrementally as Mongo gets them (like transactions). If this wasn't a startup we would probably rewrite.
xeromal 6y ago
I just want remind people that this video exists. https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs
- sorokod 6y ago
  Well, does dev/null supports sharding?
g0ldenb0ugh 6y ago
> We had to write a ton of logic around their poor implementation and I was shocked to see the code underneath. Just wondering, did you submit a bug report to them about this? If so, any response?
- inglor 6y ago
  Yes, several times - we pay Mongo Atlas over $5000 per month. We reported it immediately at the highest severity and we pay for the highest tier or support - we tried to collaborate as soon as possible. It sort of went "over their head".
  - g0ldenb0ugh 6y ago
    Can you link the JIRA ticket? I use the Node driver heavily and have contributed several PRs to it in the past, would be more than happy to fix this since it seems fairly bad.
    - inglor 6y ago
      Hi, I'm not sure where I'm supposed to look for JIRAs at but we discussed this with Mongo Atlas via several support tickets and channels. For example at #00636235 "how to avoid TransientTransactionError".

by inglor 6y ago

inglor 6y ago

If you work for Mongo and are reading this. Please just fix it. I don't need to win and I don't care about being "right". I just don't want to be called to the office on a weekend anymore for this sort of BS. Production incidents with MongoDB last year: 15 Production instances with redis, elasticsearch and mysql combined last year: 2 (and with much less severity) Edit: just to add: I didn't pick Mongo, I was just the engineer called to clean that mess. I created enough of my own messes to not resent the person who made that shot for it. We are constantly on the verge of rewriting the MongoDB stuff since a database that small (~250GB) should really not have these many issues (In previous workplaces I ran ~10TB PostgreSQL deployments with much more complicated schemas and queries with far fewer issues). It's also expensive and support at Mongo Atlas hasn't been great (we should probably self host but I am not used to small databases being so problematic)

brianwawok 6y ago
This is why most of us don’t use mongo in production. It’s just not worth it. Postgres is a tank and supports Json when you really need it.
- hetspookjee 6y ago
  The Guardian posted quite a nice blog in 2018 about the switch to Postgres from MongoDB. Especially interesting because they intended to use Postgres as replacement document storage: Here's the link https://www.theguardian.com/info/2018/nov/30/bye-bye-mongo-hello-postgres https://www.theguardian.com/info/2018/nov/30/bye-bye-mongo-h...
  - guanzo 6y ago
    > Automatically generating database indexes on application startup is probably a bad idea. aw crap. oh well it probably doesn't matter for my small-ish application.
- Quekid5 6y ago
  I was actually amazed that a big CMS/E-commerce vendor proudly proclaimed in a sales meeting that they were on MongoDB. I suppose salespeople probably aren't into the nitty-gritty, but their tech people should have warned them about this. Maybe they were just trying to pull our collective leg, but I suppose that why I was at that meeting. It was obviously an instant 'No'.
  - leviathant 6y ago
    There aren't a lot of CMS/Ecommerce vendors that sit on MongoDB, so maybe we were in a meeting together! Even if we weren't - as a sales engineer on a large CMS/ECommerce platform with merchants running $150M+ in annual revenue, with an average client retention of seven years, and two decades of agency experience behind the decisions around building that platform, if you instantly said no just because of MongoDB, maybe you don't know as much about MongoDB as you think you do. I came from a SQL background myself, and had reservations based on all the things I'd read about MongoDB as we decided to build a platform after doing things bespoke for two decades, but time has proven our architecture choices out. It's easy to be proud of something that works well.

jfkebwjsbx 6y ago

> Even tools people joke about a lot like mysql never gave me this sort of data corruption. People rightfully joked about MySQL when they had the non-ACID engine. Same for MongoDB. A database that loses data when properly used is a joke. Yes, there are use cases out there for fast non-guaranteed writes. No, 99% of companies don’t have them.

Something1234 6y ago
Can you name a use case for a fast non guaranteed write?
- Jweb_Guru 6y ago
  Sure. Data that people don't care about enough to be worried about losing--for example, time series data from an unimportant remote sensor. Should this data be recorded at all? Maybe not, but if should then a best-effort recording may be fine. It may even be all that's possible.
  - mbreese 6y ago
    I wouldn’t go as far as to say an “unimportant” remote sensor... but I think you’re correct in spirit. I could think of an instance where you’d like to log data, but the occasional datapoint being missing wouldn’t be terrible. Maybe something like a temperature monitor — you’d like to have a record of the temperature by the minute, but if a few records dropped out, you’d be able to guess the missing values from context. Something like the data monitoring equivalent of UDP vs TCP.
- throwaway744678 6y ago
  Analytics: you don't want to slow down your app, and you don't care if you lose a few records in the process.
  - zbentley 6y ago
    Importantly, in analytics workloads, it is very important to know roughly how many writes aren't making it. Otherwise your analytics system sucks.

vorticalbox 6y ago

> Mongo can call this function multiple times with the same session object. isn't that the point? you can use a session to do multi actions within that session.

inglor 6y ago
If you have code that looks like this: withTransaction(async session => { await Promise.all([someOp(sesson), someOtherOp(session)]); }); Mongo may retry running it (calling the function again) if a "TransaientTransactionError" is raised (the transaction is retried from the client side rather than at the cluster). However, when the driver calls your function again it doesn't invalidate the `session` object - so previous calls to the same function can make updates to the database. Let's say `someOp` does something that causes the transaction to retry and `someOtherOp` is doing something non-mongo-related in the meantime (like pulling a value from redis). Now `someOtherOp` reached the mongo part of its code and it is executing it happily with the same session object (so operations succeed although they really shouldn't) The point of transactions like you said is to perform multiple operations atomically and for them to happen "exactly once or not at all". With Mongo in practice it is very easy to get "Once and some leftovers from a previous attempt".
- IgorPartola 6y ago
  Sorry, I haven’t had my coffee yet. If I am reading this correctly, either someOp() or someOtherOp() may execute first, no? And if you introduce an external database, why do you expect Mongo to handle that rollback? Say someOtherOp() increments a Redis value by 1. If that part executed first since both are asynchronous here, what would a Mongo session have to do with it? What exactly would invalidating that session object do here? And what would the session object do after it was invalidated?
  - deleted 6y ago
    [deleted]
  - waheoo 6y ago
    It sounds like the old session object is reused and becomes live again or something.

hodgesrm 6y ago

> Even tools people joke about a lot like mysql never gave me this sort of data corruption. That's about a decade out of date at this point. MySQL/InnoDB is the standard table engine and corruption is exceedingly rare. As of 2014, when I last directly worked on MySQL prod systems, there was no practical difference from PostgreSQL in terms of transactional guarantees. That includes APIs like JDBC which we used for billions of transactions.

morelisp 6y ago
MySQL still has no transactional DDL (and I think still even autocommits if you try). This is a major difference from Postgres which I believe supports everything short of dropping tables.
- takeda 6y ago
  I wouldn't use that particular thing against MySQL. DDL normally supposed to be always outside of a transaction, it's just PostgreSQL feature that you can use them inside and be able to rollback. BTW I'm convinced you also can drop table within a transaction in PostgreSQL.
  - morelisp 6y ago
    No, MySQL stands out here. Postgres, SQL Server, DB2, and Firebird all give at least some way to do some major DDL transactionally. Usability varies (e.g. Oracle supports a very specific kind of change that is not its normal DDL statements), but it's at least possible. https://wiki.postgresql.org/wiki/Transactional_DDL_in_PostgreSQL:_A_Competitive_Analysis#Oracle https://wiki.postgresql.org/wiki/Transactional_DDL_in_Postgr... That MySQL autocommits is also even worse than just "doesn't support it."
  - dragonwriter 6y ago
    > DDL normally supposed to be always outside of a transaction A basic element of the relational model is that metadata is stored as relational data and that the same guarantees that apply to manipulating main data in the database apply to manipulating the schema metadata. It's true that many real relational databases compromise on this element in various ways at times, but it is absolutely not the case that DDL “is supposed to be” non-transactional.

lossolo 6y ago

When I was evaluating MongoDb couple of years ago (around the time they were switching to WiredTiger engine), I've found memory leak in their Node.js client on day one, I've submitted a ticket on their Jira and the same time I had a look at other issues they had there. I saw there memory leak after memory leak, memory corruption everywhere, data disappearing without any reason, segfaults etc. After that MongoDB was dropped as a candidate for a DB in project I was working on, we went with Postgres and never regretted it.

takeda 6y ago

MySQL is less of a joke than MongoDB is. They similarly started by someone who didn't know anything about databases and learned about them on the go. Actually both of them started as much faster alternatives to other databases, both ended up having complete rewrite of its engine written by someone from outside that knew their stuff. MySQL ISAM then MyISAM and then InnoDB (written by an outsider). Similarly MongoDB got a WiredTiger written. The thing is that MySQL is older so it went through all of it earlier, but it still suffers from poor decisions from the past. This is contrasting with PostgreSQL, where correctness and reliability was #1 from the beginning. It started as an awfully slow database, but performance for improved and we now have correct, reliable and fast database.

berns 6y ago
MySQL is no joke, nothing is perfect and Postgresql is not 100% reliable. Remember: Transaction ID wraparound: https://twitter.com/bcantrill/status/1110647418008133632 https://twitter.com/bcantrill/status/1110647418008133632 Incorrect use of fsync: https://news.ycombinator.com/item?id=19119991 https://news.ycombinator.com/item?id=19119991
- goatinaboat 6y ago
  MySQL is no joke If you were around back in the day you will remember the MySQL team claiming that no one needed transactions or referential integrity, that you should just do it yourself in the application...
  - twic 6y ago
    Does MySQL support check constraints yet?
    - dnissley 6y ago
      It does finally! In 8.0.16+
  - beatrobot 6y ago

hintymad 6y ago

Just curious, what was the reason that your team decided to work around the problem instead of migrating away from MongoDB?

capableweb 6y ago
Not the author but done similar things (patching something rather than migrating away from it). Usually it's way more work to migrate away than just patching it again to fit your use-case. Once you find yourself having to patch it too often, you start thinking about migrating away. Then the research slowly begins ad-hoc until it hits "seems we need to migrate away now, otherwise we're spending too much time working around something / fixing their broken shit", that's when you sit down and decide to migrate away from it. Also would depend on how long time you think the application will be around. You're building an MVP to evaluate something? Just hack together whatever will work (then throw away). You're maintain software for a library/archive that will most likely stick around for a long time, even if they say it's just temporary? Do decisions that will help in the future, always.
inglor 6y ago
We have a complicated system and migration is ~3 months we won't be shipping features. We have a roadmap we need to meet and so far we have been trying to spill money on it rather than developers (paying mongo atlas) and adding features incrementally as Mongo gets them (like transactions). If this wasn't a startup we would probably rewrite.

xeromal 6y ago

I just want remind people that this video exists. https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs

sorokod 6y ago
Well, does dev/null supports sharding?

g0ldenb0ugh 6y ago

> We had to write a ton of logic around their poor implementation and I was shocked to see the code underneath. Just wondering, did you submit a bug report to them about this? If so, any response?

inglor 6y ago
Yes, several times - we pay Mongo Atlas over $5000 per month. We reported it immediately at the highest severity and we pay for the highest tier or support - we tried to collaborate as soon as possible. It sort of went "over their head".
- g0ldenb0ugh 6y ago
  Can you link the JIRA ticket? I use the Node driver heavily and have contributed several PRs to it in the past, would be more than happy to fix this since it seems fairly bad.
  - inglor 6y ago
    Hi, I'm not sure where I'm supposed to look for JIRAs at but we discussed this with Mongo Atlas via several support tickets and channels. For example at #00636235 "how to avoid TransientTransactionError".