4 ms·

From the engineering friends I know at Stripe, you'd be correct: there's a strong cultural push to not question the status quo / prevailing wisdom. On the tech

From the engineering friends I know at Stripe, you'd be correct: there's a strong cultural push to not question the status quo / prevailing wisdom. On the tech side, they continue to build on top of a broken foundation (MongoDB), resulting in millions of man hours wasted dealing with the complete lack of transactions. Mongo now has transactions, but last I heard Stripe was still running a very outdated version and spending absurd amounts of time dealing with issues that transactions would have entirely avoided. If you suggested that maybe it'd be worth changing course to something like Postgres because of the insane amount of work being wasted, you'd be shut down for not being optimistic enough.

wbronitsky 7y ago
It is always amazing to me that people downvote actual anecdotes that highlight the arguments being made, while at the same time upvote rampant speculation that has no basis in fact. Having worked at Stripe, these exact examples came up more than once. I can verify their authenticity.
rat9988 7y ago
Wouldn't be easier to just upgrade mongodb, which is no longer broken? Unless its api is not backward compatible?
- stingraycharles 7y ago
  From what I understand there have been various attempts at this, as well as other data storage solutions, but they always run into issues. MongoDB is being used as a database and data warehouse and there are always complications.
  - Slartie 7y ago
    Updating any non-trivial layer in your stack usually gets you into "complications" - that alone must never be an acceptable reason to be stuck on old versions, especially if there are serious problems with them. You simply chew through those complications. Again and again. And try to optimize your application for less complications in the future, if possible.
cookiecaper 7y ago
The fact that Stripe ever used MongoDB is dubious enough, but that they've endured on it well past the fad's expiry is reason enough for me to avoid doing business with them in the future. Handling money is serious business and if one insists on using the unproven flavor-of-the-week to do so, they'd better make sure it has at least the fundamental requirements to accomplish its task, e.g., transactions/MVCC. Stripe apparently failed to do so, and has chosen to propagate that failure for many years. One can make excuses for the spunky startup with a handful of employees needing to save time by using what they know or whatever, but Stripe is well past that point, and serious people should've taken over by now. I'd expect "replace Mongo with a real database" to be near the top of the todo list for any serious people.
- nkohari 7y ago
  Transactions aren't the only solution for that problem space, and using them can introduce other (likely even harder-to-solve) problems with distributed systems.
- danpalmer 7y ago
  FWIW, having worked with the Stripe API for ~5 years and put all our sales through them for the lifetime of the business, I wouldn’t have guessed they used MongoDB. The API is mostly excellent, and it has always presented a consistent view of the world to our side. The same cannot he said for a number of other services we use, including those from some companies with supposedly very good tech credentials. We’ve had to back out of integrations due to the clear evidence that they are non transactional and/or eventually consistent.
  - benologist 7y ago
    You couldn't tell what backend they use just from their API unless there's some errant exceptions leaking information. Their test API might give us some clues. It can only purge your data one record at a time, at about 1.5 records a second, and locks your API credentials for however long it takes to crash or finish iterating through each object. I don't know what storage engine would have that constraint but they've stuck with it for years and simulate a fast test API with a whole other piece of software instead of just having a fast test API. What storage options don't have the ability to delete all your records at once quickly? If it was MongoDB there would surely be an index attaching the data to your account. Same with PostgreSQL.
eternalban 7y ago
I'm currently dealing with precisely this exact scenario and database. Dated Mongo, homegrown ORM, a closet full of opaque hacks that are apparently all related to some MongoDB feature, etc. And I don't know if this is a cultural issue with MongoDB users, but my god the database has its hooks throughout the stack, all the way to the UI. It is amazing to see shit like this in 2019 and I've seen some heaps over the past few decades in this field. But the optimism point resonated. Before reading your comment, I actually reflected on whether I am being pessimistic. I'll tell you why: at this point in the game, cleaning up the software would effectively require a rewrite. And the company is in no position to do this. They do not have the talent, leadership, or effective recruiting necessary to undertake a meaningful second system effort. So I think I've come to the realization that the job is a poor fit and I should move on, but while I am here it actually wouldn't help pointing out the obvious issues with the system. There really is no point.

wbronitsky 7y ago

It is always amazing to me that people downvote actual anecdotes that highlight the arguments being made, while at the same time upvote rampant speculation that has no basis in fact. Having worked at Stripe, these exact examples came up more than once. I can verify their authenticity.

rat9988 7y ago

Wouldn't be easier to just upgrade mongodb, which is no longer broken? Unless its api is not backward compatible?

stingraycharles 7y ago
From what I understand there have been various attempts at this, as well as other data storage solutions, but they always run into issues. MongoDB is being used as a database and data warehouse and there are always complications.
- Slartie 7y ago
  Updating any non-trivial layer in your stack usually gets you into "complications" - that alone must never be an acceptable reason to be stuck on old versions, especially if there are serious problems with them. You simply chew through those complications. Again and again. And try to optimize your application for less complications in the future, if possible.

cookiecaper 7y ago

The fact that Stripe ever used MongoDB is dubious enough, but that they've endured on it well past the fad's expiry is reason enough for me to avoid doing business with them in the future. Handling money is serious business and if one insists on using the unproven flavor-of-the-week to do so, they'd better make sure it has at least the fundamental requirements to accomplish its task, e.g., transactions/MVCC. Stripe apparently failed to do so, and has chosen to propagate that failure for many years. One can make excuses for the spunky startup with a handful of employees needing to save time by using what they know or whatever, but Stripe is well past that point, and serious people should've taken over by now. I'd expect "replace Mongo with a real database" to be near the top of the todo list for any serious people.

nkohari 7y ago
Transactions aren't the only solution for that problem space, and using them can introduce other (likely even harder-to-solve) problems with distributed systems.
danpalmer 7y ago
FWIW, having worked with the Stripe API for ~5 years and put all our sales through them for the lifetime of the business, I wouldn’t have guessed they used MongoDB. The API is mostly excellent, and it has always presented a consistent view of the world to our side. The same cannot he said for a number of other services we use, including those from some companies with supposedly very good tech credentials. We’ve had to back out of integrations due to the clear evidence that they are non transactional and/or eventually consistent.
- benologist 7y ago
  You couldn't tell what backend they use just from their API unless there's some errant exceptions leaking information. Their test API might give us some clues. It can only purge your data one record at a time, at about 1.5 records a second, and locks your API credentials for however long it takes to crash or finish iterating through each object. I don't know what storage engine would have that constraint but they've stuck with it for years and simulate a fast test API with a whole other piece of software instead of just having a fast test API. What storage options don't have the ability to delete all your records at once quickly? If it was MongoDB there would surely be an index attaching the data to your account. Same with PostgreSQL.

eternalban 7y ago

I'm currently dealing with precisely this exact scenario and database. Dated Mongo, homegrown ORM, a closet full of opaque hacks that are apparently all related to some MongoDB feature, etc. And I don't know if this is a cultural issue with MongoDB users, but my god the database has its hooks throughout the stack, all the way to the UI. It is amazing to see shit like this in 2019 and I've seen some heaps over the past few decades in this field. But the optimism point resonated. Before reading your comment, I actually reflected on whether I am being pessimistic. I'll tell you why: at this point in the game, cleaning up the software would effectively require a rewrite. And the company is in no position to do this. They do not have the talent, leadership, or effective recruiting necessary to undertake a meaningful second system effort. So I think I've come to the realization that the job is a poor fit and I should move on, but while I am here it actually wouldn't help pointing out the obvious issues with the system. There really is no point.