3 ms·
In my previous job, at a payment service provider, I wrote a "velocity" system. This was required to block transactions that met certain conditions within a cer
by leejo 8y ago
In my previous job, at a payment service provider, I wrote a "velocity" system. This was required to block transactions that met certain conditions within a certain time frame. A typical example was: block transactions that are from the same IP if that happens more than 5 times in a 10 minute period. Or: block transactions from the same card number if it is used more than 3 times in a minute.
The problem was that the "velocity" blocking requirements could be completely arbitrary, and could be anything a merchant required as long as they used any of the data that was available from the transaction/previous transactions - block a transaction if it's more than 500 GBP and we've had 3 other transactions greater than 500 GBP in the same country in the previous 10 minutes.
This essentially meant we would either have to do dev work whenever a merchant had a new "velocity" rule, write our own complex framework to handle the addition of arbitrary rules, or just stuff the data into a schemaless NoSQL store and then leverage the power of the engine's query syntax as part of the merchant's configuration.
We went for the schemaless NoSQL solution and we used mongodb v1.6 - around 2010 IIRC, before it reached peak hype, before it "fixed" a lot of the out of the box defaults, before it got a lot of hate. It worked perfectly and ran from the time we deployed it until i left the company a few years later. Maybe I left a mass of technical debt, but the solution ended up being so simple and so little code I doubt it.
The other nice thing was that the velocity system was not essential - if the time to do a velocity check took more than 0.05s we would ignore it. If the backend wasn't responding we ignored it. If the write to the storage failed it didn't matter. We didn't need to keep more than a couple of hours worth of data. If we lost all of that data it didn't matter.
I don't know if I'd take the same approach now, nine years later, but at the time the use of mongodb worked perfectly. It was only a couple of weeks work, and solved the problem elegantly. So in response to the original question: yes, but as usual RTFM and make sure the pros and the cons fit your use cases.
- kevingadd 8y agoI'm not clear on why the datastore needed to be schemaless though? It sounds like the schema for the transaction data was well understood and only the blocking requirements were not, but the blocking requirements were formulas/assertions over the well-formed data.
- leejo 8y agoThe transaction data was well understood, but did change over time. It was well normalised in a relational backend that was (at the time) a few hundred GB, but it did not have any partitioning. The problem we were trying to solve was that allowing, essentially, arbitrary queries against the main database had the potential to cause problems because we would have to ensure that those queries were optimal and all the relations were correct before we allowed those rules to be put into place - i.e. more dev and DBA work and potentially schema changes (addition of more indexes, etc). It was far easier to have an ephemeral duplicate of the transaction data with a flat structure, a single key value document, as that mitigated the risks and removed the aforementioned problem. The velocity system was essentially a firewall in front of the main database server - this was one of the main requirements, the velocity system must not have any affect on the load of the main transaction system. We could have added another db server, but that would have put us on the route to more servers, more complexity, etc, as we were self hosting at the time.
- jacques_chester 8y agoWhat I'm hearing is that you used MongoDB as a read cache, rather than as a database per se.
- rujuladanh 8y agoIf you have almost no guarantees to provide, almost any solution would have worked. In other words, you claim it worked for years, but anything would if you can lose all data, reboot it, have no real-time requirements, etc. In summary, it is not really a useful data point on whether MongoDB is useful or not.
- leejo 8y ago> If you have almost no guarantees to provide, almost any solution would have worked. Within reason. If the failure rate is 1 in 1000 then it's probably not as a big a concern as 1 in 10 depending on your use case > In summary, it is not really a useful data point on whether MongoDB is useful or not. As I said in the last sentence - it worked perfectly. In other words the "no guarantees to provide" never actually came up and I don't recall us having any data loss, response time, or write issues.