5 ms·

I currently work for a company that uses Aerospike quite heavily. In the past couple weeks, we have begun to notice data inconsistencies in our counters. We are

by Ankhers 11y ago

I currently work for a company that uses Aerospike quite heavily. In the past couple weeks, we have begun to notice data inconsistencies in our counters. We are seeing fluctuations in the data, despite having no decrement operations. We have the enterprise edition of Aerospike, allowing us to be in constant contact with their support team and developers. A couple weeks later, and we still have no idea why this is happening. When dealing with monetary values, these fluctuations are very bad for us. Needless to say, we have begun migrating away from Aerospike.

smt88 11y ago
Yikes. What made you choose Aerospike in the first place?
- Ankhers 11y ago
  Sadly, a requirement of the application is that it needs to be backward compatible with the previous system. I'm sure many of you know, this leads to quite a few issues.
obituary_latte 11y ago
Can you add any detail to this anecdote? It's interesting and important, but detail might help steer others appropriately. What kind of inconsistency? What kind of fluctuation?
- Ankhers 11y ago
  I'm current working in AdTech. We are using counters to keep track of a lot of things that are important to us (e.g., reasons for not bidding on a given campaign, money spent within a given transaction, number of bid requests we get by exchange, etc). I personally have found two different, yet very similar, data fluctuations. The first I found when debugging an issue. I noticed the counters going up, dip down, then continuing up. Rinse repeat. (e.g., 158 -> 160 -> 158 -> 170 -> 175 -> 173 -> 180) The second I found was when trying to debugging the previous issue. I noticed the counters were essentially cycling. (e.g., 158 -> 160 -> 170 -> 158 -> 160 -> 170). This just repeated for the duration we watched the counters (approximately five minutes). Please note that I used small numbers here. The counters I was monitoring were in the hundred millions, and I saw decrements in average between 2-3k.
  - 23david 11y ago
    Very strange. I've never encountered something like that with any database or cache. Wonder if it's somehow related to the way that your cluster is setup? How big is this cluster? Are you writing and reading to the entire cluster, or do you have certain nodes that you write to and others that you read from?
    - Ankhers 11y ago
      I have to admit, I'm not 100% on the entire configuration. However... We have two clusters of 8 nodes each. Each cluster is setup with 2 factor replication. The clusters are setup with cross datacenter replication. Your read / write question is a little hard to answer. In Aerospike, a given key will always reside on the same node, something to do with how they optimize their storage. Which means that anytime you write to, or read from, a given key your query will always be routed to the same node.
      - ngx1234 11y ago
        When Aerospike ships XDR batches it does not replay events, it just re-syncs the data. This is true even for increments. So if cluster A has 10 increments of n to n+10, and cluster B has 20 increments of n to n+20, it's possible XDR will ship A to B and cluster B gets set to n+10. XDR only guarantees data consistency if writes are 15 mins apart and your cross datacenter network doesn't go down. The suggested method of solving this is to have two keys, one for each cluster, and XDR both keys. Then add them together in the app. You can maybe do it through a lua script, though I haven't tried.
      - kayser68 11y ago
        Did you see this effect during split brain, or while all nodes were up and operating? I think you would have mentioned that what you saw was split brain..... With datacenter replication, do you have a loop (multi-master) configured? Aerospike uses async replication, so using multi-master counters can be tricky....
    - bbq 11y ago
      The analysis in the article shows that Aerospike is designed, intentionally or not, as a loosely accurate data store. It doesn't matter how you set it up or use it.
jboggan 11y ago
What is the rationale for storing monetary values in this sort of system? Not being snarky, just legitimately curious what scale of service could possibly necessitate that and what solutions didn't work beforehand.
- CyrusL 11y ago
  Transactions in AdTech are different than normal payments. For example, imagine an ad campaign spending $30k/month at a rate of $5 per 1,000 impressions. The customer may want their budget spread evenly throughout the month, so the software sets a daily budget of $1000. But this really represents 200,000 daily impressions, each of which is a transaction that subtracts from the available balance in real-time. The buyers software is talking to an ad exchange and keeping track of the budget every time an individual impression is won. To add some more complexity, the impressions are probably billed as second-price auctions, so they aren't all exactly $0.005 each. Some are $0.00493, some are $0.00471, ect. Each one of these numbers is reported back from the exchange to the buyer's software in real time and the buyer is responsible for managing their budget. This is just an example, but hopefully it illustrates how it can become impractical to account for this kind of thing using something more traditional like PostgreSQL. It would be reasonable to log all the impressions to something like Hadoop for the analytical piece of the software, but there needs to be something more real-time for budgeting to prevent overspending. The big ad exchanges can host hundreds of thousands or even millions of auctions per second, so not turning off bidding can be very costly. This process of auctioning ad impressions across many buyers through an API is called real-time bidding.
  - hurin 11y ago
    Why does this need to be in real-time, if their daily budget is $1000, you can still wait quite a bit and then apply increments in aggregate (e.g. hourly)? More so it sounds the customers aren't inter-connected - it hardly seems like the complex distributed problem.
    - cldellow 11y ago
      The impressions may be sparse, e.g. say you're retargeting CEOs (demographic information you're getting from a DSP) who have visited your website in the last month (via a pixel you drop) who are in New York City (via a geoIP DB). So, fine, a probabilistic model might work well. And you might decide to bid on 100% of impressions. And you might decide that you have to bid $200 CPM to win -- which you're OK doing, because they're sparse. And then say that FooConf happens in NYC and your aggressive $200 CPM bid 100% of the time blows out your budget. Often you can charge the customer you're acting on behalf of your actual spend + X% up to their campaign threshold. So you really want to ensure that you spend as much as possible, without spending too much. Pacing is hard. Google AdWords, for example, only promise to hit your budget +/- 20% over a 1 month period.
digitalzombie 11y ago
What are you migrating to ? As is away from Aerospike to... ? Thanks, very interesting anecdote/case.
- Ankhers 11y ago
  We are first evaluating MongoDB. I believe the main reason behind this is we are already using Mongo in other parts of our application, so there is no additional setup when converting. Note that nothing is set in stone. The decision to begin migrations only happened today. It is possible that we will end up using some other technology altogether, or even we find out the issues we are having with Aerospike and continue using that service.
  - deleted 11y ago
    [deleted]
  - misframer 11y ago
    You should take a look at "Call me maybe: MongoDB" [0] [0] https://aphyr.com/posts/284-call-me-maybe-mongodb https://aphyr.com/posts/284-call-me-maybe-mongodb
    - sans-culottes 11y ago
      Not to mention "Call me maybe: Redis" [0]. Rinse and repeat for every NoSQL database out there. Point is, @aphyr skewers everybody. Aerospike is just the flavor of the month. [0] https://aphyr.com/posts/283-call-me-maybe-redis https://aphyr.com/posts/283-call-me-maybe-redis
      - jbergens 11y ago
        I think FoundationDb got great scores from Aphyr, sadly it is now owned by Apple. For the others the big problem is when they promise some kind of ACID and cannot achieve it, if they were explicit with what the supported all customers could make informed decisions. Relations systems are generally bad ad horizontal scaling and can get very slow with full ACID over many servers.
cmpxchg16b 11y ago
This is interesting, and of course I have a couple of questions, but only two of them really matter: what client are you using, and are you using the cross-datacenter (XDR) replication functionality? We* tested the increment functionality heavily (300K-1M aggregate ops/sec) before we turned it on in revenue service. We use it for a couple of different things, event counting is absolutely the major use case. In a single-cluster world, it works phenomenally well. In a XDR world, things get a little tricky, and we had to change the way our application logic worked to compensate for it. Any more information you can share about your use case? *a big ad tech company that uses Aerospike heavily
perlgeek 11y ago
When storing financial data, I'd certainly go with some kind of event sourcing: store deltas / financial transactions, not counters. The counters are just a sum over all deltas. If performance is an issue, you can make the counters available in a second database that's only for reading, and updated from the original deltas.
- obstinate 11y ago
  It seems from this article that Aerospike can lose acknowledged writes, so that would not be enough to save them.
drvsrinivasan 11y ago
Today, the inconsistencies here were diagnosed by company and Aerospike to be caused by two clusters connected with XDR concurrently writing data to the same counter and shipping to each other and intentionally overwriting some data (bad design that somehow slipped through the cracks). So, this issue is unrelated to the Jepsen network partitioning tests that is the subject of the original article. The work that @Aphyr is doing is very valuable and much appreciated. (Aerospike Founder)

by Ankhers 11y ago

smt88 11y ago

Yikes. What made you choose Aerospike in the first place?

Ankhers 11y ago
Sadly, a requirement of the application is that it needs to be backward compatible with the previous system. I'm sure many of you know, this leads to quite a few issues.

obituary_latte 11y ago

Can you add any detail to this anecdote? It's interesting and important, but detail might help steer others appropriately. What kind of inconsistency? What kind of fluctuation?

Ankhers 11y ago
I'm current working in AdTech. We are using counters to keep track of a lot of things that are important to us (e.g., reasons for not bidding on a given campaign, money spent within a given transaction, number of bid requests we get by exchange, etc). I personally have found two different, yet very similar, data fluctuations. The first I found when debugging an issue. I noticed the counters going up, dip down, then continuing up. Rinse repeat. (e.g., 158 -> 160 -> 158 -> 170 -> 175 -> 173 -> 180) The second I found was when trying to debugging the previous issue. I noticed the counters were essentially cycling. (e.g., 158 -> 160 -> 170 -> 158 -> 160 -> 170). This just repeated for the duration we watched the counters (approximately five minutes). Please note that I used small numbers here. The counters I was monitoring were in the hundred millions, and I saw decrements in average between 2-3k.
- 23david 11y ago
  Very strange. I've never encountered something like that with any database or cache. Wonder if it's somehow related to the way that your cluster is setup? How big is this cluster? Are you writing and reading to the entire cluster, or do you have certain nodes that you write to and others that you read from?
  - Ankhers 11y ago
    I have to admit, I'm not 100% on the entire configuration. However... We have two clusters of 8 nodes each. Each cluster is setup with 2 factor replication. The clusters are setup with cross datacenter replication. Your read / write question is a little hard to answer. In Aerospike, a given key will always reside on the same node, something to do with how they optimize their storage. Which means that anytime you write to, or read from, a given key your query will always be routed to the same node.
    - ngx1234 11y ago
      When Aerospike ships XDR batches it does not replay events, it just re-syncs the data. This is true even for increments. So if cluster A has 10 increments of n to n+10, and cluster B has 20 increments of n to n+20, it's possible XDR will ship A to B and cluster B gets set to n+10. XDR only guarantees data consistency if writes are 15 mins apart and your cross datacenter network doesn't go down. The suggested method of solving this is to have two keys, one for each cluster, and XDR both keys. Then add them together in the app. You can maybe do it through a lua script, though I haven't tried.
    - kayser68 11y ago
      Did you see this effect during split brain, or while all nodes were up and operating? I think you would have mentioned that what you saw was split brain..... With datacenter replication, do you have a loop (multi-master) configured? Aerospike uses async replication, so using multi-master counters can be tricky....
  - bbq 11y ago
    The analysis in the article shows that Aerospike is designed, intentionally or not, as a loosely accurate data store. It doesn't matter how you set it up or use it.

jboggan 11y ago

What is the rationale for storing monetary values in this sort of system? Not being snarky, just legitimately curious what scale of service could possibly necessitate that and what solutions didn't work beforehand.

CyrusL 11y ago
Transactions in AdTech are different than normal payments. For example, imagine an ad campaign spending $30k/month at a rate of $5 per 1,000 impressions. The customer may want their budget spread evenly throughout the month, so the software sets a daily budget of $1000. But this really represents 200,000 daily impressions, each of which is a transaction that subtracts from the available balance in real-time. The buyers software is talking to an ad exchange and keeping track of the budget every time an individual impression is won. To add some more complexity, the impressions are probably billed as second-price auctions, so they aren't all exactly $0.005 each. Some are $0.00493, some are $0.00471, ect. Each one of these numbers is reported back from the exchange to the buyer's software in real time and the buyer is responsible for managing their budget. This is just an example, but hopefully it illustrates how it can become impractical to account for this kind of thing using something more traditional like PostgreSQL. It would be reasonable to log all the impressions to something like Hadoop for the analytical piece of the software, but there needs to be something more real-time for budgeting to prevent overspending. The big ad exchanges can host hundreds of thousands or even millions of auctions per second, so not turning off bidding can be very costly. This process of auctioning ad impressions across many buyers through an API is called real-time bidding.
- hurin 11y ago
  Why does this need to be in real-time, if their daily budget is $1000, you can still wait quite a bit and then apply increments in aggregate (e.g. hourly)? More so it sounds the customers aren't inter-connected - it hardly seems like the complex distributed problem.
  - cldellow 11y ago
    The impressions may be sparse, e.g. say you're retargeting CEOs (demographic information you're getting from a DSP) who have visited your website in the last month (via a pixel you drop) who are in New York City (via a geoIP DB). So, fine, a probabilistic model might work well. And you might decide to bid on 100% of impressions. And you might decide that you have to bid $200 CPM to win -- which you're OK doing, because they're sparse. And then say that FooConf happens in NYC and your aggressive $200 CPM bid 100% of the time blows out your budget. Often you can charge the customer you're acting on behalf of your actual spend + X% up to their campaign threshold. So you really want to ensure that you spend as much as possible, without spending too much. Pacing is hard. Google AdWords, for example, only promise to hit your budget +/- 20% over a 1 month period.

digitalzombie 11y ago

What are you migrating to ? As is away from Aerospike to... ? Thanks, very interesting anecdote/case.

Ankhers 11y ago
We are first evaluating MongoDB. I believe the main reason behind this is we are already using Mongo in other parts of our application, so there is no additional setup when converting. Note that nothing is set in stone. The decision to begin migrations only happened today. It is possible that we will end up using some other technology altogether, or even we find out the issues we are having with Aerospike and continue using that service.
- deleted 11y ago
  [deleted]
- misframer 11y ago
  You should take a look at "Call me maybe: MongoDB" [0] [0] https://aphyr.com/posts/284-call-me-maybe-mongodb https://aphyr.com/posts/284-call-me-maybe-mongodb
  - sans-culottes 11y ago
    Not to mention "Call me maybe: Redis" [0]. Rinse and repeat for every NoSQL database out there. Point is, @aphyr skewers everybody. Aerospike is just the flavor of the month. [0] https://aphyr.com/posts/283-call-me-maybe-redis https://aphyr.com/posts/283-call-me-maybe-redis
    - jbergens 11y ago
      I think FoundationDb got great scores from Aphyr, sadly it is now owned by Apple. For the others the big problem is when they promise some kind of ACID and cannot achieve it, if they were explicit with what the supported all customers could make informed decisions. Relations systems are generally bad ad horizontal scaling and can get very slow with full ACID over many servers.

cmpxchg16b 11y ago

This is interesting, and of course I have a couple of questions, but only two of them really matter: what client are you using, and are you using the cross-datacenter (XDR) replication functionality? We* tested the increment functionality heavily (300K-1M aggregate ops/sec) before we turned it on in revenue service. We use it for a couple of different things, event counting is absolutely the major use case. In a single-cluster world, it works phenomenally well. In a XDR world, things get a little tricky, and we had to change the way our application logic worked to compensate for it. Any more information you can share about your use case? *a big ad tech company that uses Aerospike heavily

perlgeek 11y ago

When storing financial data, I'd certainly go with some kind of event sourcing: store deltas / financial transactions, not counters. The counters are just a sum over all deltas. If performance is an issue, you can make the counters available in a second database that's only for reading, and updated from the original deltas.

obstinate 11y ago
It seems from this article that Aerospike can lose acknowledged writes, so that would not be enough to save them.

drvsrinivasan 11y ago

Today, the inconsistencies here were diagnosed by company and Aerospike to be caused by two clusters connected with XDR concurrently writing data to the same counter and shipping to each other and intentionally overwriting some data (bad design that somehow slipped through the cracks). So, this issue is unrelated to the Jepsen network partitioning tests that is the subject of the original article. The work that @Aphyr is doing is very valuable and much appreciated. (Aerospike Founder)