3 ms·

Billing was by month. This was a business decision. It makes sense too. Billing requires a lot of man hours to pull off. Invoices, auditing, excel spreadsheets

Billing was by month. This was a business decision. It makes sense too. Billing requires a lot of man hours to pull off. Invoices, auditing, excel spreadsheets, etc. It's a people problem not an engineering problem. I'm not sure of the disconnect here, but de-duplication is not trivial. If you do it every day nothing is easier than if you do it every month. Doing it for all time is completely infeasible. There was not a database of every piece of social media data sent out the door. That's what you would need to make sure not to record the entry again. All we had were flat files in s3. Big flat files. It took hours to download and merge them all. Once we had a database (Cassandra) it was updated continually (by a kafka consumer) and we could query it in a few minutes.

Jupe 8y ago
Fair enough... but somehow your system knew of failed attempts and the need to retry, correct? I'm sure I'm trivializing a rather complicated workflow, but if there's a way to detect failed attempts, there should be a way to remove them from the billing. (perhaps a separate S3 file that can be used to reduce the number of charged attempts, or split the ones you have into 'success' and 'fail' versions?) Oh, and Cassandra is great... used it for years. Just beware of over-sized results. I've seen run-away queries crash the instance(s) running the query; ugly stuff. Glad you found a solution that works!
hinkley 8y ago
I’m not sure I follow your argument here. You switched to a system where the data was downloaded continuously. You could have, with the shell script, easily: * downloaded the files daily * sorted and removed the intraday duplication in each new file. * sorted and removed intraweek duplication each week and on the first of the month. * You could give your customers a weekly estimate of their bill. All of this would spread the work out and tell you when the system had glitched out weeks before the results were due.

Jupe 8y ago

Fair enough... but somehow your system knew of failed attempts and the need to retry, correct? I'm sure I'm trivializing a rather complicated workflow, but if there's a way to detect failed attempts, there should be a way to remove them from the billing. (perhaps a separate S3 file that can be used to reduce the number of charged attempts, or split the ones you have into 'success' and 'fail' versions?) Oh, and Cassandra is great... used it for years. Just beware of over-sized results. I've seen run-away queries crash the instance(s) running the query; ugly stuff. Glad you found a solution that works!

hinkley 8y ago

I’m not sure I follow your argument here. You switched to a system where the data was downloaded continuously. You could have, with the shell script, easily: * downloaded the files daily * sorted and removed the intraday duplication in each new file. * sorted and removed intraweek duplication each week and on the first of the month. * You could give your customers a weekly estimate of their bill. All of this would spread the work out and tell you when the system had glitched out weeks before the results were due.