3 ms·
Here are a few suggestions based on 6+ years in adtech (which have just came to a close, never again thank you): * Use a Queue. RabbitMQ is quite good. Instead
by harel 11y ago
Here are a few suggestions based on 6+ years in adtech (which have just came to a close, never again thank you):
* Use a Queue. RabbitMQ is quite good. Instead of writing to files, generate data/tasks on the queue and have them consumed by more than one client. The clients should handle inserting the data to the database. You can control the pipe by the number of clients you have consuming tasks, and/or by rate limiting them. Break those queue consuming clients to small pieces. Its ok to queue item B on the queue while processing item A.
* If you data is more fluid and changing all the time, and/or if it comes in JSON serializable format, consider switching to postgresql ^9.4, and use the JSONB columns to store this data. You can index/query those columns and performance wise its on par (or surpasses) MongoDB.
* Avoid AWS at this stage. like commented by someone here - bare metal is a better friend to you. You'll also know exactly how much you're paying each month. no surprises. I can't recommend Softlayer enough.
* Don't over complicate things. If you can think of a simple solution to something - its preferable than the complicated solution you might have had before.
* If you're going the queue route suggested above, you can pre-process the data while you get it in. If its going to be placed into buckets, do it then, if its normalised - do it then. The tasks on the queue should be atomic and idempotent. You can use something like memcached if you need your clients to communicate between eachother (like checking if a queue item is not already processed by another consumer and thus is locked).
- unknownknowns 11y ago> * If you data is more fluid and changing all the time, and/or if it comes in JSON serializable format, consider switching to postgresql ^9.4, and use the JSONB columns to store this data. You can index/query those columns and performance wise its on par (or surpasses) MongoDB. Would it be worthwhile to switch from MongoDB to Postgres (when MySQL is being used alongside Mongo)? I'd have to do lots of testing with it to know for sure though, I guess. I use Mongo for timeseries-like data, and have >1 billion documents in Mongo currently. Querying super old data is pretty slow (and not needed generally), but querying recent data (recent being the last month) is "fast enough".
- fludlight 11y agoCheck out HDF5 for your old data.
- Normal_gaussian 11y agoIf you can reliably externalise the timestamps into Postgres for the documents, then yes. Just index and primarily query on the timestamps and you will be golden. Also switching from mysql to Postgres should see performance gains (though unlikely to be a bottle neck).
- harel 11y agoIf your setup makes sense for you, its fine - stay with it. focus on real problem, not ones people on this thread think you have :) I'm not one of those mongo bashers. I've used mongo before for the same purpose you do - time series statistical data along side a postgres db for everything else. And it worked fine for me. However, at that time postgresql was pre 9.4. Today I would have kept it all in postgres and simply archived older data if needed. And to be fair, its not even a "problem" until you need to filter by one data source and sort by the other. We had to be creative when we got to those problems. As for my choice of Postgres over mySql - that is for 2 reasons - first being that postgres is not owned by Oracle, and second that postgres is more mature, feature rich and reliable than mysql (i.e., better).
- click170 11y agoI don't mean to diss your experiences clearly softlayer worked for you in your environment but I'm honestly shocked to see Softlayer suggested here at all. We just finished trying to set up some servers in softlayer and it was thee biggest pain in the ass we've had to deal with in a LONG time. Everything required a support ticket. And that's if you're willing to overlook their reputation. Just Google "softlayer spam" and you'll see what I mean right away. Glad that it worked for you. But wow... Never again.
- harel 11y agoIn about 4+ years using Softlayer, I didn't have a single problem with them. The bill at the end of the month was reasonable for the amount of servers and traffic, and even one time where an errant worker on our app generated terabytes of wasted bandwidth and our bill for that topped our monthly one, they figured out a way to make it go away by pooling out bandwidth from all the servers together. They could have easily forced us to pay it. I guess its all down to personal experiences, and mine was great.
- faizshah 11y agoI agree with stay away from AWS, but centurylink does have lower prices for compute servers comparatively and google cloud has a better pricing model for compute. I'm about to test this out empirically but based on my initial research this is what I think. But I agree with 'bare metal is your friend'.