5 ms·

That does make sense. I am just trying to think of it in terms of my current personal project. How I could lay it out, but the problem is it is mostly ad hoc qu

by jjuel 8y ago

That does make sense. I am just trying to think of it in terms of my current personal project. How I could lay it out, but the problem is it is mostly ad hoc queries since it is historical data. Like storing game results and statistics. I think relational for my current situation works best. However, this is pretty cool in the future for projects I would have just used relational before. Watching the videos about it have been a great help. Helps me figure out the use cases.

grogenaut 8y ago
What's the TPS on any of your ad hoc queries? If it's low and very specialized than relational or a lucene or redshift type system likely is what you want, eg more like ERP. If say you're doing dotabuff type stuff then just chunking the records you want out by player or even just an S3 Blob that is updated might be better. Then the rest of your tables are literally just indexes to the large amounts of data. It totes depends on what you're doing. The big issue with relational really is that you hit a point where transactions just can't keep up with locking across the cluster. If your system doesn't do this, then your next problem is just sharding on tons of data. If you don't have enough data then relational is fine. If you do have that much data then you're going to have to figure out a sane sharding and joining strategy across a cluster, which is more work. At some point in size of data you either have each user on their own shards of boxes, or you are keeping only indexes in the databases to retrieve other data. Both take a bunch of work. Much of the motivation behind that talk and amazon's drive away from SQL is really just the business risk of hitting that scaling limit while you have a business that is doubling in revenue. Your options when you hit a limit on sql (esp with transactions) is to heavily re-architect it. And that takes a while, so you have to pause growth during that time. At which point a second mover can come roaring past you. If I was building somthing like dotabuff I'd go with relational database sharded by users or some type of columnar database for the ad-hoc querying. Tho really for dotabuff you could build almost the entire site to be statically served off of cloudfront and just re-crunch data when a new match comes in. However I'd have dynamo as the front end data ingest record keeper. Things like "did I see this game data already" to avoid using things like FIFO queues. Happy to talk about it more if you provide more examples of what you're trying to map. And again, if you don't have these problems, relational is just totally fine. You can scale quite high on relational, it's just that you run the risk of hitting that asymptotal tipping point where the system just goes down and you can't get bigger hardware. If you're not going to hit it, the tool you know will serve you well. Another random thought, I find that people who thing in SQL forget how damn awkward it is... For instance, ask someone who uses Relational regularly how to create tables for users's shipping address. They'll spout a denormalized form that works in many many places. Ask a college hire the same question, you might get more than one table. The thing that is obvious to the regular user will be non-obvious and confusing to the college hire. And I would argue that the table layout is somthing that evolved over 30+ years of the industry doing it over and over.

by jjuel 8y ago

grogenaut 8y ago

What's the TPS on any of your ad hoc queries? If it's low and very specialized than relational or a lucene or redshift type system likely is what you want, eg more like ERP. If say you're doing dotabuff type stuff then just chunking the records you want out by player or even just an S3 Blob that is updated might be better. Then the rest of your tables are literally just indexes to the large amounts of data. It totes depends on what you're doing. The big issue with relational really is that you hit a point where transactions just can't keep up with locking across the cluster. If your system doesn't do this, then your next problem is just sharding on tons of data. If you don't have enough data then relational is fine. If you do have that much data then you're going to have to figure out a sane sharding and joining strategy across a cluster, which is more work. At some point in size of data you either have each user on their own shards of boxes, or you are keeping only indexes in the databases to retrieve other data. Both take a bunch of work. Much of the motivation behind that talk and amazon's drive away from SQL is really just the business risk of hitting that scaling limit while you have a business that is doubling in revenue. Your options when you hit a limit on sql (esp with transactions) is to heavily re-architect it. And that takes a while, so you have to pause growth during that time. At which point a second mover can come roaring past you. If I was building somthing like dotabuff I'd go with relational database sharded by users or some type of columnar database for the ad-hoc querying. Tho really for dotabuff you could build almost the entire site to be statically served off of cloudfront and just re-crunch data when a new match comes in. However I'd have dynamo as the front end data ingest record keeper. Things like "did I see this game data already" to avoid using things like FIFO queues. Happy to talk about it more if you provide more examples of what you're trying to map. And again, if you don't have these problems, relational is just totally fine. You can scale quite high on relational, it's just that you run the risk of hitting that asymptotal tipping point where the system just goes down and you can't get bigger hardware. If you're not going to hit it, the tool you know will serve you well. Another random thought, I find that people who thing in SQL forget how damn awkward it is... For instance, ask someone who uses Relational regularly how to create tables for users's shipping address. They'll spout a denormalized form that works in many many places. Ask a college hire the same question, you might get more than one table. The thing that is obvious to the regular user will be non-obvious and confusing to the college hire. And I would argue that the table layout is somthing that evolved over 30+ years of the industry doing it over and over.