3 ms·
I'm a little surprised that 37signals hadn't sharded some of their applications. Sharding can be hard because you often don't have a logical way to separate ou
by mdasen 18y ago
I'm a little surprised that 37signals hadn't sharded some of their applications. Sharding can be hard because you often don't have a logical way to separate out data. For example, on a site like this, you could shard articles based on year, but then if you wanted to get all the articles that a single user had submitted, you'd have to query several servers.
However, the inherent design of 37signals applications make sharding a lot easier. With 37signals apps, you're not dealing with all the users in the system or all of the data. You're dealing with one company's data and the users from that company. You could much more easily put data from companies with names A-I in DB1 and J-Z in DB2 and call it a day than most sites can shard. You wouldn't have to worry about things like losing cross database joins or having to lookup in multiple databases for certain reads.
In fact, with Basecamp, there's 6 possible domains that your site can be a subdomain of (projectpath.com, clientsecton.com. . .). Each one of those could point to a different set of app/DB servers. If you ran metrics on usage, you'd have to run it 6 times and aggregate the data, but in terms of user usage, each could be its own little separate system not even knowing the others existed.
Sharding becomes a problem when you can't fit all the data that will be used together on a single server. For example, Facbeook could shard by college, but you're allowed to have friends at different colleges from your own. So, you have to do things like cross shard lookups and that mess. With 37signals products (to my knowledge), everyone gets an account with your company's site and interacts with your company's stuff and if they want to interact with another company's stuff they log out and log back in with a different username/pass. So, everything can be neatly sharded by company since you don't have things that cross that boundary.
- ROFISH 18y agoIn the case of Facebook, instead you would shard by feature. Every database server has replicateds of the user/friends data, but there'd be a separate server for the events or for groups.
- mdasen 18y agoAnd what about when there are too many events for a single shard? And maybe I'm wrong, but I feel like the friends data would be the biggest issue. Eventually, you get into some decently ugly stuff if you get big enough. Most people never have to worry about sharding (even 37signals doesn't), but there aren't just easy answers.
- axod 18y agoThe easy answer is to avoid the db wherever possible. For example, when people decide to write a chatroom, they often decide to put each message in a database. Which is sort of a ridiculous thing to do. Using in memory tables, flat files, etc where possible beats databases.
- jhawk28 18y agoNot sure how one would shard the data based on a RoR app.
- mdasen 18y agoHypothetical: So, you have 26 companies using Basecamp (Company A-Company Z). Your single database server can't handle all 26 companies so you decide to shard them. Your previous setup: Webhead1-|- Database1 Webhead2/ So, you have two webheads reading from one database. You need to shard and have a second database: Webhead1--Database1 (Companies A-M) Webhead2--Database2 (Companies N-Z) Easy! Think of it this way: because a person at Company A is never accessing Company Z's data, there doesn't need to be any communication between the databases. So, it shards in the same way that Wordpress shards: different sites are on different servers. With Basecamp, you don't need every company to be on the same server because you don't move data between companies. But you don't need a separate server for every company either. So, you can put a bunch of companies on the same server and a bunch on another server and you're done. There's nothing magic or RoR specific.
- joshwa 18y agohttp://github.com/fiveruns/data_fabric/tree/master http://github.com/fiveruns/data_fabric/tree/master Or monkey-patch activerecord yourself to use a shard-index.