5 ms·

So wrong it's hard to know where to start...

by davidu 18y ago

Retric 18y ago
How about: 1) It's not just a DB problem.
- antidaily 18y ago
  exactly.
- hank777 18y ago
  Uh... yes it is. All scaling problems are database problems. ALL of them. They can be solved by different strategies such as caching, distributed design, etc. But at the end of the day the problem is always that you cant get to the data fast enough. That is a database problem.
  - nostrademons 18y ago
    Including systems that don't use databases, or are you defining a database loosely as "anything that stores data"?
    - hank777 18y ago
      Yes, I am defining database as "anything that stores data" It does not have to be SQL for me to consider it a database problem. And in fact my solution is well above the sql level.
      - Excedrin 18y ago
        Did you mean 100M users per box?
  - sanswork 18y ago
    Unless it's a problem with a bad algorithm slowing down the processing of the data. So not all scaling problems are database problems.
  - edw519 18y ago
    "All scaling problems are database problems." This is an argument in semantics. You suggest storing all users physical addresses in memory across multiple machines. You also advocate "shard splitting" a la B-tree indexes which split when the node's population reaches 2n. OK, so when a split occurs, how do you maintain that massive table in memory. Do you lock it? Do you examine your existing queue of requests (which may soon point to the wrong node)? How do you handle new queue requests during "shard split maintenance"? Calling these "database problems" is waving your hands. Why not just call them "computer problems"?
    - hank777 18y ago
      Well ok. I dont have to call them database problems, but I object to someone saying they are not database problems. I agree it is a bit of semantics and I would not have brought it up if he hadnt. Although I have to say I consider this work (and the good questions you ask) to be in the database domain. Regarding when shard split how you maintain the table, the easy answer is Terracotta. It is amazing for this and handles all that kind of stuff totally in the background. If you didnt have terracotta it would be a harder problem indeed. It handles locking and all of that magic for you. I highly recommend it. Regarding how you handle queue requests while the split is going on, what happens is that while the split is happening, the original shard continues to respond, but all new updates are sent to the old shard and the new one that is being created in the split, so during a split there is addition messaging going on. Once its done, the original shards range of users is cut back, but actually it is still storing all the old data. At that point you can delete the data that has been moved to the new shard - though this is not necessary.
      - edw519 18y ago
        "and all of that magic" Right. Just because we didn't write it doesn't mean it's magic. There is still a lot of work to be done. It can get very complicated for high velocity apps. And it takes time. So you're right. Bigger minds than the two of us are probably struggling with this (and lots of other stuff) as we write.
        18y ago
  - ericb 18y ago
    Most scaling problems are database problems. Not all. For example, I load tested my last rails app (load testing is my background-I have consulted for major pharma, tax, and other companies) and on a wimpy inherited server, processor maxed out first for the rails processes. I popped in a second CPU, then was able to get to the point where we could support 200 concurrent users, which was all we needed at the time. I have seen limitations on bandwidth, firewall problems, load balancer misconfiguration, poorly written queries, memory usage,processor usage, thread deadlocks, and licensing limits (max database connections on a trial license) all be bottlenecks. At the larger scale, yes, database is more often the problem, but it's not clear twitter should be using a database in the way they are, anyway. Your statement goes too far.
    - hank777 18y ago
      Yeah, certainly at the small scale there can be a myriad of reasons why a system doesn't perform. At a large scale its hard to imagine real world scenarios where storage has not been the bottleneck. Regarding whether twitter should be using a database, unless they dont want to store there data they need a "database". Does that mean sql? No. It just means storage. As paul Graham has demonstrated a file system can be a great database. But it still a database.
      - Retric 18y ago
        Think about a MMO with 100 users in 1 room, the number of relationstips in that room is (n-1)^2. Add 100 more users walking around and you now need to send 4x as many location updates. You will quickly run out of bandwidth long before storing that much information is a problem. Now consider a voting system you have 1,000 items and 1,000 users and you want to show how much people like them like each item? How well does that work when there are 100,000 users and 100,000 items?
hank777 18y ago
Its fascinating funny how you are so smart you can't even deign to share your wisdom on this. If I am wrong, tell me where. Given that I have spent the last several years studying/developing distributed databases if I just don't know what I am doing I'd certainly like someone as smart as you to tell me.
- cstejerean 18y ago
  "Given that I have spent the last several years studying/developing distributed databases..." That doesn't make your opinion of what's wrong with Twitter any more valid than the crap TC keeps printing unless you happen to also have lots of internal details of the Twitter architecture.
  - hank777 18y ago
    true, but it makes me curious what brilliance someone bring to the table when they have nothing specific to say other than you are wrong. Also given that much of what I am talking about is running code, I at least know that it solves a major class of problems. If it doesnt apply to twitter it should be pretty easy to explain why.
- nostrademons 18y ago
  I don't use Twitter, but as I understand it, it's a massively many-to-many messaging service, where it's not unusual to follow hundreds of people's tweets and have thousands of people follow yours. Can you imagine the performance if every message required hundreds of disk writes as the "received" copies get spread across a hundred shards?
  - hank777 18y ago
    Actually, this is the beauty of the design: All of the writes happen in parallel. So there if you write to 2000 recipients, but all of the writes happen in parallel you have no bottleneck. And any time a given machine's load gets too heavy, you do a shard split. You really dont care how much writing you do as long as on every machine you write to is not being hit too hard. So if a given server has lets say 10,000 users on it. And each one of those users receives 10 updates an hour. Then it would mean handling, lets say 100,000 writes in an hour, or 27 writes a second. If this was the threshold for this machine (which would be low) you then split the shard so that it then is only handling 13 or 14 writes per second.

So wrong it's hard to know where to start...

by davidu 18y ago

So wrong it's hard to know where to start...

Retric 18y ago

How about: 1) It's not just a DB problem.

antidaily 18y ago
exactly.
hank777 18y ago
Uh... yes it is. All scaling problems are database problems. ALL of them. They can be solved by different strategies such as caching, distributed design, etc. But at the end of the day the problem is always that you cant get to the data fast enough. That is a database problem.
- nostrademons 18y ago
  Including systems that don't use databases, or are you defining a database loosely as "anything that stores data"?
  - hank777 18y ago
    Yes, I am defining database as "anything that stores data" It does not have to be SQL for me to consider it a database problem. And in fact my solution is well above the sql level.
    - Excedrin 18y ago
      Did you mean 100M users per box?
- sanswork 18y ago
  Unless it's a problem with a bad algorithm slowing down the processing of the data. So not all scaling problems are database problems.
- edw519 18y ago
  "All scaling problems are database problems." This is an argument in semantics. You suggest storing all users physical addresses in memory across multiple machines. You also advocate "shard splitting" a la B-tree indexes which split when the node's population reaches 2n. OK, so when a split occurs, how do you maintain that massive table in memory. Do you lock it? Do you examine your existing queue of requests (which may soon point to the wrong node)? How do you handle new queue requests during "shard split maintenance"? Calling these "database problems" is waving your hands. Why not just call them "computer problems"?
  - hank777 18y ago
    Well ok. I dont have to call them database problems, but I object to someone saying they are not database problems. I agree it is a bit of semantics and I would not have brought it up if he hadnt. Although I have to say I consider this work (and the good questions you ask) to be in the database domain. Regarding when shard split how you maintain the table, the easy answer is Terracotta. It is amazing for this and handles all that kind of stuff totally in the background. If you didnt have terracotta it would be a harder problem indeed. It handles locking and all of that magic for you. I highly recommend it. Regarding how you handle queue requests while the split is going on, what happens is that while the split is happening, the original shard continues to respond, but all new updates are sent to the old shard and the new one that is being created in the split, so during a split there is addition messaging going on. Once its done, the original shards range of users is cut back, but actually it is still storing all the old data. At that point you can delete the data that has been moved to the new shard - though this is not necessary.
    - edw519 18y ago
      "and all of that magic" Right. Just because we didn't write it doesn't mean it's magic. There is still a lot of work to be done. It can get very complicated for high velocity apps. And it takes time. So you're right. Bigger minds than the two of us are probably struggling with this (and lots of other stuff) as we write.
      - 18y ago
- ericb 18y ago
  Most scaling problems are database problems. Not all. For example, I load tested my last rails app (load testing is my background-I have consulted for major pharma, tax, and other companies) and on a wimpy inherited server, processor maxed out first for the rails processes. I popped in a second CPU, then was able to get to the point where we could support 200 concurrent users, which was all we needed at the time. I have seen limitations on bandwidth, firewall problems, load balancer misconfiguration, poorly written queries, memory usage,processor usage, thread deadlocks, and licensing limits (max database connections on a trial license) all be bottlenecks. At the larger scale, yes, database is more often the problem, but it's not clear twitter should be using a database in the way they are, anyway. Your statement goes too far.
  - hank777 18y ago
    Yeah, certainly at the small scale there can be a myriad of reasons why a system doesn't perform. At a large scale its hard to imagine real world scenarios where storage has not been the bottleneck. Regarding whether twitter should be using a database, unless they dont want to store there data they need a "database". Does that mean sql? No. It just means storage. As paul Graham has demonstrated a file system can be a great database. But it still a database.
    - Retric 18y ago
      Think about a MMO with 100 users in 1 room, the number of relationstips in that room is (n-1)^2. Add 100 more users walking around and you now need to send 4x as many location updates. You will quickly run out of bandwidth long before storing that much information is a problem. Now consider a voting system you have 1,000 items and 1,000 users and you want to show how much people like them like each item? How well does that work when there are 100,000 users and 100,000 items?

hank777 18y ago

Its fascinating funny how you are so smart you can't even deign to share your wisdom on this. If I am wrong, tell me where. Given that I have spent the last several years studying/developing distributed databases if I just don't know what I am doing I'd certainly like someone as smart as you to tell me.

cstejerean 18y ago
"Given that I have spent the last several years studying/developing distributed databases..." That doesn't make your opinion of what's wrong with Twitter any more valid than the crap TC keeps printing unless you happen to also have lots of internal details of the Twitter architecture.
- hank777 18y ago
  true, but it makes me curious what brilliance someone bring to the table when they have nothing specific to say other than you are wrong. Also given that much of what I am talking about is running code, I at least know that it solves a major class of problems. If it doesnt apply to twitter it should be pretty easy to explain why.
nostrademons 18y ago
I don't use Twitter, but as I understand it, it's a massively many-to-many messaging service, where it's not unusual to follow hundreds of people's tweets and have thousands of people follow yours. Can you imagine the performance if every message required hundreds of disk writes as the "received" copies get spread across a hundred shards?
- hank777 18y ago
  Actually, this is the beauty of the design: All of the writes happen in parallel. So there if you write to 2000 recipients, but all of the writes happen in parallel you have no bottleneck. And any time a given machine's load gets too heavy, you do a shard split. You really dont care how much writing you do as long as on every machine you write to is not being hit too hard. So if a given server has lets say 10,000 users on it. And each one of those users receives 10 updates an hour. Then it would mean handling, lets say 100,000 writes in an hour, or 27 writes a second. If this was the threshold for this machine (which would be low) you then split the shard so that it then is only handling 13 or 14 writes per second.