4 ms·

This is "how I would make Twitter scale" all over again. Unless you're an engineer working on that particular system, you can't credibly give advice on how to i

by luigi 16y ago

This is "how I would make Twitter scale" all over again. Unless you're an engineer working on that particular system, you can't credibly give advice on how to improve it. Doing so is an insult to the profession.

megaman821 16y ago
What a bull-shit statement. So Twitter engineers can't take any outside advice because those people aren't working on Twitter's particular system. No one else in the world could possibly know how to make large scale messaging systems work. I am not saying any of the arm-chair analysts are right but it is pretty presumptuous to dismiss them outright.
- cubes 16y ago
  While luigi's statement is a little extreme, I think the gist is right. Yes outsiders can offer advice on a system, but it's really difficult to offer sound advice without specific details of the system. Perhaps outsider advice shouldn't be dismissed outright, but it ought be taken with a grain of salt.
  - megaman821 16y ago
    I very much agree that the devil is in the details, and the largest detail here is how Foursquare goes about calculating badges. So without knowing that the article can't fully evaluate alternatives. That has less to do with the author not being an engineer on Foursquare's team rather Foursquare not detailing the calculation. Maybe on Riak map/reduce is fast enough and Foursquare can shard by checkin, which would be much easier to balance. Maybe they should just implement their own sharding based on user activity. Maybe a few SSD drives on their Postgres server would solve they issue. They through the issue out there, if nothing else works tell us why.
    - cubes 16y ago
      Some informative links, in case you haven't seen them already: * http://www.quora.com/What-caused-Foursquares-downtime-on-October-4-2010 http://www.quora.com/What-caused-Foursquares-downtime-on-Oct... * http://blog.foursquare.com/2010/10/05/so-that-was-a-bummer/ http://blog.foursquare.com/2010/10/05/so-that-was-a-bummer/ * https://groups.google.com/group/mongodb-user/browse_thread/thread/66752f49af68619?pli=1 https://groups.google.com/group/mongodb-user/browse_thread/t... Someone on this thread suggested that Foursquare's performance problem is related to the calculation of badges. I'm not sure where this idea came from. I haven't seen any mention of the issue being related to badges in first or secondhand sources. That said, Mongo DB's map/reduce operation would not be a reasonable solution at this time. Mongo DB's map/reduce performance is, at present, somewhat lacking because it runs via the javascript engine which is currently single threaded. I know there are plans to improve performance of Mongo DB's javascript engine by switching to V8, but I don't know if V8 is multithreaded. Often design decisions that look bad in hindsight get baked in early. By the time you realize that an alternate design would yield better performance, there may be too much data to migrate so you just have to live with it.
      - megaman821 16y ago
        I am not sure if badge calculation is the bottleneck. Foursquare did say the calculation is why they have to keep everything in memory though. I just assumed that they sharded by user to keep all of the user's check-ins in the user document and that they are doing calculation with that set of data.
      - whakojacko 16y ago
        harryh mentioned it here: http://news.ycombinator.com/item?id=1769909 http://news.ycombinator.com/item?id=1769909 Computing the badges online (ie at each checkin time) requires having the whole db in ram for acceptable performance.
metageek 16y ago
Maybe, but this article isn't really a look at Foursquare's problem; it's looking at these other databases, using Foursquare's problem as a lens. Not so useful for Foursquare, but useful for people who are considering which database to use.
- blutonium 16y ago
  Not so. They built a system that abused their datastore, and switching it out wouldn't solve anything.
Qz 16y ago
Did you not read to the end for the Conclusion paragraph?

by luigi 16y ago

megaman821 16y ago

What a bull-shit statement. So Twitter engineers can't take any outside advice because those people aren't working on Twitter's particular system. No one else in the world could possibly know how to make large scale messaging systems work. I am not saying any of the arm-chair analysts are right but it is pretty presumptuous to dismiss them outright.

cubes 16y ago
While luigi's statement is a little extreme, I think the gist is right. Yes outsiders can offer advice on a system, but it's really difficult to offer sound advice without specific details of the system. Perhaps outsider advice shouldn't be dismissed outright, but it ought be taken with a grain of salt.
- megaman821 16y ago
  I very much agree that the devil is in the details, and the largest detail here is how Foursquare goes about calculating badges. So without knowing that the article can't fully evaluate alternatives. That has less to do with the author not being an engineer on Foursquare's team rather Foursquare not detailing the calculation. Maybe on Riak map/reduce is fast enough and Foursquare can shard by checkin, which would be much easier to balance. Maybe they should just implement their own sharding based on user activity. Maybe a few SSD drives on their Postgres server would solve they issue. They through the issue out there, if nothing else works tell us why.
  - cubes 16y ago
    Some informative links, in case you haven't seen them already: * http://www.quora.com/What-caused-Foursquares-downtime-on-October-4-2010 http://www.quora.com/What-caused-Foursquares-downtime-on-Oct... * http://blog.foursquare.com/2010/10/05/so-that-was-a-bummer/ http://blog.foursquare.com/2010/10/05/so-that-was-a-bummer/ * https://groups.google.com/group/mongodb-user/browse_thread/thread/66752f49af68619?pli=1 https://groups.google.com/group/mongodb-user/browse_thread/t... Someone on this thread suggested that Foursquare's performance problem is related to the calculation of badges. I'm not sure where this idea came from. I haven't seen any mention of the issue being related to badges in first or secondhand sources. That said, Mongo DB's map/reduce operation would not be a reasonable solution at this time. Mongo DB's map/reduce performance is, at present, somewhat lacking because it runs via the javascript engine which is currently single threaded. I know there are plans to improve performance of Mongo DB's javascript engine by switching to V8, but I don't know if V8 is multithreaded. Often design decisions that look bad in hindsight get baked in early. By the time you realize that an alternate design would yield better performance, there may be too much data to migrate so you just have to live with it.
    - megaman821 16y ago
      I am not sure if badge calculation is the bottleneck. Foursquare did say the calculation is why they have to keep everything in memory though. I just assumed that they sharded by user to keep all of the user's check-ins in the user document and that they are doing calculation with that set of data.
    - whakojacko 16y ago
      harryh mentioned it here: http://news.ycombinator.com/item?id=1769909 http://news.ycombinator.com/item?id=1769909 Computing the badges online (ie at each checkin time) requires having the whole db in ram for acceptable performance.

metageek 16y ago

Maybe, but this article isn't really a look at Foursquare's problem; it's looking at these other databases, using Foursquare's problem as a lens. Not so useful for Foursquare, but useful for people who are considering which database to use.

blutonium 16y ago
Not so. They built a system that abused their datastore, and switching it out wouldn't solve anything.

Qz 16y ago

Did you not read to the end for the Conclusion paragraph?