5 ms·

Notes on MongoDB, GridFS, sharding and deploying in the cloud

54 points by mvip 15y ago

nknight 15y ago
I might be missing something due lack of experience with MongoDB, but what did disk I/O have to do with the secondaries falling behind, and why did effectively eliminating one of the secondaries "fix" that? The description makes me think there's a pathological case lurking in MongoDB that needs to be looked at more closely.
- mvip 15y ago
  The reason why the secondaries were falling behind was because the I/O was maxed out on the primary. It was barely able to keep up with the regular load, and hence reducing the load (by only having one node replicating instead of two) reduce the I/O load significantly. That said, reducing the number of secondaries was just a short-term solution. The long-term solution is to convert to a sharded environment and offload the one maxed out node.
  - mnutt 15y ago
    That makes sense, but unless bandwidth is the limiting factor, it sounds like there is a potential optimization where the I/O is only performed once for both secondaries.
    - mvip 15y ago
      I'm sure there is plenty of room for optimizing many things in Mongo. I don't know if secondaries can replicate from each other (if one is behind the other), but that would make sense. Mongo is a great product, but there is room for improvement of course.
  - nknight 15y ago
    I'm still confused. Why does replication hit the disk on the primary? If the secondaries aren't already way behind, shouldn't the data be going straight from RAM out the network interface?
    - mvip 15y ago
      In our case with GridFS, the objects wouldn't fit in RAM. Only perhaps the last two objects would. The node had 4GB of RAM and many of the objects sent Mongo were around 0.5-4GB.
      - nknight 15y ago
        Ahhh, and presumably replication doesn't start until the entire object is loaded. Now it makes sense. There are ways that could be fixed, but at least now it's just a lack of optimization instead of sounding like something's completely turned on its head. :)
  - nosequel 15y ago
    Wouldn't the proper behavior be to provide some sort of back pressure so that it cannot decide for you to no longer backup your data to the replica set? I would expect that if you say you want data replicated, that if it can't handle that under load, it would back off on the load that it is capable of instead of going the pathological route of simply not backing up. If under load the primary goes down with a serious disk failure, you now just lost a ton of data that you expected were being backed up all along.
    - nknight 15y ago
      You can apparently invoke a level of synchronous behavior in MongoDB with the getLastError command: http://www.mongodb.org/display/DOCS/getLastError+Command http://www.mongodb.org/display/DOCS/getLastError+Command Conceptually, its abilities look a lot like Cassandra's "configurable consistency" model.
      - jbellis 15y ago
        Consistency and durability are two different things.
mvip 15y ago
Also, if you're curious about bandwidth limitations in this particular cloud, take a look at these benchmarks: http://viktorpetersson.com/2012/01/23/benchmarking-virtual-network-drivers-under-freebsd-9/ http://viktorpetersson.com/2012/01/23/benchmarking-virtual-n... http://viktorpetersson.com/2012/01/24/benchmarking-and-tuning-freebsds-virtio-network-driver/ http://viktorpetersson.com/2012/01/24/benchmarking-and-tunin... (Using CloudSigma as the cloud-vendor and FreeBSD 9 as the host)

54 points by mvip 15y ago

I might be missing something due lack of experience with MongoDB, but what did disk I/O have to do with the secondaries falling behind, and why did effectively eliminating one of the secondaries "fix" that? The description makes me think there's a pathological case lurking in MongoDB that needs to be looked at more closely.

mvip 15y ago
The reason why the secondaries were falling behind was because the I/O was maxed out on the primary. It was barely able to keep up with the regular load, and hence reducing the load (by only having one node replicating instead of two) reduce the I/O load significantly. That said, reducing the number of secondaries was just a short-term solution. The long-term solution is to convert to a sharded environment and offload the one maxed out node.
- mnutt 15y ago
  That makes sense, but unless bandwidth is the limiting factor, it sounds like there is a potential optimization where the I/O is only performed once for both secondaries.
  - mvip 15y ago
    I'm sure there is plenty of room for optimizing many things in Mongo. I don't know if secondaries can replicate from each other (if one is behind the other), but that would make sense. Mongo is a great product, but there is room for improvement of course.
- nknight 15y ago
  I'm still confused. Why does replication hit the disk on the primary? If the secondaries aren't already way behind, shouldn't the data be going straight from RAM out the network interface?
  - mvip 15y ago
    In our case with GridFS, the objects wouldn't fit in RAM. Only perhaps the last two objects would. The node had 4GB of RAM and many of the objects sent Mongo were around 0.5-4GB.
    - nknight 15y ago
      Ahhh, and presumably replication doesn't start until the entire object is loaded. Now it makes sense. There are ways that could be fixed, but at least now it's just a lack of optimization instead of sounding like something's completely turned on its head. :)
- nosequel 15y ago
  Wouldn't the proper behavior be to provide some sort of back pressure so that it cannot decide for you to no longer backup your data to the replica set? I would expect that if you say you want data replicated, that if it can't handle that under load, it would back off on the load that it is capable of instead of going the pathological route of simply not backing up. If under load the primary goes down with a serious disk failure, you now just lost a ton of data that you expected were being backed up all along.
  - nknight 15y ago
    You can apparently invoke a level of synchronous behavior in MongoDB with the getLastError command: http://www.mongodb.org/display/DOCS/getLastError+Command http://www.mongodb.org/display/DOCS/getLastError+Command Conceptually, its abilities look a lot like Cassandra's "configurable consistency" model.
    - jbellis 15y ago
      Consistency and durability are two different things.

mvip 15y ago

Also, if you're curious about bandwidth limitations in this particular cloud, take a look at these benchmarks: http://viktorpetersson.com/2012/01/23/benchmarking-virtual-network-drivers-under-freebsd-9/ http://viktorpetersson.com/2012/01/23/benchmarking-virtual-n... http://viktorpetersson.com/2012/01/24/benchmarking-and-tuning-freebsds-virtio-network-driver/ http://viktorpetersson.com/2012/01/24/benchmarking-and-tunin... (Using CloudSigma as the cloud-vendor and FreeBSD 9 as the host)