5 ms·

I'm the author (Sam Mcleod) of this post and just saw this was posted on HN. I'm quite a bit further down the track now and should have a much better write up

by mrmondo 12y ago

I'm the author (Sam Mcleod) of this post and just saw this was posted on HN. I'm quite a bit further down the track now and should have a much better write up to post in the next week or so time providing. I can say that I'm hitting over 1,000,000 IOP/s on 4k random reads and around 850,000 IOP/s random 4k writes per 1U unit without cutting any corners with data safety. What I haven't finished yet is the cross-unit replication tuning and the load balancing / multipathing. So far everything has gone to plan and is progressing well as a serious replacement for our 'traditional' storage. Regardless, there's still quite a lot of testing to be done and once I have more of that completely I'll post an update with some serious numbers and observations.

moe 12y ago
cross-unit replication tuning and the load balancing / multipathing. That's going to be interesting. At those speeds (1.5GB/s, 850k IOPS) you must bump[1] into all kinds of DRBD bottlenecks? Also I'm very curious about your approach to redundancy/multipath. Dual-primary or failover? [1] http://blog.gmane.org/gmane.comp.linux.drbd/page=41 http://blog.gmane.org/gmane.comp.linux.drbd/page=41
- mrmondo 12y ago
  I have no illusions that I'll be able to maintain such performance with DRBD replicating and iSCSI overheads. My current feeling on load balancing is to keep it simple, have half the clients treat node A as their active storage and the other half using node B. If node A becomes unavailable failover will occur and node B will serve all clients. As far as multipathing goes - my approach is that I honestly have no idea. That's something that's going to be learnt along the way and I'm in uncertain if it'll make it into the final build or not. Edit: I can't get that link to load on my connection at present but I've bookmarked to look at tomorrow. If you have any experience / advice I'm always very open to assistance!
  - FlyingAvatar 12y ago
    I was quite amused to see this posted as we just built nearly the exact same thing about 6 months ago (minus the NVMe drives which weren't available). We are using ZFSonLinux on top of DRBD instead of mdmadm, which has been surprisingly successful. The biggest frustration that we met was using LACP in trying to get a better data rate out of DRBD. Regardless of the LACP mode, we were not able to get DRBD to scale to full bandwidth (20gbps) and had to settle for using it for redundancy only. I imagine with your setup (read: much faster peak write capability), you might actually be able to bottleneck DRBD pretty well over a single 10GBE pipe if you were writing at peak capacity. I'll be interested to see if you happen upon a work-around. Load balancing as you've suggested is an intriguing compromise. Have you tried doing a two-way heartbeat setup where each both boxes have their own active IP and can fail over to each other?
    - tedchs 12y ago
      I wonder if the lack of help from LACP was because DRBD is using only 1 TCP session, which would only get hashed to one or the other member link?
      - cgb_ 12y ago
        Might be a case where MPTCP (http://www.multipath-tcp.org/ http://www.multipath-tcp.org/) would help with aggregating multiple links.
        mrmondo 12y ago
        Correct me if I'm wrong but I'm not sure MPTCP would help as LACP is generally SRC/DST MAC based and shares a uses a single IP?
        cgb_ 12y ago
        You are right in that LACP bonding methods cannot increase the throughput of a single flow to gt any single link in the group (the balancing method can utilise src/dst MAC, IP and sometime L4 ports). MPTCP establishes multiple subflows across individual IP paths and can load balance or failover across all subflows. Applications do not need to be rewritten to take advantage of it. I'm not sure if that includes kernel modules like DRBD though. I suppose someone needs to find out :)
baruch 12y ago
I would strongly recommend also testing for when things fail. This is the hardest part in troubleshooting and building a reliable system. You can make do with READ/WRITE LONG if the disks are SAS to generate medium errors, be aware that it may kill your disk if you do it too much to the same disk. You can also use relatively inexpensive SAS/SATA link error injectors from Quarch. Be sure to test hot pulls of the disks to simulate a hard failure. Simulating other soft failures is much harder, sometimes the disk just stops responding but doesn't go offline so the system finds it harder to detect the case and may reach timeout conditions.
- mrmondo 12y ago
  I couldn't agree with you more, in the time that I have set aside to establish the new storage I have around 75% of it dedicated to testing, around half of that is dedicated to failure scenarios. A have a list of 'bad things to do' that I'll be executing and reporting on.

by mrmondo 12y ago

moe 12y ago

cross-unit replication tuning and the load balancing / multipathing. That's going to be interesting. At those speeds (1.5GB/s, 850k IOPS) you must bump[1] into all kinds of DRBD bottlenecks? Also I'm very curious about your approach to redundancy/multipath. Dual-primary or failover? [1] http://blog.gmane.org/gmane.comp.linux.drbd/page=41 http://blog.gmane.org/gmane.comp.linux.drbd/page=41

mrmondo 12y ago
I have no illusions that I'll be able to maintain such performance with DRBD replicating and iSCSI overheads. My current feeling on load balancing is to keep it simple, have half the clients treat node A as their active storage and the other half using node B. If node A becomes unavailable failover will occur and node B will serve all clients. As far as multipathing goes - my approach is that I honestly have no idea. That's something that's going to be learnt along the way and I'm in uncertain if it'll make it into the final build or not. Edit: I can't get that link to load on my connection at present but I've bookmarked to look at tomorrow. If you have any experience / advice I'm always very open to assistance!
- FlyingAvatar 12y ago
  I was quite amused to see this posted as we just built nearly the exact same thing about 6 months ago (minus the NVMe drives which weren't available). We are using ZFSonLinux on top of DRBD instead of mdmadm, which has been surprisingly successful. The biggest frustration that we met was using LACP in trying to get a better data rate out of DRBD. Regardless of the LACP mode, we were not able to get DRBD to scale to full bandwidth (20gbps) and had to settle for using it for redundancy only. I imagine with your setup (read: much faster peak write capability), you might actually be able to bottleneck DRBD pretty well over a single 10GBE pipe if you were writing at peak capacity. I'll be interested to see if you happen upon a work-around. Load balancing as you've suggested is an intriguing compromise. Have you tried doing a two-way heartbeat setup where each both boxes have their own active IP and can fail over to each other?
  - tedchs 12y ago
    I wonder if the lack of help from LACP was because DRBD is using only 1 TCP session, which would only get hashed to one or the other member link?
    - cgb_ 12y ago
      Might be a case where MPTCP (http://www.multipath-tcp.org/ http://www.multipath-tcp.org/) would help with aggregating multiple links.
      - mrmondo 12y ago
        Correct me if I'm wrong but I'm not sure MPTCP would help as LACP is generally SRC/DST MAC based and shares a uses a single IP?
        cgb_ 12y ago
        You are right in that LACP bonding methods cannot increase the throughput of a single flow to gt any single link in the group (the balancing method can utilise src/dst MAC, IP and sometime L4 ports). MPTCP establishes multiple subflows across individual IP paths and can load balance or failover across all subflows. Applications do not need to be rewritten to take advantage of it. I'm not sure if that includes kernel modules like DRBD though. I suppose someone needs to find out :)

baruch 12y ago

I would strongly recommend also testing for when things fail. This is the hardest part in troubleshooting and building a reliable system. You can make do with READ/WRITE LONG if the disks are SAS to generate medium errors, be aware that it may kill your disk if you do it too much to the same disk. You can also use relatively inexpensive SAS/SATA link error injectors from Quarch. Be sure to test hot pulls of the disks to simulate a hard failure. Simulating other soft failures is much harder, sometimes the disk just stops responding but doesn't go offline so the system finds it harder to detect the case and may reach timeout conditions.

mrmondo 12y ago
I couldn't agree with you more, in the time that I have set aside to establish the new storage I have around 75% of it dedicated to testing, around half of that is dedicated to failure scenarios. A have a list of 'bad things to do' that I'll be executing and reporting on.