3 ms·

Could you discuss how you selected the Mellanox cards? Were other cards such as Solarflare, Broadcom, Chelsio considered as well?

drewg123 9y ago
They were the first to market with 100GbE NICs by quite a while, and when we were starting, they were the only game in town
- bogomipz 9y ago
  Thanks.
- shaklee3 9y ago
  Not only first, but they have a dual 100Gbps, which is more than pcie currently supports. They're way ahead of everyone else.
jlgaddis 9y ago
Not OP, but I'd wager that one factor is that Mellanox has high-quality, well-maintained (by Mellanox themselves), open-source drivers for their 100 GbE NICs in the FreeBSD kernel. I'm not sure if the Chelsio cxgbe driver supports 100 GbE yet (I don't think it does) and I'm pretty certain that there aren't yet any drivers for any 100 GbE NICs from any of the others. I don't have a need for anything close to 100 GbE so I haven't really kept up on this so there could be by now but, even if there are, Mellanox was first.
- kev009 9y ago
  I'll probably get some scowls for this but Mellanox sales and support has rubbed me the wrong way numerous times. I'm trying to mend that at the moment for my company. Chelsio has a more technically interesting product to me so far, although they weren't as quick to market for 25/100g. Their T6 line does the same line speeds as ConnectX4/5. Chelsio prices are haggle free and puzzling low, which I think vendors like Broadcom, Intel and Mellanox put a ton of wiggle room in their NIC pricing to give your procurement person a nice softball win on how much they "saved" on MSRP. Meh, I'd rather not play games.
  - bogomipz 9y ago
    Nah, I've heard similar about both Mellanox and Solarflare sales people. Are you running the Chesio cards in production then? Any feedback there? Can you recommend them?
    - kev009 9y ago
      I have many thousand in revenue service, as does Netflix (https://openconnect.netflix.com/en/hardware/ https://openconnect.netflix.com/en/hardware/). It's the only NIC we buy for 2+ years at $LLNW. T6 requires a little bit of care on Linux, the LTS kernels don't have all the bells and whistles but should pass packets due to common mailbox API. I think kernel 4.13 is fairly complete. It's plug and play on supported FreeBSD versions (10.4, 11.1, -STABLE, HEAD). We run fbsd HEAD for main product line and some Ubuntu 16.04 for other product lines where I packaged up the out of tree driver for kernel 4.4.
drewg123 9y ago
I should also mention that in addition to being the first to market with 100GbE NICS, Mellanox has been just phenomenal to work with. The RSS assisted LRO mentioned in the blog is just one example of the help they've given us. They have also driven the adoption of hardware packet pacing into FreeBSD, as well as NIC queue backpressure, and a few other things we're currently working on. They've been a huge help to FreeBSD networking.
- bogomipz 9y ago
  >" They have also driven the adoption of hardware packet pacing into FreeBSD, as well as NIC queue backpressure," Could you elaborate these? What is "packet pacing" I am not familiar with this term. Also how does the NIC queue backpressure work? I don't see references to these in the post. Thanks.

drewg123 9y ago

They were the first to market with 100GbE NICs by quite a while, and when we were starting, they were the only game in town

bogomipz 9y ago
Thanks.
shaklee3 9y ago
Not only first, but they have a dual 100Gbps, which is more than pcie currently supports. They're way ahead of everyone else.

jlgaddis 9y ago

Not OP, but I'd wager that one factor is that Mellanox has high-quality, well-maintained (by Mellanox themselves), open-source drivers for their 100 GbE NICs in the FreeBSD kernel. I'm not sure if the Chelsio cxgbe driver supports 100 GbE yet (I don't think it does) and I'm pretty certain that there aren't yet any drivers for any 100 GbE NICs from any of the others. I don't have a need for anything close to 100 GbE so I haven't really kept up on this so there could be by now but, even if there are, Mellanox was first.

kev009 9y ago
I'll probably get some scowls for this but Mellanox sales and support has rubbed me the wrong way numerous times. I'm trying to mend that at the moment for my company. Chelsio has a more technically interesting product to me so far, although they weren't as quick to market for 25/100g. Their T6 line does the same line speeds as ConnectX4/5. Chelsio prices are haggle free and puzzling low, which I think vendors like Broadcom, Intel and Mellanox put a ton of wiggle room in their NIC pricing to give your procurement person a nice softball win on how much they "saved" on MSRP. Meh, I'd rather not play games.
- bogomipz 9y ago
  Nah, I've heard similar about both Mellanox and Solarflare sales people. Are you running the Chesio cards in production then? Any feedback there? Can you recommend them?
  - kev009 9y ago
    I have many thousand in revenue service, as does Netflix (https://openconnect.netflix.com/en/hardware/ https://openconnect.netflix.com/en/hardware/). It's the only NIC we buy for 2+ years at $LLNW. T6 requires a little bit of care on Linux, the LTS kernels don't have all the bells and whistles but should pass packets due to common mailbox API. I think kernel 4.13 is fairly complete. It's plug and play on supported FreeBSD versions (10.4, 11.1, -STABLE, HEAD). We run fbsd HEAD for main product line and some Ubuntu 16.04 for other product lines where I packaged up the out of tree driver for kernel 4.4.

drewg123 9y ago

I should also mention that in addition to being the first to market with 100GbE NICS, Mellanox has been just phenomenal to work with. The RSS assisted LRO mentioned in the blog is just one example of the help they've given us. They have also driven the adoption of hardware packet pacing into FreeBSD, as well as NIC queue backpressure, and a few other things we're currently working on. They've been a huge help to FreeBSD networking.

bogomipz 9y ago
>" They have also driven the adoption of hardware packet pacing into FreeBSD, as well as NIC queue backpressure," Could you elaborate these? What is "packet pacing" I am not familiar with this term. Also how does the NIC queue backpressure work? I don't see references to these in the post. Thanks.