4 ms·

I priced out the hardware that they specced out to be around $1.4 million: https://gitlab.com/gitlab-com/infrastructure/issues/727#note_18193242 https://gitlab

by diziet 10y ago

I priced out the hardware that they specced out to be around $1.4 million: https://gitlab.com/gitlab-com/infrastructure/issues/727#note_18193242 https://gitlab.com/gitlab-com/infrastructure/issues/727#note... The R730xd are probably around $50k after dell discounts - depends a bit on what they ended up configuring with regards to support, exact network configuration, etc The R830s are about $50k as configured - 1.5gb ram is expensive as the R830 only has 48 DIMMs and they need the relatively expensive 32GB RDIMMs The R630s should be about 15k each: The switches say they are 48x 40G QSFP+ which are very expensive (I'd put them at 30k each from Dell) 50k * 20 + 50k * 4 + 15k * 10k + 2 * 30k 1m + 200k + 150k + 60k ~= $1.4m invested updated with better R830s pricing From my perspective consumer Samsung 850 EVO drives which can be under-provisioned to match the 1.8tb (and get better performance characteristics) would give Gitlab cheaper and more reliable storage in terms of IOPS/Latency when compared to 10k 1.8tb drives.

connorshea 10y ago
Unfortunately I don't believe we'll be able to comment on the accuracy of this since the quotes from companies are private information, but I do love math like this :D (Community Advocate at GitLab)
redstripe 10y ago
What kind of dell discounts does dell typically give larger customers? We get 0 (from HP) at my workplace and I've always wondered. I'm sure it depends on how much you buy. Does it vary by components? They seem to charge a lot for drives so I'm guessing those can be heavily discounted.
- icelancer 10y ago
  I helped design a DC for a big data mining / science company and we got just under 20% off the higher margin packages. I felt very good about the number and Dell was great to work with on the purchase side.
- late2part 10y ago
  Please stop and read this and show it to your boss. If you are really getting 0% discount you need to private message me and send me $10k so I can tell you this one weird trick. You simply ask for them. Just do that. Everyone gets discounts.
  - diziet 10y ago
    Well, it depends on how much stuff you're buying and from whom. If you're buying $800 dollar workstations you won't get as many discounts as when you're buying 400+GB quad CPU 2U servers
    - late2part 10y ago
      Agreed. But even a single purchase is likely to be below list price.
- eonw 10y ago
  the last time i was in a position that i oversaw hardware purchasing... you get better deals based on your overall spend. after we had spend 250k we got a new sales rep that had access to offer us lower pricing, same thing happened once we breached the $1m mark. new sales rep and pricing went lower. In fact, our first sales rep couldn't even make up a reasonable deal at $80k because he didn't have access to that level of discounting. usually i always send everything back and tell them to do better. my biggest gripe about dell pricing was no line items, just a bottomline price. so i couldnt tell which things i could save money on by sourcing them elsewhere(ie: larger drives, large amounts of memory)
patrickg_zill 10y ago
Subtract about 30% , maybe more. Dell pricing can be aggressive, if you are buying enough.
- diziet 10y ago
  Indeed, there would be discounts from Dell, but I also expect companies to buy some extra add-ons and subscription/support services that cancel them out that I didn't see in the high level specs.
- Maarten88 10y ago
  Pro-tip: always buy around the end of their sales quarter. Have internal approval ready. Just promise you'll sign immediately if they lower the price to XXX. Around the close of the quarter their sales guys are pushed to get deals done, and they are sometimes willing to give even bigger discounts. I think at Dell the quarters are one month off (end of januari, april etc)
Rapzid 10y ago
I would take a look at Supermicro servers and compare pricing. Worked as cloud engineer at a hosting provider built entirely on Supermicro kit. Didn't see anything there I didn't see from the Dell's and HP's when I worked at Rackspace. And Rackspace sure didn't build their cloud on top of HP's or Dell's.
- nixgeek 10y ago
  Fetch a quote from Supermicro and tell your Dell or HP folks that you'll buy that if they don't match the price. If you're large enough, then you should be talking to other folks including Wiwynn, ZT, Quanta, Celestica, et al. They won't always get all the way there, depends on volume, but often they'll get pretty close, and it'll probably be a 25%+ additional discount vs. what they'll offer without you twisting their arm using a cheaper manufacturer as leverage. They have to believe you're serious, obviously, which can mean visibly throwing a couple small orders to another manufacturer to tell your sales team they're not the only game in town. In quality terms, Supermicro is worse vs. Dell/HP, which can still be fine if you've got enough scale to work through all the little issues with firmware.
  - Rapzid 10y ago
    We never had issues and I never noticed a quality issue over Supermicro vs Dell's and HP's at Rackspace. There were firmware niggles and updates with both; we were always upgrading server, storage card, and storage controller firmware at the Rack. I'm not privy to studies at scales large enough to tell which is higher quality. I'm not beholden to either though haha. If you can get Dell to come close to Supermicro prices then that might make sense. When I was last in the infrastructure game a few years ago we also ran into configuration limitations with Dell. At the time we could get more storage and RAM in the form factors we needed from Supermicro.
    - nixgeek 10y ago
      This is fair and buying at end of quarter or end of year is often the little extra kick needed to get Dell/HP to compete. One of my main beefs with Supermicro is their tooling around remote patching and configuration of BIOS and other firmware, they charge for it, on top of warranty, and the UX is awful. With both Dell and HP if you're under a support contract all the updates and tools to apply them are included, and whilst both manufacturers have some "sharp edges" to their tools for managing large fleets, neither is "stab yourself in the eyeballs with a blunt fork" bad like Supermicro.
yuhong 10y ago
32GB DDR4 RDIMMs are not that expensive anymore. 8Gbit DDR4 is close to crossover now.
- diziet 10y ago
  Yeah, you're actually right. I relied on my memory of pricing them out a bit back, and it looks like the prices are much closer to other DIMMs now.
deleted 10y ago
[deleted]
sijoe 10y ago
Without knowing more on the specific needs (I followed some of the threads to try to grok it), it would be hard to guess what they really need. [commercial alert] My company (Scalable Informatics) literally builds very high performance Ceph (and other) appliances, specifically for people with huge data flow/load/performance needs. Relevant links via shortener: Main site: http://scalableinformatics.com http://scalableinformatics.com (everything below is at that site under the FastPath->Unison tab) http://bit.ly/1vp3hGd http://bit.ly/1vp3hGd Ceph appliance: http://bit.ly/1qiOYpy http://bit.ly/1qiOYpy Especially relevant given the numbers I saw on the benchmarking ... Ceph appliance benchmark whitepaper: http://bit.ly/2fMahfJ http://bit.ly/2fMahfJ Our EC test was about 2x better than the Dell unit (and the Supermicro unit), and our Librados tests were even more significantly ahead. Petabyte scale appliances: http://bit.ly/2fuTTAH http://bit.ly/2fuTTAH We've even got some very nice SSD and NVM units, the latter starting around $1USD/GB. [end commercial alert] I noticed the 10k RPM drives ... really, drop them and go with SSDs if possible. You won't regret it. Someone suggested underprovisioned 850 EVO. Our strong recommendation is against this, based upon our experience with Ceph, distributed storage, and consumer SSDs. You will be sorry if you go that route, as you will lose journals/MDS or whatever you put on there. Additionally, I saw a thread about using RAIDs underneath. Just ... don't. Ceph doesn't like this ... or better, won't be able to make as effective use of it. Use the raw devices. Depending upon the IOP needs (multi-tenant/massive client systems usually devolve into a DDoS against your storage platforms anyway), we'd probably recommend a number of specific SSD variations at various levels. The systems we build are generally for people doing large scale genomics and financial processing (think thousands of cores hitting storage over 40-100Gb networks, where latency matters, and sustained performance needs to always be high). We do this with disk, flash, and NVMe. I am at landman _at_ the company name above with no spaces, and a dot com at that end .
sytse 10y ago
Thanks for looking into this. I do believe that the linked comment configuration was over the top, we'll have less memory and 7.2k rpm drives. We're working on the configuration in https://docs.google.com/spreadsheets/d/1XG9VXdDxNd8ipgPlEr7Nb7Eg22twXPuzgDwsOhtdYKQ/edit#gid=405936431 https://docs.google.com/spreadsheets/d/1XG9VXdDxNd8ipgPlEr7N... feel free to leave comments there.

by diziet 10y ago

connorshea 10y ago

Unfortunately I don't believe we'll be able to comment on the accuracy of this since the quotes from companies are private information, but I do love math like this :D (Community Advocate at GitLab)

redstripe 10y ago

What kind of dell discounts does dell typically give larger customers? We get 0 (from HP) at my workplace and I've always wondered. I'm sure it depends on how much you buy. Does it vary by components? They seem to charge a lot for drives so I'm guessing those can be heavily discounted.

icelancer 10y ago
I helped design a DC for a big data mining / science company and we got just under 20% off the higher margin packages. I felt very good about the number and Dell was great to work with on the purchase side.
late2part 10y ago
Please stop and read this and show it to your boss. If you are really getting 0% discount you need to private message me and send me $10k so I can tell you this one weird trick. You simply ask for them. Just do that. Everyone gets discounts.
- diziet 10y ago
  Well, it depends on how much stuff you're buying and from whom. If you're buying $800 dollar workstations you won't get as many discounts as when you're buying 400+GB quad CPU 2U servers
  - late2part 10y ago
    Agreed. But even a single purchase is likely to be below list price.
eonw 10y ago
the last time i was in a position that i oversaw hardware purchasing... you get better deals based on your overall spend. after we had spend 250k we got a new sales rep that had access to offer us lower pricing, same thing happened once we breached the $1m mark. new sales rep and pricing went lower. In fact, our first sales rep couldn't even make up a reasonable deal at $80k because he didn't have access to that level of discounting. usually i always send everything back and tell them to do better. my biggest gripe about dell pricing was no line items, just a bottomline price. so i couldnt tell which things i could save money on by sourcing them elsewhere(ie: larger drives, large amounts of memory)

patrickg_zill 10y ago

Subtract about 30% , maybe more. Dell pricing can be aggressive, if you are buying enough.

diziet 10y ago
Indeed, there would be discounts from Dell, but I also expect companies to buy some extra add-ons and subscription/support services that cancel them out that I didn't see in the high level specs.
Maarten88 10y ago
Pro-tip: always buy around the end of their sales quarter. Have internal approval ready. Just promise you'll sign immediately if they lower the price to XXX. Around the close of the quarter their sales guys are pushed to get deals done, and they are sometimes willing to give even bigger discounts. I think at Dell the quarters are one month off (end of januari, april etc)

Rapzid 10y ago

I would take a look at Supermicro servers and compare pricing. Worked as cloud engineer at a hosting provider built entirely on Supermicro kit. Didn't see anything there I didn't see from the Dell's and HP's when I worked at Rackspace. And Rackspace sure didn't build their cloud on top of HP's or Dell's.

nixgeek 10y ago
Fetch a quote from Supermicro and tell your Dell or HP folks that you'll buy that if they don't match the price. If you're large enough, then you should be talking to other folks including Wiwynn, ZT, Quanta, Celestica, et al. They won't always get all the way there, depends on volume, but often they'll get pretty close, and it'll probably be a 25%+ additional discount vs. what they'll offer without you twisting their arm using a cheaper manufacturer as leverage. They have to believe you're serious, obviously, which can mean visibly throwing a couple small orders to another manufacturer to tell your sales team they're not the only game in town. In quality terms, Supermicro is worse vs. Dell/HP, which can still be fine if you've got enough scale to work through all the little issues with firmware.
- Rapzid 10y ago
  We never had issues and I never noticed a quality issue over Supermicro vs Dell's and HP's at Rackspace. There were firmware niggles and updates with both; we were always upgrading server, storage card, and storage controller firmware at the Rack. I'm not privy to studies at scales large enough to tell which is higher quality. I'm not beholden to either though haha. If you can get Dell to come close to Supermicro prices then that might make sense. When I was last in the infrastructure game a few years ago we also ran into configuration limitations with Dell. At the time we could get more storage and RAM in the form factors we needed from Supermicro.
  - nixgeek 10y ago
    This is fair and buying at end of quarter or end of year is often the little extra kick needed to get Dell/HP to compete. One of my main beefs with Supermicro is their tooling around remote patching and configuration of BIOS and other firmware, they charge for it, on top of warranty, and the UX is awful. With both Dell and HP if you're under a support contract all the updates and tools to apply them are included, and whilst both manufacturers have some "sharp edges" to their tools for managing large fleets, neither is "stab yourself in the eyeballs with a blunt fork" bad like Supermicro.

yuhong 10y ago

32GB DDR4 RDIMMs are not that expensive anymore. 8Gbit DDR4 is close to crossover now.

diziet 10y ago
Yeah, you're actually right. I relied on my memory of pricing them out a bit back, and it looks like the prices are much closer to other DIMMs now.

deleted 10y ago

[deleted]

sijoe 10y ago

Without knowing more on the specific needs (I followed some of the threads to try to grok it), it would be hard to guess what they really need. [commercial alert] My company (Scalable Informatics) literally builds very high performance Ceph (and other) appliances, specifically for people with huge data flow/load/performance needs. Relevant links via shortener: Main site: http://scalableinformatics.com http://scalableinformatics.com (everything below is at that site under the FastPath->Unison tab) http://bit.ly/1vp3hGd http://bit.ly/1vp3hGd Ceph appliance: http://bit.ly/1qiOYpy http://bit.ly/1qiOYpy Especially relevant given the numbers I saw on the benchmarking ... Ceph appliance benchmark whitepaper: http://bit.ly/2fMahfJ http://bit.ly/2fMahfJ Our EC test was about 2x better than the Dell unit (and the Supermicro unit), and our Librados tests were even more significantly ahead. Petabyte scale appliances: http://bit.ly/2fuTTAH http://bit.ly/2fuTTAH We've even got some very nice SSD and NVM units, the latter starting around $1USD/GB. [end commercial alert] I noticed the 10k RPM drives ... really, drop them and go with SSDs if possible. You won't regret it. Someone suggested underprovisioned 850 EVO. Our strong recommendation is against this, based upon our experience with Ceph, distributed storage, and consumer SSDs. You will be sorry if you go that route, as you will lose journals/MDS or whatever you put on there. Additionally, I saw a thread about using RAIDs underneath. Just ... don't. Ceph doesn't like this ... or better, won't be able to make as effective use of it. Use the raw devices. Depending upon the IOP needs (multi-tenant/massive client systems usually devolve into a DDoS against your storage platforms anyway), we'd probably recommend a number of specific SSD variations at various levels. The systems we build are generally for people doing large scale genomics and financial processing (think thousands of cores hitting storage over 40-100Gb networks, where latency matters, and sustained performance needs to always be high). We do this with disk, flash, and NVMe. I am at landman _at_ the company name above with no spaces, and a dot com at that end .

sytse 10y ago

Thanks for looking into this. I do believe that the linked comment configuration was over the top, we'll have less memory and 7.2k rpm drives. We're working on the configuration in https://docs.google.com/spreadsheets/d/1XG9VXdDxNd8ipgPlEr7Nb7Eg22twXPuzgDwsOhtdYKQ/edit#gid=405936431 https://docs.google.com/spreadsheets/d/1XG9VXdDxNd8ipgPlEr7N... feel free to leave comments there.