6 ms·

Why do you buy servers with metal flakes in it? No quality controll on your side?

by nix23 4y ago

jeffbee 4y ago
Are you saying that part of the expected savings from going on-prem is that you will have to disassemble equipment bought from major OEMs and examine it for microscopic metal dust? That doesn't sound like it will save much money, honestly.
- yobbo 4y ago
  Did they pass typical memory/reliability tests and so on?
  - nix23 4y ago
    Maybe in the first day's they survive it, but the flakes are 99% from the fans/bearings, that's why you test servers at max load for at least 1 week and HD's for 2-4 weeks. But i don't think they made even a initial load-/stresstest. Unpack it, trow it into the rack, no checking of internal plug's just nothing...pretty sure about that.
    - varjag 4y ago
      Metal chips is squarely in the long tail of failure modes that you can't really anticipate (but of course really easy to be smug about in hindsight). It is also extremely unlikely the bearings, most likely these are from chassis frames assy not cleaned up properly.
      - nix23 4y ago
        I had some metaldust and it was from bearings, but op said something flakes and then microscopic particles. Particles = bearings, flakes = chassis or even stickers, but anyway just because of transport you dont trow a server into production without testing and inspection. I am beeing smug about not testing your hardware as you do it with software....shitty testing is shitty testing, counts for software hardware firmware and everything between. Even for your diesel generator ;-)
        linker3000 4y ago
        I heard tale of a banking centre that had a diesel generator installed by a local company. Load and simulated power failure tests all passed. Then some time later there was a total power cut and that's when they realised the generator had an electric start wired to the mains supply.
        nix23 4y ago
        And there is also the true story when "someone" forgot to fill the tank after 5 years of regular monthly tests, then the real thing happened. > had an electric start wired to the mains supply. But that's a good one, humans being humans...but it worked every time before today ;))
- nix23 4y ago
  Are you saing you just buy some server unpack them and throw them into production.....oh man...the lost art of systemadmin, if your system is not stable (in testing) you for sure disassemble it, or send it back. How much money have you lost playing around with your unstable database? Was it more then test your servers for some weeks? Do you buy/build software and throw it into production without testing? You can test your stuff and be still profitable henzer aws etc would make no money otherwise....you know they test their server much more (sometimes weeks/month)
- jotterbrain 4y ago
  They’re saying it’s a surprise to hear that Dropbox doesn’t know what QC and order acceptance means. And it is, I agree. That you spent the time investigating it, implying those servers were in production, is a shibboleth to those of us that know what we’re doing when designing hardware usage that Dropbox doesn’t. It is, however, your self sourced report and we don’t have an idea of scale, so maybe they do and you’re just unlucky. And no, operators don’t disassemble to perform QC. And no, I could hire an entire division of people buying servers at Best Buy, and disassembling them, and stress testing them, and all of that overhead including the fuel to drive to the store would still clock in under cloud’s profit margin depending on what you’re doing. You’re of course entitled to develop your cloud opinion from that experience. That’s like finding a stain in a new car and swearing off internal combustion as a useful technology, though, without any awareness of how often new cars are defective.
  - jeffbee 4y ago
    Many hardware problems do not surface at burn-in. Even at Google, the infamous "Platform A" from the paper "DRAM Errors in the Wild" was in large-scale production before they realized it was garbage.
    - jotterbrain 4y ago
      Filings from the chassis stamper, which yours certainly were given the combination of circumstances and vendor, are present when the machine is installed. If you’re buying racks, your integrator inspects them. If you’re buying U, you do. It’s a five minute job to catch your thank-God-my-career-was-short story before the machine is even energized, which I know because I’ve caught the same thing from the same vendor twice. (It’s common; notice several comments point to it.) Why do you think QC benches have magnifiers and loupes? It’s a capital expenditure and an asset, so of course it’s rigorously inspected before the company accepts it, right? That’s not strange, is it? You can point at Google and speak in abstracts but it doesn’t address the point being made, nor that your rationale for your extreme position on cloud isn’t as firm as you thought it was. Is Dropbox the only time you’ve worked with hardware? I’m genuinely asking because manufacturing defects can top 5% of incoming kit depending on who you’re dealing with. Google knew that when they built Platform A. The lie of cloud is that dismissing those problems is worth the margin (it ain’t; you send it back, make them refire the omelette, and eat the toast you baked into your capacity plan while you wait).
mbesto 4y ago
Wait, are you saying that an org needs expertise to QC all of the the hardware they procure? How expensive is that? How easy it is to hire that type of QC? Do you see how these costs all start to add up?
- ClumsyPilot 4y ago
  Well, are you saying that an org needs expertise to inspect faulty cars, like, by calling a mechanic? Is that like too much these days for companies that owb fleets of cars? is opening a server harder than checking whars wrong with a car? like a cable comes loose and that's gane over?
  - mbesto 4y ago
    If I procure a fleet of cars I expect none of them to be faulty...how about you?
    - nix23 4y ago
      >I expect none of them to be faulty So you don't even test the car's, you just expect that the tire pressure is correct, tank is full? Expect that something "just" works is exactly why pilots have checklist's. Expectations are the main point for disappointments, you would never do that with software right?
      - jeffbee 4y ago
        The point, which you seem so dedicated to avoiding, is that "in the cloud" these steps are not my problem. Inspecting a literal shipload of computers for subtle defects is a pain in the ass. Amazon does it for me. When I get on an airplane I do not personally have to run the checklists. The airline does it for me.
        nix23 4y ago
        >The point, which you seem so dedicated to avoiding Not true the point was you pay for it (cloud), or you do it yourself (but then do it right, and not like a amateur who build's his first "gaming-pc"). And if you do it yourself you can still be very much competitive vs cloud.

Why do you buy servers with metal flakes in it? No quality controll on your side?

by nix23 4y ago

Why do you buy servers with metal flakes in it? No quality controll on your side?

Are you saying that part of the expected savings from going on-prem is that you will have to disassemble equipment bought from major OEMs and examine it for microscopic metal dust? That doesn't sound like it will save much money, honestly.

yobbo 4y ago
Did they pass typical memory/reliability tests and so on?
- nix23 4y ago
  Maybe in the first day's they survive it, but the flakes are 99% from the fans/bearings, that's why you test servers at max load for at least 1 week and HD's for 2-4 weeks. But i don't think they made even a initial load-/stresstest. Unpack it, trow it into the rack, no checking of internal plug's just nothing...pretty sure about that.
  - varjag 4y ago
    Metal chips is squarely in the long tail of failure modes that you can't really anticipate (but of course really easy to be smug about in hindsight). It is also extremely unlikely the bearings, most likely these are from chassis frames assy not cleaned up properly.
    - nix23 4y ago
      I had some metaldust and it was from bearings, but op said something flakes and then microscopic particles. Particles = bearings, flakes = chassis or even stickers, but anyway just because of transport you dont trow a server into production without testing and inspection. I am beeing smug about not testing your hardware as you do it with software....shitty testing is shitty testing, counts for software hardware firmware and everything between. Even for your diesel generator ;-)
      - linker3000 4y ago
        I heard tale of a banking centre that had a diesel generator installed by a local company. Load and simulated power failure tests all passed. Then some time later there was a total power cut and that's when they realised the generator had an electric start wired to the mains supply.
        nix23 4y ago
        And there is also the true story when "someone" forgot to fill the tank after 5 years of regular monthly tests, then the real thing happened. > had an electric start wired to the mains supply. But that's a good one, humans being humans...but it worked every time before today ;))
nix23 4y ago
Are you saing you just buy some server unpack them and throw them into production.....oh man...the lost art of systemadmin, if your system is not stable (in testing) you for sure disassemble it, or send it back. How much money have you lost playing around with your unstable database? Was it more then test your servers for some weeks? Do you buy/build software and throw it into production without testing? You can test your stuff and be still profitable henzer aws etc would make no money otherwise....you know they test their server much more (sometimes weeks/month)
jotterbrain 4y ago
They’re saying it’s a surprise to hear that Dropbox doesn’t know what QC and order acceptance means. And it is, I agree. That you spent the time investigating it, implying those servers were in production, is a shibboleth to those of us that know what we’re doing when designing hardware usage that Dropbox doesn’t. It is, however, your self sourced report and we don’t have an idea of scale, so maybe they do and you’re just unlucky. And no, operators don’t disassemble to perform QC. And no, I could hire an entire division of people buying servers at Best Buy, and disassembling them, and stress testing them, and all of that overhead including the fuel to drive to the store would still clock in under cloud’s profit margin depending on what you’re doing. You’re of course entitled to develop your cloud opinion from that experience. That’s like finding a stain in a new car and swearing off internal combustion as a useful technology, though, without any awareness of how often new cars are defective.
- jeffbee 4y ago
  Many hardware problems do not surface at burn-in. Even at Google, the infamous "Platform A" from the paper "DRAM Errors in the Wild" was in large-scale production before they realized it was garbage.
  - jotterbrain 4y ago
    Filings from the chassis stamper, which yours certainly were given the combination of circumstances and vendor, are present when the machine is installed. If you’re buying racks, your integrator inspects them. If you’re buying U, you do. It’s a five minute job to catch your thank-God-my-career-was-short story before the machine is even energized, which I know because I’ve caught the same thing from the same vendor twice. (It’s common; notice several comments point to it.) Why do you think QC benches have magnifiers and loupes? It’s a capital expenditure and an asset, so of course it’s rigorously inspected before the company accepts it, right? That’s not strange, is it? You can point at Google and speak in abstracts but it doesn’t address the point being made, nor that your rationale for your extreme position on cloud isn’t as firm as you thought it was. Is Dropbox the only time you’ve worked with hardware? I’m genuinely asking because manufacturing defects can top 5% of incoming kit depending on who you’re dealing with. Google knew that when they built Platform A. The lie of cloud is that dismissing those problems is worth the margin (it ain’t; you send it back, make them refire the omelette, and eat the toast you baked into your capacity plan while you wait).

mbesto 4y ago

Wait, are you saying that an org needs expertise to QC all of the the hardware they procure? How expensive is that? How easy it is to hire that type of QC? Do you see how these costs all start to add up?

ClumsyPilot 4y ago
Well, are you saying that an org needs expertise to inspect faulty cars, like, by calling a mechanic? Is that like too much these days for companies that owb fleets of cars? is opening a server harder than checking whars wrong with a car? like a cable comes loose and that's gane over?
- mbesto 4y ago
  If I procure a fleet of cars I expect none of them to be faulty...how about you?
  - nix23 4y ago
    >I expect none of them to be faulty So you don't even test the car's, you just expect that the tire pressure is correct, tank is full? Expect that something "just" works is exactly why pilots have checklist's. Expectations are the main point for disappointments, you would never do that with software right?
    - jeffbee 4y ago
      The point, which you seem so dedicated to avoiding, is that "in the cloud" these steps are not my problem. Inspecting a literal shipload of computers for subtle defects is a pain in the ass. Amazon does it for me. When I get on an airplane I do not personally have to run the checklists. The airline does it for me.
      - nix23 4y ago
        >The point, which you seem so dedicated to avoiding Not true the point was you pay for it (cloud), or you do it yourself (but then do it right, and not like a amateur who build's his first "gaming-pc"). And if you do it yourself you can still be very much competitive vs cloud.