4 ms·

Weights are just numbers (probably floats?), right? This model has 3.12TB of floats??? That's insane. How do you load that into memory for inferencing?

exikyut 6y ago
Use x1e.32xlarge on AWS with 3TB of RAM. Just $12,742/mo - https://calculator.aws/#/estimate?id=7428fa81192c57087ac8cdf88b461fccebce3e7e https://calculator.aws/#/estimate?id=7428fa81192c57087ac8cdf... Alternatively order something like the HP Z8 with 3TB RAM configured, which is only $75k - https://zworkstations.com/configurations/2040422/ https://zworkstations.com/configurations/2040422/ It's interesting. It would take ~six years for the Z8 to break even compared to AWS, but traffic into and out of the machine would be $0, and I don't think you're running directly on the metal with AWS, so performance would probably be a bit higher. And then there's storage - I configured, uhh, 120TB of a mixture of SSDs and HDDs. I'm not even going to try and ask AWS for a comparible quote there. I may or may not have added dual Xeon Platinum 8280s to the Z8 as well. :P
- jsnell 6y ago
  > It would take ~six years for the Z8 to break even Do you mean six months?
  - exikyut 6y ago
    Oh *dear*. I definitely tripped over there, and I didn't even notice. Yup.
- visarga 6y ago
  hate to break the party but this model only loads a small part of itself in RAM when inferencing
  - exikyut 6y ago
    That's a good thing. Less completely means more energy for interestingness, and less expense means more accessibility.
- tjbiddle 6y ago
  When you're spending that kind of money on a machine, there's no way you're paying retail price. Sales reps would give you a significant discount. Also - think you meant 6 months, not 6 years anyhow :)
  - exikyut 6y ago
    Interesting. I'm very curious... 20%? 35%? And I did mean 6 months, woops. Didn't even notice...
- high_byte 6y ago
  Z8 sounds like fun. but I might just buy two teslas (roadster and X, or a cybertruck) and a gaming PC. :D

exikyut 6y ago

Use x1e.32xlarge on AWS with 3TB of RAM. Just $12,742/mo - https://calculator.aws/#/estimate?id=7428fa81192c57087ac8cdf88b461fccebce3e7e https://calculator.aws/#/estimate?id=7428fa81192c57087ac8cdf... Alternatively order something like the HP Z8 with 3TB RAM configured, which is only $75k - https://zworkstations.com/configurations/2040422/ https://zworkstations.com/configurations/2040422/ It's interesting. It would take ~six years for the Z8 to break even compared to AWS, but traffic into and out of the machine would be $0, and I don't think you're running directly on the metal with AWS, so performance would probably be a bit higher. And then there's storage - I configured, uhh, 120TB of a mixture of SSDs and HDDs. I'm not even going to try and ask AWS for a comparible quote there. I may or may not have added dual Xeon Platinum 8280s to the Z8 as well. :P

jsnell 6y ago
> It would take ~six years for the Z8 to break even Do you mean six months?
- exikyut 6y ago
  Oh *dear*. I definitely tripped over there, and I didn't even notice. Yup.
visarga 6y ago
hate to break the party but this model only loads a small part of itself in RAM when inferencing
- exikyut 6y ago
  That's a good thing. Less completely means more energy for interestingness, and less expense means more accessibility.
tjbiddle 6y ago
When you're spending that kind of money on a machine, there's no way you're paying retail price. Sales reps would give you a significant discount. Also - think you meant 6 months, not 6 years anyhow :)
- exikyut 6y ago
  Interesting. I'm very curious... 20%? 35%? And I did mean 6 months, woops. Didn't even notice...
high_byte 6y ago
Z8 sounds like fun. but I might just buy two teslas (roadster and X, or a cybertruck) and a gaming PC. :D