8 ms·

"Huge compute bills" usually come from training, or to be more precise, hyperparameter search that's required before you find a model that works well. You could

by m0zg 7y ago

"Huge compute bills" usually come from training, or to be more precise, hyperparameter search that's required before you find a model that works well. You could also fail to find such a model, but that's another discussion. So yeah, you could spend one or two FTE salaries' (or one deep learning PhD's) worth of cash on finding such models for your startup if you insist on helping Jeff Bezos to wipe his tears with crisp hundred dollar bills. That's if you know what you're doing of course. Literally unlimited amounts could be spent if you don't. Or you could do the same for a fraction of the cost by stuffing a rack in your office with consumer grade 2080ti's. Just don't call it a "datacenter" or NVIDIA will have a stroke. Is that too much money? Not in most typical cases, I'd think. If the competitive advantage of what you're doing with DL does not offset the cost of 2 meatspace FTEs, you're doing it wrong. That, once again, assumes that you know what you're doing, and aren't doing deep learning for the sake of deep learning. Also, if your startup is venture funded, AWS will give you $100K in credit, hoping that you waste it by misconfiguring your instances and not paying attention to their extremely opaque billing (which is what most of their startup customers proceed to doing pretty much straight away). If you do not make these mistakes, that $100K will last for some time, after which you could build out the aforementioned rack full of 2080ti's on prem.

avip 7y ago
For balance, all big cloud providers - aws, gcp, azure, oracle [0] have pretty similar startup plans. Y$$MV (I'm in full agreement with everything you've written + it's well-phrased and funny. gj!) [0] that's not a typo - there is such thing as "Oracle cloud"
alephnan 7y ago
> Just don't call it a "datacenter" or NVIDIA will have a stroke. Context please :) ?
- mereel 7y ago
  Just a guess but maybe it's some licensing issue? https://www.nvidia.com/en-us/drivers/geforce-license/ https://www.nvidia.com/en-us/drivers/geforce-license/ No Datacenter Deployment. The SOFTWARE is not licensed for datacenter deployment, except that blockchain processing in a datacenter is permitted.
  - mam2 7y ago
    well they are the one writing the rules, so i'd side with OP
  - derefr 7y ago
    > except that blockchain processing in a datacenter is permitted Well, Nvidia, y'see, my new blockchain does AI training as its Proof-of-Work step...
- ThePadawan 7y ago
  Datacenter GPUs are mostly identical to the much cheaper consumer versions. The only thing preventing you from running a datacenter with consumer hardware is the licensing agreement you accept.
  - KaiserPro 7y ago
    And the cooling, amount of ram and the doubles performance. the chip might be the same, but the rest of it isn't Granted, its not worth the $3k price bump, but thats a different issue.
    - zwaps 7y ago
      Nah that's not really it. The reason NVIDIA doesn't allow this is precisely because the additional RAM - functionally the only difference - is not cost efficient. People would like (and did) use a bunch of consumer 1080s, which is why NVIDIA disallowed precisely that. You had to buy the equivalent pro grade card, which costs easily two or three times that and offers a couple more GB of RAM.
  - endorphone 7y ago
    "The only thing preventing you from running a datacenter with consumer hardware is the licensing agreement you accept." The consumer cards don't use ECC and memory errors are a common issue (GDDR6 running at the edge of its capabilities). In a gaming situation that means a polygon might be wrong, a visual glitch occurs, a texture isn't rendered right -- things that just don't matter. For scientific purposes that same glitch could be catastrophic. The "datacenter" cards offer significantly higher performance for some case (tensor cores, double precision), are designed for data center use, are much more scalable, etc. They also come with over double the memory (which is one of the primary limitations forcing scale outs). Going with the consumer cards is one of those things that might be Pyrrhic. If funds are super low and you want to just get started, sure, but any implication that the only difference is a license is simply incorrect.
    - luisfmh 7y ago
      Learned a new word today. Pyrrhic.
- OkGoDoIt 7y ago
  NVIDIA forces you to buy significantly more expensive cards that perform marginally better if you are using them for datacenter use. They try to enforce not letting businesses use consumer grade gaming cards. I assume this is so cloud providers don't buy up all the supply of graphics cards and make it hard for gamers to get decent cards, like what happened during the bitcoin craze.
  - nkassis 7y ago
    No it's just pure price discrimination. They don't care about gamers they just know businesses will pay more if forced to while gamers can't.
    - aj7 7y ago
      Exactly.
    - gbba 7y ago
      I wouldn't say they don't care about gamers, considering that gaming makes up about half of their revenue: https://www.anandtech.com/show/15513/nvidia-releases-q4-fy2020-earnings-a-strong-quarter-closes-our-a-weaker-year https://www.anandtech.com/show/15513/nvidia-releases-q4-fy20...
      - nkassis 7y ago
        Sure sorry I realize now the point I was trying to make doesn't match my wording. They do care about selling to gamers but availability to gamers is not in anyway why they are forcing more expensive models of essentially the same cards on the HPC and server market. It's all because they can and the business market is able to bear the cost. Now if they found a true competitor in that market, I think this pricing model would fall apart fast.
- deleted 7y ago
  [deleted]
- deleted 7y ago
  [deleted]
fizixer 7y ago
- Or AMD could change their policy of 'never miss an opportunity to miss an opportunity' and offer high-performance OpenCL GPGPU offerings. Then nVidia could have all the stroke they wanted. - Or Tensorflow/Pytorch could've crapped on OpenCL a little less by releasing a fully functional OpenCL version everytime they released a fully functional Cuda version, instead of worshipping Cuda year in and year out. - Or Google could start selling their TPUv2, if not TPUv3, while they're on the verge of releasing TPUv4. - Or one of the other big-tech's Facebook/Microsoft/Intel could make and start selling a TPU-equivalent device. - Or I could finish school and get funded to do all/most of the above ;) edit: On a more serious note, a cloud/on-prem hybrid is absolutely the right way to go. You should have a 4x 2080 ti rig available 24x7 for every ML engineer. It costs about $6k-8k a piece [0]. Prototype the hell out of your models on on-prem hardware. Then when your setup is in working condition and starts producing good results on small problems, you're ready to do a big computation for final model training. Then you send it to the cloud, for final production run. (Guess what, on a majority of your projects, you might realize, the final production run could be carried out on on-prem itself; you just have to keep it running 24 hours-a-day for a few days or up to a couple weeks.) [0]: https://l7.curtisnorthcutt.com/the-best-4-gpu-deep-learning-rig https://l7.curtisnorthcutt.com/the-best-4-gpu-deep-learning-...
- m0zg 7y ago
  As someone who has actually worked on this stuff soup to nuts, it's not as easy as people imagine, because you can't just support some subset of available ops and call it a day. If you want to make OpenCL pie from scratch, you must first make the universe, and support every single stupid thing (among thousands) and even mimic some of the bugs so that models work "the same". This is hard and time consuming, and this field is hard enough as it is. What makes it even harder is that only NVIDIA has decent, mature tooling. There is some work on ROCM though, so AMD is not _totally_ dead in the water. I'd say they're about 90% dead in the water.
  - simonebrunozzi 7y ago
    Are you in the Bay area? Would love to chat. Thinking of an idea where your expertise could be very handy. $my-hn-username at gmail.
bob1029 7y ago
I find it fun how the cost of the cloud is forcing people to consider what absolutely must run in the cloud (presumably for stability and compliance reasons) and what can be brought back on-prem. We don't train ML models, but we are in a similar boat regarding cloud compute costs. Building our solutions for our clients is a compute-heavy task which is getting expensive in the cloud. We are considering options such as building commodity threadripper rigs, throwing them in various developers' (home) offices, installing a VPN client on each and then attaching as build agents to our AWS-hosted jenkins instance. In this configuration we could drop down to a t3a.micro for Jenkins and still see much faster builds. The reduction in iteration time over a month would easily pay for the new hardware. An obvious next step up from this is to do proper colocation, but I am of a mindset that if I have to start racking servers I am bringing 100% of our infrastructure out of the cloud.
- m0zg 7y ago
  This is not a new phenomenon. As early as in 2009 I worked for a company (ads, but not Google) which outgrew the typical "cloud" cost structure at the time, and moved everything to a more traditional datacenter, and saved substantial money even considering 3 more SREs they had to hire to absorb the increased support needs. AWS charges what the market will bear, and as such it was never designed to make sense for everyone. One needs to re-evaluate on the back of the napkin from time to time.
- buckminster 7y ago
  I once had a borrowed Sun blade server in my home office. The fan in it sounded like an industrial vacuum cleaner. It got moved to a different room and was powered on as little as possible. Your plan makes sense but be mindful of the acoustics or your devs may grow to hate you.
  - m0zg 7y ago
    BTW, the only reason why consumer vacuum cleaners are so loud is because consumers associate loudness with suction power. "Backpack" style commercial vacuum cleaners have more suction, and are barely audible in comparison.
ignoramous 7y ago
As someone hoping to build a world-wide footprint, say 25 to 50 DCs, of servers to deploy to with unmetered bandwidth, what are some alternatives to the usual suspects? I have come across fly.io, Vultr, Scaleway, Stackpath, Hetzner, and OVH but either they are expensive (in that they charge for bandwidth and uptime) or do not have a wide enough foot-print. I guess colos are the way to go, but how does one work with colos, allocate servers, deploy to them, ensure security and uptime and so on from a single place, 'cause dealing with them individually might slow down the process? Is there a tooling that deals with multi-colos like the ones for multi-cloud like min.io, k8s, Triton etc;
- mrkurt 7y ago
  (Hi, I'm from fly.io) It depends what you need in your datacenters! If you just want servers, and don't care about doing something like anycast, you can find a bunch of local dedicated server providers in a bunch of cities and go to town. But you can't get them all from one provider, really, not with any kind of reasonable budget. You _could_ buy colo from a place like Equinix in a bunch of cities, and then either use their transit or buy from other transit providers. But also, unmetered bandwidth isn't a very sustainable service, so I'm curious what you're after? You're usually either going to have to pay for usage, or pay large monthly fixed prices to get reasonable transit connections in each datacenter. In our case, we're constrained by Anycast. To expand past the 17 usual cities you end up needing to do your own network engineering which we'd rather not do yet.
  - ignoramous 7y ago
    (thanks mrkurt) It is anycast that I'm going after. Requirement for unmetered bandwidth (or cheaper than AWS et al) is because of the kind of workloads (TURN relays, proxy, tunnels etc) we'd deal with gets expensive, otherwise. For another related workload, per-request pricing gets expensive, again, due to the nature of the workload (to the tune 100k requests per user per month). So far, for the former (TURN relays etc), I've found using AWS Global Accelerator and/or GCP's GLB to be the easiest way to do anycast but the bandwidth is slightly on the expensive side. Fly.io matches the pricing in terms of network bandwidth (as promised on the website), so that's a positive but GCP/AWS have a wider footprint. Cloudflare's Magic Transit is another potential solution, but requires an enterprise plan and one needs to bring-your-own-anycast-IP and origin-servers. For the latter (latency-sensitive workload with ~100k+ reqs / month), Cloudflare Workers (200+ locations minus China) are a great fit though would get expensive once we hit a certain scale. Plus, they're limited to L7 HTTP reqs, only. Whilst, I believe, fly.io can do L4.
pridkett 7y ago
There’s also the issue that data scientists often want to go running to hyperparameter optimization and neural architecture search. In most cases improving your data pipelines and ensuring the data are clean and efficient will pay off much more quickly.
- fxtentacle 7y ago
  But manually improving the data pipeline requires an understanding of the problem, whereas doing a hyperparameter optimized architecture search just needs $$$ hardware and no clue on the side of the operator.
  - derefr 7y ago
    Or, to put that another way: if you knew what algorithm the AI would be using to discriminate the signal from the noise in your data, why would you need the AI? Just write that algorithm.
    - fxtentacle 7y ago
      Exactly :) In most cases, unsupervised learning is nothing more than having the AI try to approximate the solution of your highly non-linear loss function. So if there's any way of solving that loss function directly, it will perform like a well-trained AI.
      - mxcrossb 7y ago
        > In most cases In what cases is this not true?
    - nl 7y ago
      That's not the same thing at all. You can't build hand-build a feature detector as accurate as (say) a ResNet50 by hand. Before 2014 people tried to do this with feature detectors like SIFT and HOG. These were patented and made the inventors significant money. If it was still possible to do it then someone would be and making profit from it. Hyperparameter search is just optimising the training parameters (things like batch size, or optimiser parameters). This might give another 1% lift on accuracy, but isn't generally a significant factor.
fxtentacle 7y ago
No, also inference is quite expensive. You'll have 100% usage on a $10,000 GPU for 3s per customer image for a decently sized optical flow network. That's 3 hours of compute time for 1 minute of 60fps video. Now let's say your customer wants to analyze 2 hours = 120 minutes of video and doesn't want to wait more than those 3 hours, then suddenly you need 120 servers with one $10k GPU each to service this one customer within 3 hours of waiting. Good luck reaching that $1,200,000 customer lifetime value to get a positive ROI on your hardware investment. When I talk about AI, I usually call it "beating the problem to death with cheap computing power". And looking at the average cleverness of AI algorithm training formulas, that seems to be exactly what everyone else is doing, too. And since I'm being snarky anyway, there's two subdivisions to AI: supervised learning => remember this unsupervised learning => approximate this Both approaches don't put much emphasis on intelligence ;) And both approaches can usually be implemented more efficiently without AI, if you know what you are doing.
- m0zg 7y ago
  Some kinds of inference are expensive, yes, not going to dispute that. But 99.95% of it is actually surprisingly inexpensive. Hell, a lot of useful workloads can be deployed on a cell phone nowadays, and that fraction will increase over time, further reducing inference costs or eliminating them outright (or rather moving them to the consumer). For the vast majority of people the main expense is creating the combination of a dataset and model that works for their practical problem, with the dataset being the harder (and sometimes more expensive) problem of the two. The dataset is also their "moat", even though most of them don't realize it, and don't put enough care into that part of the pipeline.
  - fxtentacle 7y ago
    The algorithms that run on cell phones tend to be specially optimized and quality-reduced neural networks. For example, https://arxiv.org/abs/1704.04861 https://arxiv.org/abs/1704.04861
    - jiofih 7y ago
zitterbewegung 7y ago
I was training ML models on AWS / Google Colab. After racking up a few hundred dollars on AWS I bought a Titan RTX (I also play video games so it does that very well also.
artsyca 7y ago
Slow clap
paulddraper 7y ago
> Also, if your startup is venture funded, AWS will give you $100K in credit AFAIK that is limited to <$20k and it expires.
- webel0 7y ago
  We got $100k but boy oh boy once you’re on it it’s hard to get off. Now we’re close to 50-50 gcloud/aws
  - wpietri 7y ago
    Hah! "First hit is free, man!" Makes sense, though. Anybody offering a "free" $100k of anything can afford a lot of MBA time to make sure that the CLTV is well over costs.
walshemj 7y ago
Yep if you getting huge bills you should be doing on prem HPC eg where a 15k budget means 15kw per container and your into exotic network designs where 10g wont cut it any more. eg from 2011 6400 Hadoop nodes like http://bradhedlund.com/2011/11/05/hadoop-network-design-challenge/ http://bradhedlund.com/2011/11/05/hadoop-network-design-chal... God only knows what fun you could get up to with modern tech - I miss bleeding edge rnd
calebkaiser 7y ago
Inference is also becoming a bigger contributor to compute bills, especially as models get bigger. With big models like GPT-2, its not unheard of for teams to scale up to hundreds of GPU instances to handle a surprisingly small number of concurrent users. Things can get expensive pretty quick.
liuliu 7y ago
I've been playing with custom-built 2080 Ti workstation for a while: https://www.youtube.com/watch?v=OF3JYEIsjH8 https://www.youtube.com/watch?v=OF3JYEIsjH8 Several issues: 1. electricity bill is still an issue, I've been paying anywhere between $500 to $1000 per month for this workstation (always have something to train). 2. something with a decent memory size (Titan RTX and RTX 8000) cost way too much; 3. once you reached a point of 4-2080Ti-is-not-fast-enough, power management and connectivity setup would be a nightmare. Would love to know other people's opinions on the on-prem setup, especially whether a consumer-grade 10Ghe is enough for connectivity-wise.
- eyegor 7y ago
  10gbe will depend on the workload. In general, I'd assume it's fine because it takes a parallel raid setup to saturate. Upgrading to 100gbe is pretty unreasonable cost wise unless you buy network gear from a back alley van dealer. Although once you reach 4 2080ti, you ought to consider switching to a titanium grade psu and rewiring if you're in a 100-120v country. If you're feeling cheap, just steal the phases from two different circuits. Last I looked, most psu operate around 5% lower efficiency on 115 vs 230.
  - liuliu 7y ago
    I've encountered some latency issues with allreduce on transformer models due to vocabulary sizes when communication have to cross PCIe lanes. Increasing batch size helps a lot, but low-latency & high-throughput is universally more helpful to lift these minor concerns (I don't really want to care about my batch size to improve allreduce performance). Hence worried not only throughput, but also latency on consumer-grade 10ghe equipments.
  - xyzzyz 7y ago
    If you're feeling cheap, just steal the phases from two different circuits. I chukled, but more seriously, if you can't rewire your house to get a normal 240V circuit, you should not be fucking around with hacks like above.
- m0zg 7y ago

by m0zg 7y ago

avip 7y ago

For balance, all big cloud providers - aws, gcp, azure, oracle [0] have pretty similar startup plans. Y$$MV (I'm in full agreement with everything you've written + it's well-phrased and funny. gj!) [0] that's not a typo - there is such thing as "Oracle cloud"

alephnan 7y ago

> Just don't call it a "datacenter" or NVIDIA will have a stroke. Context please :) ?

mereel 7y ago
Just a guess but maybe it's some licensing issue? https://www.nvidia.com/en-us/drivers/geforce-license/ https://www.nvidia.com/en-us/drivers/geforce-license/ No Datacenter Deployment. The SOFTWARE is not licensed for datacenter deployment, except that blockchain processing in a datacenter is permitted.
- mam2 7y ago
  well they are the one writing the rules, so i'd side with OP
- derefr 7y ago
  > except that blockchain processing in a datacenter is permitted Well, Nvidia, y'see, my new blockchain does AI training as its Proof-of-Work step...
ThePadawan 7y ago
Datacenter GPUs are mostly identical to the much cheaper consumer versions. The only thing preventing you from running a datacenter with consumer hardware is the licensing agreement you accept.
- KaiserPro 7y ago
  And the cooling, amount of ram and the doubles performance. the chip might be the same, but the rest of it isn't Granted, its not worth the $3k price bump, but thats a different issue.
  - zwaps 7y ago
    Nah that's not really it. The reason NVIDIA doesn't allow this is precisely because the additional RAM - functionally the only difference - is not cost efficient. People would like (and did) use a bunch of consumer 1080s, which is why NVIDIA disallowed precisely that. You had to buy the equivalent pro grade card, which costs easily two or three times that and offers a couple more GB of RAM.
- endorphone 7y ago
  "The only thing preventing you from running a datacenter with consumer hardware is the licensing agreement you accept." The consumer cards don't use ECC and memory errors are a common issue (GDDR6 running at the edge of its capabilities). In a gaming situation that means a polygon might be wrong, a visual glitch occurs, a texture isn't rendered right -- things that just don't matter. For scientific purposes that same glitch could be catastrophic. The "datacenter" cards offer significantly higher performance for some case (tensor cores, double precision), are designed for data center use, are much more scalable, etc. They also come with over double the memory (which is one of the primary limitations forcing scale outs). Going with the consumer cards is one of those things that might be Pyrrhic. If funds are super low and you want to just get started, sure, but any implication that the only difference is a license is simply incorrect.
  - luisfmh 7y ago
    Learned a new word today. Pyrrhic.
OkGoDoIt 7y ago
NVIDIA forces you to buy significantly more expensive cards that perform marginally better if you are using them for datacenter use. They try to enforce not letting businesses use consumer grade gaming cards. I assume this is so cloud providers don't buy up all the supply of graphics cards and make it hard for gamers to get decent cards, like what happened during the bitcoin craze.
- nkassis 7y ago
  No it's just pure price discrimination. They don't care about gamers they just know businesses will pay more if forced to while gamers can't.
  - aj7 7y ago
    Exactly.
  - gbba 7y ago
    I wouldn't say they don't care about gamers, considering that gaming makes up about half of their revenue: https://www.anandtech.com/show/15513/nvidia-releases-q4-fy2020-earnings-a-strong-quarter-closes-our-a-weaker-year https://www.anandtech.com/show/15513/nvidia-releases-q4-fy20...
    - nkassis 7y ago
      Sure sorry I realize now the point I was trying to make doesn't match my wording. They do care about selling to gamers but availability to gamers is not in anyway why they are forcing more expensive models of essentially the same cards on the HPC and server market. It's all because they can and the business market is able to bear the cost. Now if they found a true competitor in that market, I think this pricing model would fall apart fast.
deleted 7y ago
[deleted]
deleted 7y ago
[deleted]

fizixer 7y ago

- Or AMD could change their policy of 'never miss an opportunity to miss an opportunity' and offer high-performance OpenCL GPGPU offerings. Then nVidia could have all the stroke they wanted. - Or Tensorflow/Pytorch could've crapped on OpenCL a little less by releasing a fully functional OpenCL version everytime they released a fully functional Cuda version, instead of worshipping Cuda year in and year out. - Or Google could start selling their TPUv2, if not TPUv3, while they're on the verge of releasing TPUv4. - Or one of the other big-tech's Facebook/Microsoft/Intel could make and start selling a TPU-equivalent device. - Or I could finish school and get funded to do all/most of the above ;) edit: On a more serious note, a cloud/on-prem hybrid is absolutely the right way to go. You should have a 4x 2080 ti rig available 24x7 for every ML engineer. It costs about $6k-8k a piece [0]. Prototype the hell out of your models on on-prem hardware. Then when your setup is in working condition and starts producing good results on small problems, you're ready to do a big computation for final model training. Then you send it to the cloud, for final production run. (Guess what, on a majority of your projects, you might realize, the final production run could be carried out on on-prem itself; you just have to keep it running 24 hours-a-day for a few days or up to a couple weeks.) [0]: https://l7.curtisnorthcutt.com/the-best-4-gpu-deep-learning-rig https://l7.curtisnorthcutt.com/the-best-4-gpu-deep-learning-...

m0zg 7y ago
As someone who has actually worked on this stuff soup to nuts, it's not as easy as people imagine, because you can't just support some subset of available ops and call it a day. If you want to make OpenCL pie from scratch, you must first make the universe, and support every single stupid thing (among thousands) and even mimic some of the bugs so that models work "the same". This is hard and time consuming, and this field is hard enough as it is. What makes it even harder is that only NVIDIA has decent, mature tooling. There is some work on ROCM though, so AMD is not _totally_ dead in the water. I'd say they're about 90% dead in the water.
- simonebrunozzi 7y ago
  Are you in the Bay area? Would love to chat. Thinking of an idea where your expertise could be very handy. $my-hn-username at gmail.

bob1029 7y ago

I find it fun how the cost of the cloud is forcing people to consider what absolutely must run in the cloud (presumably for stability and compliance reasons) and what can be brought back on-prem. We don't train ML models, but we are in a similar boat regarding cloud compute costs. Building our solutions for our clients is a compute-heavy task which is getting expensive in the cloud. We are considering options such as building commodity threadripper rigs, throwing them in various developers' (home) offices, installing a VPN client on each and then attaching as build agents to our AWS-hosted jenkins instance. In this configuration we could drop down to a t3a.micro for Jenkins and still see much faster builds. The reduction in iteration time over a month would easily pay for the new hardware. An obvious next step up from this is to do proper colocation, but I am of a mindset that if I have to start racking servers I am bringing 100% of our infrastructure out of the cloud.

m0zg 7y ago
This is not a new phenomenon. As early as in 2009 I worked for a company (ads, but not Google) which outgrew the typical "cloud" cost structure at the time, and moved everything to a more traditional datacenter, and saved substantial money even considering 3 more SREs they had to hire to absorb the increased support needs. AWS charges what the market will bear, and as such it was never designed to make sense for everyone. One needs to re-evaluate on the back of the napkin from time to time.
buckminster 7y ago
I once had a borrowed Sun blade server in my home office. The fan in it sounded like an industrial vacuum cleaner. It got moved to a different room and was powered on as little as possible. Your plan makes sense but be mindful of the acoustics or your devs may grow to hate you.
- m0zg 7y ago
  BTW, the only reason why consumer vacuum cleaners are so loud is because consumers associate loudness with suction power. "Backpack" style commercial vacuum cleaners have more suction, and are barely audible in comparison.

ignoramous 7y ago

As someone hoping to build a world-wide footprint, say 25 to 50 DCs, of servers to deploy to with unmetered bandwidth, what are some alternatives to the usual suspects? I have come across fly.io, Vultr, Scaleway, Stackpath, Hetzner, and OVH but either they are expensive (in that they charge for bandwidth and uptime) or do not have a wide enough foot-print. I guess colos are the way to go, but how does one work with colos, allocate servers, deploy to them, ensure security and uptime and so on from a single place, 'cause dealing with them individually might slow down the process? Is there a tooling that deals with multi-colos like the ones for multi-cloud like min.io, k8s, Triton etc;

mrkurt 7y ago
(Hi, I'm from fly.io) It depends what you need in your datacenters! If you just want servers, and don't care about doing something like anycast, you can find a bunch of local dedicated server providers in a bunch of cities and go to town. But you can't get them all from one provider, really, not with any kind of reasonable budget. You _could_ buy colo from a place like Equinix in a bunch of cities, and then either use their transit or buy from other transit providers. But also, unmetered bandwidth isn't a very sustainable service, so I'm curious what you're after? You're usually either going to have to pay for usage, or pay large monthly fixed prices to get reasonable transit connections in each datacenter. In our case, we're constrained by Anycast. To expand past the 17 usual cities you end up needing to do your own network engineering which we'd rather not do yet.
- ignoramous 7y ago
  (thanks mrkurt) It is anycast that I'm going after. Requirement for unmetered bandwidth (or cheaper than AWS et al) is because of the kind of workloads (TURN relays, proxy, tunnels etc) we'd deal with gets expensive, otherwise. For another related workload, per-request pricing gets expensive, again, due to the nature of the workload (to the tune 100k requests per user per month). So far, for the former (TURN relays etc), I've found using AWS Global Accelerator and/or GCP's GLB to be the easiest way to do anycast but the bandwidth is slightly on the expensive side. Fly.io matches the pricing in terms of network bandwidth (as promised on the website), so that's a positive but GCP/AWS have a wider footprint. Cloudflare's Magic Transit is another potential solution, but requires an enterprise plan and one needs to bring-your-own-anycast-IP and origin-servers. For the latter (latency-sensitive workload with ~100k+ reqs / month), Cloudflare Workers (200+ locations minus China) are a great fit though would get expensive once we hit a certain scale. Plus, they're limited to L7 HTTP reqs, only. Whilst, I believe, fly.io can do L4.

pridkett 7y ago

There’s also the issue that data scientists often want to go running to hyperparameter optimization and neural architecture search. In most cases improving your data pipelines and ensuring the data are clean and efficient will pay off much more quickly.

fxtentacle 7y ago
But manually improving the data pipeline requires an understanding of the problem, whereas doing a hyperparameter optimized architecture search just needs $$$ hardware and no clue on the side of the operator.
- derefr 7y ago
  Or, to put that another way: if you knew what algorithm the AI would be using to discriminate the signal from the noise in your data, why would you need the AI? Just write that algorithm.
  - fxtentacle 7y ago
    Exactly :) In most cases, unsupervised learning is nothing more than having the AI try to approximate the solution of your highly non-linear loss function. So if there's any way of solving that loss function directly, it will perform like a well-trained AI.
    - mxcrossb 7y ago
      > In most cases In what cases is this not true?
  - nl 7y ago
    That's not the same thing at all. You can't build hand-build a feature detector as accurate as (say) a ResNet50 by hand. Before 2014 people tried to do this with feature detectors like SIFT and HOG. These were patented and made the inventors significant money. If it was still possible to do it then someone would be and making profit from it. Hyperparameter search is just optimising the training parameters (things like batch size, or optimiser parameters). This might give another 1% lift on accuracy, but isn't generally a significant factor.

fxtentacle 7y ago

No, also inference is quite expensive. You'll have 100% usage on a $10,000 GPU for 3s per customer image for a decently sized optical flow network. That's 3 hours of compute time for 1 minute of 60fps video. Now let's say your customer wants to analyze 2 hours = 120 minutes of video and doesn't want to wait more than those 3 hours, then suddenly you need 120 servers with one $10k GPU each to service this one customer within 3 hours of waiting. Good luck reaching that $1,200,000 customer lifetime value to get a positive ROI on your hardware investment. When I talk about AI, I usually call it "beating the problem to death with cheap computing power". And looking at the average cleverness of AI algorithm training formulas, that seems to be exactly what everyone else is doing, too. And since I'm being snarky anyway, there's two subdivisions to AI: supervised learning => remember this unsupervised learning => approximate this Both approaches don't put much emphasis on intelligence ;) And both approaches can usually be implemented more efficiently without AI, if you know what you are doing.

m0zg 7y ago
Some kinds of inference are expensive, yes, not going to dispute that. But 99.95% of it is actually surprisingly inexpensive. Hell, a lot of useful workloads can be deployed on a cell phone nowadays, and that fraction will increase over time, further reducing inference costs or eliminating them outright (or rather moving them to the consumer). For the vast majority of people the main expense is creating the combination of a dataset and model that works for their practical problem, with the dataset being the harder (and sometimes more expensive) problem of the two. The dataset is also their "moat", even though most of them don't realize it, and don't put enough care into that part of the pipeline.
- fxtentacle 7y ago
  The algorithms that run on cell phones tend to be specially optimized and quality-reduced neural networks. For example, https://arxiv.org/abs/1704.04861 https://arxiv.org/abs/1704.04861
  - jiofih 7y ago

zitterbewegung 7y ago

I was training ML models on AWS / Google Colab. After racking up a few hundred dollars on AWS I bought a Titan RTX (I also play video games so it does that very well also.

artsyca 7y ago

Slow clap

paulddraper 7y ago

> Also, if your startup is venture funded, AWS will give you $100K in credit AFAIK that is limited to <$20k and it expires.

webel0 7y ago
We got $100k but boy oh boy once you’re on it it’s hard to get off. Now we’re close to 50-50 gcloud/aws
- wpietri 7y ago
  Hah! "First hit is free, man!" Makes sense, though. Anybody offering a "free" $100k of anything can afford a lot of MBA time to make sure that the CLTV is well over costs.

walshemj 7y ago

Yep if you getting huge bills you should be doing on prem HPC eg where a 15k budget means 15kw per container and your into exotic network designs where 10g wont cut it any more. eg from 2011 6400 Hadoop nodes like http://bradhedlund.com/2011/11/05/hadoop-network-design-challenge/ http://bradhedlund.com/2011/11/05/hadoop-network-design-chal... God only knows what fun you could get up to with modern tech - I miss bleeding edge rnd

calebkaiser 7y ago

Inference is also becoming a bigger contributor to compute bills, especially as models get bigger. With big models like GPT-2, its not unheard of for teams to scale up to hundreds of GPU instances to handle a surprisingly small number of concurrent users. Things can get expensive pretty quick.

liuliu 7y ago

I've been playing with custom-built 2080 Ti workstation for a while: https://www.youtube.com/watch?v=OF3JYEIsjH8 https://www.youtube.com/watch?v=OF3JYEIsjH8 Several issues: 1. electricity bill is still an issue, I've been paying anywhere between $500 to $1000 per month for this workstation (always have something to train). 2. something with a decent memory size (Titan RTX and RTX 8000) cost way too much; 3. once you reached a point of 4-2080Ti-is-not-fast-enough, power management and connectivity setup would be a nightmare. Would love to know other people's opinions on the on-prem setup, especially whether a consumer-grade 10Ghe is enough for connectivity-wise.

eyegor 7y ago
10gbe will depend on the workload. In general, I'd assume it's fine because it takes a parallel raid setup to saturate. Upgrading to 100gbe is pretty unreasonable cost wise unless you buy network gear from a back alley van dealer. Although once you reach 4 2080ti, you ought to consider switching to a titanium grade psu and rewiring if you're in a 100-120v country. If you're feeling cheap, just steal the phases from two different circuits. Last I looked, most psu operate around 5% lower efficiency on 115 vs 230.
- liuliu 7y ago
  I've encountered some latency issues with allreduce on transformer models due to vocabulary sizes when communication have to cross PCIe lanes. Increasing batch size helps a lot, but low-latency & high-throughput is universally more helpful to lift these minor concerns (I don't really want to care about my batch size to improve allreduce performance). Hence worried not only throughput, but also latency on consumer-grade 10ghe equipments.
- xyzzyz 7y ago
  If you're feeling cheap, just steal the phases from two different circuits. I chukled, but more seriously, if you can't rewire your house to get a normal 240V circuit, you should not be fucking around with hacks like above.
m0zg 7y ago