6 ms·

This question potentially lacks the situational context necessary to provide the answer/insight it's really looking for. What's the SaaS going to look like tec

This question potentially lacks the situational context necessary to provide the answer/insight it's really looking for. What's the SaaS going to look like technically? Lots of compute processing? Or maybe large volumes of data? Perhaps machine learning? Or is it just a simple website and app deal? Then there's the question of the technical competence on the ground (and the opinions that tag along for the ride). On the one hand, what's the expertise level? On the other hand, what do people prefer using? (And separately, what to people have experience with?) I wonder (I don't have enough context to competently move the slider to "proper accusation") if somewhere along the way a general search for un-turned-over rocks got specialized/narrowed into "the overhead's all in the sheet metal benders" (the hardware). If this *is* the case, the only correct course of action (IMO) is to wind the train of thought backwards (oohc oohc) back to that frame of reference, then apply https://en.wikipedia.org/wiki/Five_whys https://en.wikipedia.org/wiki/Five_whys until you're somewhere alien and interesting. (Reiterating the caveat at the start of the previous paragraph, this is all massive conjecture and just-in-case assumption.) If there was in fact a point at which hardware was specifically called out as a primary focus of optimization, I would loudly note that this type of hyperfocus leaves space for entire forests' worth of trees to fall over without ever being noticed, in this case for all of the software. Not just certain bits of it but like the whole kit and caboodle, unnoticed. A related alternate possibility is that hardware optimization might be being treated like an axis point to orbit around, which can contribute to seeing things like immovable Mt Everests, and the construction of great and confusing Rube Goldberg machines to work around... perceived resistance that isn't really there. To offer a 180-degree counterpoint that sorta flies in the face of the abstraction-away you might be trying to do here, if you want to look at different providers, I would suggest adopting a pat-answer strategy of keeping the architecture cloud-agnostic *where reasonable to do so*, and further suggest trying multiple providers - how does their support help out when it's 3am and you have no attention span and you accidentally something ridiculously straightforward? What's the performance like relative to the workload? What do all the engineers that are going to be headdesking against this stuff every day think about different options? Etc. It's quite possible this reply is entirely misguided, in which case please ignore. I'm still learning the art of reading intent through text, I have a long way to go.

tomrod 4y ago
> It's quite possible this reply is entirely misguided, in which case please ignore. I'm still learning the art of reading intent through text, I have a long way to go. Your mindfulness is quite appreciated and your answer is on point! We are at the design phase at this point. Fairly data intensive and will definitely have ML components (but that could be centralized and managed with standard MLOps practices). But the data is more on the backend, forward facing components are pretty standard CRUD/API calls against pre-computed and quickly updating data stores. User-facing data payloads are in design, but expected to be low-to-mid volume.
- exikyut 4y ago
  I'm trying to calibrate a working mindset to get into the industry from a bit of a distance, so thanks for the feedback - and for engaging my somewhat hastily written comment :) I recently realized (light bulb moment) that cloud platforms like AWS are awesome for prototyping and experimentation: you can spin up seriously wide+deep instances that bill $1/hr, crunch through a complex workload in a couple of hours, and use hundreds of GB of RAM connected to hundreds of CPU cores for the cost of a stereotypical cup of coffee. But that bottoms out after just a very short period of time; a literal couple of hours. I wouldn't be surprised to learn the price scales was intentionally tuned to feel accessible at the prototyping stage - this would take advantage of communicative blind spots on both sides of the management/engineering boundary during evaluation and testing, where management gives engineering a green light budget to play around and learn about the platform, the engineers get insightful field experience after not very much expenditure, it feels possible to do a lot of work without spending too much, and... woops, a short time later they're staring down the barrel of a deadline, the fastest solution is to use their newfound experience and familiarity... and the cycle repeats. The theoretical blind spot here is of course in the learning/tinkering stage that is so hard to communicate to management - and so when engineering makes positive noises about having some level of understanding and confidence about how to do a particular thing on AWS, the nuance of the hair-thin line between "thing that is generally achievable" and "I've practically played with AWS' implementation of the thing and I understand how to do it in that sandbox" is lost. (And the cycle repeats.) What's the right word for something incredibly well-engineered for altogether depressing reasons? :( (Of course manglement and disorganization can also be at fault here, although I wonder where the cultural/structural root cause was when NASA forgot the egress bandwidth costs of migrating to AWS :D https://news.ycombinator.com/item?id=22626097 https://news.ycombinator.com/item?id=22626097) Maybe the moral of the above story/thought experiment might be to theorize that AWS's market dominance has had a significant impact on broad tinkering and experimentation, and adjust for that by explicitly authorizing a broad-spectrum tinkering budget - "go and play" meets "engineering feasibility study" or something. Take the brakes off just enough that the engineering opinions that eventually come back are the product of having had the chance to stare into the horizon blankly for a bit, that sort of thing. Hypothetically. Regarding the data intensity bit you mentioned, I'm personally up to the "just buy the whole computer" point at the moment mindset-wise, since this presents the most cost savings where it's feasible to do so. One somewhat cute but relatively transferable/normalizable way of broadly articulating the hard differences is to find vendor workstation/server configurators, adjust all the settings until the big number with the $ in front won't go any higher :), then (try and) configure something comparable on EC2. An HP Z8 workstation with 2x28 core Xeon 8280s, 3TB of RAM, 172TB of storage, and 3 NVIDIA GPUs will set you back $111,575: https://zworkstations.com/products/hp-z8-workstation/?configuration=3068809 https://zworkstations.com/products/hp-z8-workstation/?config... An EC2 x2iedn.24xlarge (wat) with 96 CPUs, 3TB of RAM, 2 1.4TB SSDs, and 50TB outbound bandwidth, will set you back - $192,625 for 1 year at standard monthly rates: https://calculator.aws/#/estimate?id=e9d103d78312322c281bc0c989fd2bd4561482ac https://calculator.aws/#/estimate?id=e9d103d78312322c281bc0c... - $185,436 if you buy 1 year in advance: https://calculator.aws/#/estimate?id=c57064223f0aa745207efcf264971765a43b7357 https://calculator.aws/#/estimate?id=c57064223f0aa745207efcf... - $231,575 if you buy 3 years in advance: https://calculator.aws/#/estimate?id=2795a93261ed9353668b1bd6a51d057ebf029a63 https://calculator.aws/#/estimate?id=2795a93261ed9353668b1bd... An EC2 p4d.24xlarge (again, wat) with 96 CPUs, 1.1TB RAM, 8TB NVMe, 8 GPUs and no outbound transit (I couldn't figure out how to add any) is $196,398 for 1 year at standard monthly rates: https://calculator.aws/#/estimate?id=ef23ee773d298c703015453f783ff0b929386cc8 https://calculator.aws/#/estimate?id=ef23ee773d298c703015453... So this specific class of hardware would pay for itself in just over half a year ((111/192)*12=6.9375 months), give you total monopoly over resource availability, and redefine the bandwidth usage situation. The question then is whether the same sort of wet-fish disparity also exists within the target performance window you'd be aiming for. In practice, datacenter colocation only charges for rack U height, power consumption and network bandwidth, regardless of the cost of the hardware (very cool) - and hmm, now I'm wondering if insurance options offer different premiums for different types of colocation facility standards - and then the only ongoing costs are having solid sysadmins (oh hey, remember those? lol) and inevitable hardware replacement (something something ZFS hot/warm spares). The infuriating twist (sigh) is the ML part, and whether you want to do that on GPUs or otherwise. I read a highly impressionable comment chain from someone who'd had some great engineering experiences playing with Google TPUs a few months back (https://news.ycombinator.com/item?id=27728225 https://news.ycombinator.com/item?id=27728225), and I wonder if the status quo described there has shifted at all - perhaps the scope of access has been reined in somewhat, or billing have come in and poured a pricing bucket over the engineering parade. I'm not sure. But what's stuck with me, sadly, about accelerator technology like this is that the cloud-computing theme of normalization-of-centralization unfortunately means this sort of hardware is only network-accessible from a large vendor like Google; you can't buy them. Infuriatingly. So even if the amazing-sounding prototyping opportunity described in the above link is still a thing (oh hey... the prototyping stage... this seems familiar ......), at production scale the only question that's on the table is "what are the pricing tiers?" since there's no leverage to be had, you either want the service at the price it's offered at or you don't. Hence my question about using GPUs, which are, arguably, only barely at the break-even point in terms of buying vs renting for a lot of workloads. (Probably in large part only because enough large-scale scenarios demand on-premises provenance.) Because your alternatives are HaaS (hardware-as-a-service) for whatever the cloud providers offer. Yay. I'm not entirely sure what you were referring to in terms of ML centralization - whether from a hardware or software standpoint - but I definitely go for the the "build a moat around it" mentality myself. I don't have enough experience to have the first clue what that would look like in practice though - just that it's a giant rabbithole of risk. (And a nuanced one, since Google et al want to generate demand for their product/service! Just... within the comfort zone of rent-seeking. Adlskdfglkjsdflgd *headdesk*) In terms of what sounds like front/mid-end caching, Cloudflare currently seems to want to seriously differentiate on egress bandwidth, making it legitimately hard to dismiss them as interesting; but I do wonder what they'll do once they have both engineering familiarity and account entrenchment. I recall how Amazon Cloud Drive initially launched as both free and unlimited, then abruptly announced $60/TB pricing after people had uploaded TBs of data (and that one guy who uploaded 1 PB of test images). Cloudflare are certainly smart enough to see that similar actions would rapidly reorganize their reputation in a hurry, so I do wonder what their long-term strategy is there. Distilling the value-add down to basically "200+ PoPs and the world's lowest latency", it's totally possible to sustainably reproduce a heavy subset of that capability using commodity replication in a few key locations; and that may well suffice, supporting the argument that the value-add is not a critical requirement.

tomrod 4y ago

> It's quite possible this reply is entirely misguided, in which case please ignore. I'm still learning the art of reading intent through text, I have a long way to go. Your mindfulness is quite appreciated and your answer is on point! We are at the design phase at this point. Fairly data intensive and will definitely have ML components (but that could be centralized and managed with standard MLOps practices). But the data is more on the backend, forward facing components are pretty standard CRUD/API calls against pre-computed and quickly updating data stores. User-facing data payloads are in design, but expected to be low-to-mid volume.

exikyut 4y ago
I'm trying to calibrate a working mindset to get into the industry from a bit of a distance, so thanks for the feedback - and for engaging my somewhat hastily written comment :) I recently realized (light bulb moment) that cloud platforms like AWS are awesome for prototyping and experimentation: you can spin up seriously wide+deep instances that bill $1/hr, crunch through a complex workload in a couple of hours, and use hundreds of GB of RAM connected to hundreds of CPU cores for the cost of a stereotypical cup of coffee. But that bottoms out after just a very short period of time; a literal couple of hours. I wouldn't be surprised to learn the price scales was intentionally tuned to feel accessible at the prototyping stage - this would take advantage of communicative blind spots on both sides of the management/engineering boundary during evaluation and testing, where management gives engineering a green light budget to play around and learn about the platform, the engineers get insightful field experience after not very much expenditure, it feels possible to do a lot of work without spending too much, and... woops, a short time later they're staring down the barrel of a deadline, the fastest solution is to use their newfound experience and familiarity... and the cycle repeats. The theoretical blind spot here is of course in the learning/tinkering stage that is so hard to communicate to management - and so when engineering makes positive noises about having some level of understanding and confidence about how to do a particular thing on AWS, the nuance of the hair-thin line between "thing that is generally achievable" and "I've practically played with AWS' implementation of the thing and I understand how to do it in that sandbox" is lost. (And the cycle repeats.) What's the right word for something incredibly well-engineered for altogether depressing reasons? :( (Of course manglement and disorganization can also be at fault here, although I wonder where the cultural/structural root cause was when NASA forgot the egress bandwidth costs of migrating to AWS :D https://news.ycombinator.com/item?id=22626097 https://news.ycombinator.com/item?id=22626097) Maybe the moral of the above story/thought experiment might be to theorize that AWS's market dominance has had a significant impact on broad tinkering and experimentation, and adjust for that by explicitly authorizing a broad-spectrum tinkering budget - "go and play" meets "engineering feasibility study" or something. Take the brakes off just enough that the engineering opinions that eventually come back are the product of having had the chance to stare into the horizon blankly for a bit, that sort of thing. Hypothetically. Regarding the data intensity bit you mentioned, I'm personally up to the "just buy the whole computer" point at the moment mindset-wise, since this presents the most cost savings where it's feasible to do so. One somewhat cute but relatively transferable/normalizable way of broadly articulating the hard differences is to find vendor workstation/server configurators, adjust all the settings until the big number with the $ in front won't go any higher :), then (try and) configure something comparable on EC2. An HP Z8 workstation with 2x28 core Xeon 8280s, 3TB of RAM, 172TB of storage, and 3 NVIDIA GPUs will set you back $111,575: https://zworkstations.com/products/hp-z8-workstation/?configuration=3068809 https://zworkstations.com/products/hp-z8-workstation/?config... An EC2 x2iedn.24xlarge (wat) with 96 CPUs, 3TB of RAM, 2 1.4TB SSDs, and 50TB outbound bandwidth, will set you back - $192,625 for 1 year at standard monthly rates: https://calculator.aws/#/estimate?id=e9d103d78312322c281bc0c989fd2bd4561482ac https://calculator.aws/#/estimate?id=e9d103d78312322c281bc0c... - $185,436 if you buy 1 year in advance: https://calculator.aws/#/estimate?id=c57064223f0aa745207efcf264971765a43b7357 https://calculator.aws/#/estimate?id=c57064223f0aa745207efcf... - $231,575 if you buy 3 years in advance: https://calculator.aws/#/estimate?id=2795a93261ed9353668b1bd6a51d057ebf029a63 https://calculator.aws/#/estimate?id=2795a93261ed9353668b1bd... An EC2 p4d.24xlarge (again, wat) with 96 CPUs, 1.1TB RAM, 8TB NVMe, 8 GPUs and no outbound transit (I couldn't figure out how to add any) is $196,398 for 1 year at standard monthly rates: https://calculator.aws/#/estimate?id=ef23ee773d298c703015453f783ff0b929386cc8 https://calculator.aws/#/estimate?id=ef23ee773d298c703015453... So this specific class of hardware would pay for itself in just over half a year ((111/192)*12=6.9375 months), give you total monopoly over resource availability, and redefine the bandwidth usage situation. The question then is whether the same sort of wet-fish disparity also exists within the target performance window you'd be aiming for. In practice, datacenter colocation only charges for rack U height, power consumption and network bandwidth, regardless of the cost of the hardware (very cool) - and hmm, now I'm wondering if insurance options offer different premiums for different types of colocation facility standards - and then the only ongoing costs are having solid sysadmins (oh hey, remember those? lol) and inevitable hardware replacement (something something ZFS hot/warm spares). The infuriating twist (sigh) is the ML part, and whether you want to do that on GPUs or otherwise. I read a highly impressionable comment chain from someone who'd had some great engineering experiences playing with Google TPUs a few months back (https://news.ycombinator.com/item?id=27728225 https://news.ycombinator.com/item?id=27728225), and I wonder if the status quo described there has shifted at all - perhaps the scope of access has been reined in somewhat, or billing have come in and poured a pricing bucket over the engineering parade. I'm not sure. But what's stuck with me, sadly, about accelerator technology like this is that the cloud-computing theme of normalization-of-centralization unfortunately means this sort of hardware is only network-accessible from a large vendor like Google; you can't buy them. Infuriatingly. So even if the amazing-sounding prototyping opportunity described in the above link is still a thing (oh hey... the prototyping stage... this seems familiar ......), at production scale the only question that's on the table is "what are the pricing tiers?" since there's no leverage to be had, you either want the service at the price it's offered at or you don't. Hence my question about using GPUs, which are, arguably, only barely at the break-even point in terms of buying vs renting for a lot of workloads. (Probably in large part only because enough large-scale scenarios demand on-premises provenance.) Because your alternatives are HaaS (hardware-as-a-service) for whatever the cloud providers offer. Yay. I'm not entirely sure what you were referring to in terms of ML centralization - whether from a hardware or software standpoint - but I definitely go for the the "build a moat around it" mentality myself. I don't have enough experience to have the first clue what that would look like in practice though - just that it's a giant rabbithole of risk. (And a nuanced one, since Google et al want to generate demand for their product/service! Just... within the comfort zone of rent-seeking. Adlskdfglkjsdflgd *headdesk*) In terms of what sounds like front/mid-end caching, Cloudflare currently seems to want to seriously differentiate on egress bandwidth, making it legitimately hard to dismiss them as interesting; but I do wonder what they'll do once they have both engineering familiarity and account entrenchment. I recall how Amazon Cloud Drive initially launched as both free and unlimited, then abruptly announced $60/TB pricing after people had uploaded TBs of data (and that one guy who uploaded 1 PB of test images). Cloudflare are certainly smart enough to see that similar actions would rapidly reorganize their reputation in a hurry, so I do wonder what their long-term strategy is there. Distilling the value-add down to basically "200+ PoPs and the world's lowest latency", it's totally possible to sustainably reproduce a heavy subset of that capability using commodity replication in a few key locations; and that may well suffice, supporting the argument that the value-add is not a critical requirement.