15 ms·
NASA to launch 247 petabytes of data into AWS, but forgot about egress costs
- toomuchtodo 7y agoCue the cloud apologists that “it’s better to use the cloud than to build and manage your own infra”. This is why you build and run your own storage, similar to Backblaze (who is almost entirely bootstrapped except for one reasonable round of investment).
- pas 7y agoI thought the ultimate argument was that if you're big enough AWS will make you a deal. But maybe now AWS is just so big and already growing so fast, they don't want to make exceptions and lower their profitability.
- belval 7y agoThey got a 50% deal. From the article: "At least NASA seems to have bagged a good deal from AWS: The Register used Amazon’s cloudy cost calculator to tot up the cost of storing 247PB in the cloud giant’s S3 service. The promised pay-as-you-go price for us on the street was a staggering $5,439,526.92 per month, not taking into account the free tier discount of 12 cents. The audit, meanwhile, suggests an increased cloud spend of around $30m a year by 2025, on top of NASA’s $65m-per-year deal with AWS."
- jleahy 7y ago$5.4m/mo * 12 mo/yr = $65m/yr. My guess is the "$65m/year deal with AWS" is actually the S3 cost and the extra $30m/year of 'increased cloud spend' is the egress costs found by the audit. Otherwise it's a coincidence of the numbers.
- Karunamon 7y agoCue the cloud detractors that "a failure to do due diligence (in this case: 15 minutes on the pricing calculator) on your computing platform should be held against the whole platform". Snark aside, it entirely depends on what you're doing. AWS probably has better engineers, better processes, and more of them than your company.
- falcolas 7y agoNone of which will really help you, since AWS priority is AWS, not the uptime of your business. And no number of those better engineers or processes have prevented downtime and service interruptions on AWS.
- unethical_ban 7y agoOh, man. Better run your own Internet, after all, you care more about connectivity to your friends than your ISP does! Dogmatism is passé. There are good uses for cloud, and good times for on-premise, depending on what you need, what your skillsets are as an organization, the kinds of workloads and length of time required for that workload. AWS and others have absolutely outstanding amounts of infrastructure and tooling. Their reliability is off the charts in the past few years, and (once it actually gets figured out by your engineers) the cloud concept of IAM is incredibly secure. There are pitfalls - cost, up-front complexity and several other things - but I no longer rag on "the cloud".
- toomuchtodo 7y agoAmazon has outages all the time, hidden on their status board with a green triangle, and you still lose S3 objects once you’re operating at a large enough scale. A quick google search for “amazon outages” lists the numerous extended outages they’ve experienced.
- Karunamon 7y agoHow many of those outages were multi-region and would have taken down a properly distributed application? How many outages and instances of lost data would the average enterprise, likely without their own datacenters, redundant power, hardware staff, etc have taken in the same period?
- toomuchtodo 7y agoMost applications will never be architected to be “properly distributed” because of cost. Many popular web properties (Reddit) still have outages on AWS even when architected properly. Netflix still distributes content from their own CDN with their OpenConnect appliances, and only uses AWS for non streaming use cases (jedberg will correct me on both Netflix and Reddit points if I'm missing something and comes across this comment). https://www.usatoday.com/story/tech/news/2017/02/28/amazons-cloud-service-goes-down-sites-scramble/98530914/ https://www.usatoday.com/story/tech/news/2017/02/28/amazons-... If my app is architected for reliability, I’ll run it on bare metal and keep the costs savings. Why pay twice by building it for cloud durability and running it on expensive cloud resources? Clearly the AWS marketing is working (“you’re just building it wrong”). We’ll see what happens when CFOs take the reins from CTOs and CIOs and start putting cost controls in place during this recession (“why exactly are we paying so much in opex when this could be capex we can depreciate?”).
- ben509 7y agoTo cloud or not to cloud is the same as any outsourcing decision. For many operations, you may get to a point where it makes sense to build your own cloud. If you're a seller, you might also get to a point where you want to sell goods directly. It partly depends on your core expertise, meaning, is this part of how your business creates value? If NASA doesn't want be a datacenter provider, they should continue to outsource it. It also depends on whether their business model aligns with yours. AWS's egress rules specifically work when you are getting revenue from the data being downloaded. If you're selling software or other media, and you can factor the cost of downloads into the price of it, pay-for-egress is very sustainable. Other models like pay-for-capacity don't align as well if you want to maintain a large library of media and people are attracted by the variety, but only download the popular stuff. For NASA, pay-for-egress may be entirely justified if their budget is based on usage of the data. Or if they can simply use "requester pays" to mitigate the cost.
- deleted 7y ago[deleted]
- deleted 7y ago[deleted]
- ackbar03 7y agoOh but aws didn't forget. Aws never forgets
- gonzo41 7y agoThis is kinda bad press for AWS. If I were NASA I'd be shitty about the relationship manager not hinting and trying to help architect for lowest cost.
- NikolaeVarius 7y agoSince when the hell does NASA actually care about bad press regarding costs these days.
- johnmaguire2013 7y agoNASA or AWS? Parent said AWS.
- badwolf 7y agoNASA is spending over $1B for a launch pad that will be used no more than 4 times. https://spacenews.com/report-finds-delays-and-cost-overruns-in-sls-mobile-launch-platform-development/ https://spacenews.com/report-finds-delays-and-cost-overruns-...
- whatshisface 7y agoHow much would a launch pad that will be used four times normally cost for what they're planning to launch? Without knowing that I can't say if they overpaid 10x, 2x, got it exactly right or got an amazing bargain.
- yborg 7y agoIn 1965, the Vertical Assembly Building, which was at that time the largest enclosed volume in the world, cost $117M (on a $23.5M original construction contract). That would be about a billion dollars in 2020, but it was completed in 3 years and was used to stack 13 Saturn Vs. It was later used for the 100+ Shuttle missions as well, but there were additional costs to modify the building for this purpose. The VAB is still planned for use for future missions. https://www.popsci.com/blog-network/vintage-space/nasas-vab-garage-saturn-v/ https://www.popsci.com/blog-network/vintage-space/nasas-vab-...
- vnchr 7y agoCloud VERSUS Space. Who will come out on top?
- NikolaeVarius 7y agoSenator Shelby should get AWS to launch a new region in Alabama for NASA at this rate.
- gigatexal 7y agomight be cheaper to spin up virtual workstations on AWS and use the data there
- julienchastang 7y agoExactly. Move your computation to the data instead of the other way around. At that point, there are many ways to keep costs down such as using spot instances and tearing down VMs when your analysis is over.
- gigatexal 7y agoAnd you get to rent the latest hardware than use likely old machines ... I mean you use the existing machines as dumb terminals but still
- Dunedan 7y ago> “However, when end users download data from Earthdata Cloud, the agency, not the user, will be charged every time data is egressed. Not necessarily, depending on how the users access the data. If users access the data through their own AWS accounts, NASA could leverage S3's "Requester Pays" feature [1], to let the user pay for downloading the data. 1: https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPay...
- dpcx 7y agoI immediately thought about this as well, however I seem to recall reading somewhere (and I could be entirely wrong here) that NASA has a requirement to give away freely their science data.
- jfk13 7y agoIf there's a marginal cost for each copy of the data that's transferred to a user, I don't think asking the user to cover that cost conflicts with a requirement to "give away the data". (If they distributed their science data in printed form, surely they'd be allowed to charge people for the cost of printing & mailing the paper copies; that's quite different from charging for the data itself.)
- elcritch 7y agoWhy the downvotes? This isn't uncommon or unreasonable if you're downloading TB's of data. Also the data would be freely redistributable if someone took the data and put up a torrent. Still I'd rather see NASA host their own data. Put up an FTP server, torrent server and save a lot of money on hosting fees.
- harlanji 7y agoRecords departments always charge for copies, and that is the use I thought of immediately when I learned of Requester Pays. I’d be surprised if NASA couldn’t use it.
- 7777fps 7y agoI assume the data accessed is a heavily skewed pareto distribution. Given that, it's maybe still cheaper to build their own serving / caching layer in front to save egress costs than to have constructed the whole storage solution themselves.
- vidarh 7y agoPutting a caching layer in front of AWS is often very cost effective even without much skew in the access pattern. It tends to take a very low hit rate before it pays for itself.
- ph2082 7y ago1 Terabyte of hard disk cost ~50USD. 247 Petabyte ~ 247000 Terabyte > 50000 USD. Network cards, bandwidth, electricity cost > I can't guess. Couple of good engineers (hardware and software ones), which they definitely have. May be they could have built their own cloud in < ~10-15 million USD. And that won't be recurring cost. May be they missed article about Bank of America saving ~2 Billion USD, by building their own cloud.
- SEJeff 7y agoYou realize that the entire openstack project came from the opensourcing of NASAs opennebula project, right? They've got one of the biggest infiniband networks in the world underpinning it.
- ph2082 7y agoI didn't knew that. Thank you for telling. Now I am more curious why go along with AWS instead of using Openstack. Need to find some case study of openstack vs rest of cloud provider.
- duskwuff 7y agoBecause OpenStack is a piece of software, not a provider. And it's instructive to consider why none of the major cloud providers use it...
- rmrfstar 7y agoIn addition to saving money, they will also make the US more resilient by helping avoid a concentration of expertise and an infrastructure mono-culture. I suspect that ideas like this will become more popular as the US asks itself "what happened to our resilience?" [1] https://en.wikipedia.org/wiki/Self-Reliance https://en.wikipedia.org/wiki/Self-Reliance
- supdatecron 7y agoYour numbers are way off, as you didn't account for redundancy of the drives (any failure or bit flips of 1 of those 2,470 drives will cause corruption of likely the entire data set). > Network cards, bandwidth, electricity cost > I can't guess. This is where a huge amount of cost is. > And that won't be recurring cost. Maintenance, humans, cooling, drive replacements, property, building, land tax, payroll tax are all recurring costs.
- djrogers 7y agoI'm not saying this won't be a financial cluster - it likely will cost many times more than planned - but the headline here is just a flat-out lie. TFA says: "a March audit report [PDF] from NASA's Inspector General noticed EOSDIS hadn’t properly modeled what data egress charges would do to its cloudy plan." 'Hadn't properly modeled' is very different from 'forgot about'. And if you actually read the linked report, it says things like: "ESDIS officials said they plan to educate end users on accessing data stored in the cloud, including providing tools to enable them to process the data in the cloud to avoid egress charges." and "To mitigate the challenges associated with potential high egress costs when end-users access data, ESDIS plans to monitor such access and “throttle” back access to the data" Neither of those statements would be in the audit if the entire topic had been a surprise.
- tyingq 7y agoFrom that linked report... "In addition, ESDIS has yet to determine which data sets will transition to the cloud nor has it developed cost models with the benefit of operational experience and metrics for usage and egress." That sounds fairly close to the headline.
- ralusek 7y agoI wonder why they wouldn't use Wasabi: https://wasabi.com/cloud-storage-pricing/ https://wasabi.com/cloud-storage-pricing/ Looks like egress is free. Maybe because it's comparably untested? Does anyone here have any experience with it?
- alexfromapex 7y agoProbably need assurances or regulatory solutions that only a cloud giant like AWS could address
- Eikon 7y agoI wouldn't rely on that. Wasabi does not charge for egress but our pricing model is not suitable for use cases involving the hosting of videos in a manner where the ratio of egress downloads exceeds the amount of storage. https://wasabi-support.zendesk.com/hc/en-us/articles/360000451052-Can-I-use-Wasabi-to-host-my-videos- https://wasabi-support.zendesk.com/hc/en-us/articles/3600004...
- beastman82 7y agoTorrent FTW
- OzzyB 7y agoLooks like even the big boys get bitten by the Cloud Meme when forgetting about bandwidth costs; glad I'm not the only one.
- slowhand09 7y agoWow! I worked on EODSIS in 93-96. We estimated 16 petabytes, at the time it would be one of the worlds largest databases. We changed horses midstream moving our user interfaces from X-windows Motif to WWW. And built a very early Oracle DB accessible via WWW. There was no cloud then except missions studying atmospheric water vapor. When this was originally designed there were to be several (6-7) DAACs - Distributed Active Archive Centers (https://earthdata.nasa.gov/eosdis/daacs https://earthdata.nasa.gov/eosdis/daacs) to store data near where it was needed or captured. Now they have 12 and are storing on AWS. Amazon didn't exist when this was originally built.
- Wheaties466 7y agoat that point why not just use a P2P based system.
- ghostpepper 7y agoThere's a joke around here somewhere about AWS pricing being too difficult even for rocket scientists.
- leni536 7y agoApparently AWS pricing is not rocket science
- Spooky23 7y agoUsing AWS for this type of use case is dumb for an org as large as NASA, if cost savings is a goal. It's cheaper to just land capacity at a datacenter.
- toyg 7y agoI guess they have additional legal constraints that don’t allow them to just “land space” here or there - the vendor must probably be security-vetted, compliant to a hundred government-produced checklists, and willing to go through extra-long sales and support cycles. It will inevitably push up prices significantly. In fact, I can imagine ops-teams at Nasa licking their lips at the idea of doing away with a lot of that bureaucracy once they switch to AWS... note how the report mentions that some of the controllers are actual sponsors of the move: it’s obviously a conflict of interest, but it might well arise when the org as a whole is a bit too happy to steer away from a suboptimal situation. This said, AWS will rob them blind, simply because they can. Like all outsourcers (which is effectively what they are), they get in with the simplicity argument, then boil that frog up with extra charges. It’s good that somebody pointed out one of those charges, but I doubt anything will change substantially- Amazon will probably cut them a discount and that will be it. And once you’re invested in a cloud env to the tune of hundreds of petabytes, you’ll likely not switch away for decades.
- Karunamon 7y ago>..then boil that frog up with extra charges. That implies a level of dishonesty or nontransparency that AWS doesn't have. Their pricing is disclosed, up front, and they offer a calculator to model your costs out. Knowing how much data egress you're going to have is not some arcane art, NASA just plain forgot to do it. It may be complicated, but so is any workload at this size. Figuring the cost is part of due dilligence, and they've made it as straightforward as possible.
- Spooky23 7y agoThat's a half-truth. All of the cloud vendors de-empathize network egress costs. It's similar to products that depend on Microsoft licensing who will always omit those types of costs. (Oh, so you needed to spend another $500k in SQL Server Enterprise?) Many organizations lack the operational metrics to allow them to effectively measure their egress needs. And AWS/GCP/MS salesmen arent in the business of slowing down deals with awkward questions. This is especially true where an org like NASA probably contracts out things like network services. Going from a model where you make fixed capital investments to paying for the byte is difficult to measure.
- knorker 7y agoThis surely was entirely known to AWS, where they were rubbing their hands at the fact that every user of this data has to process it using EC2 on site. This is Cloud lock-in using data location.
- tehalex 7y agoI wonder if this includes or if they can use Direct Connect? [1] Cloud data transfers are too expensive, personally I assume that it costs more to measure and bill for bandwidth than the usage itself... 1: https://aws.amazon.com/directconnect/ https://aws.amazon.com/directconnect/
- angry_octet 7y agoThey could use direct connect, from each of their data centres, essentially turning AWS into a giant NAS. However this gives up the idea of using AWS compute to provide value added analysis.
- Havoc 7y agoCan't they just use the current DAACs as a caching layer? Seems like the least ugly way out of this mess. Also - can't they use torrent tech? I wouldn't mind helping out a bit on space & data
- unhammer 7y agoYOU ARE NOT AFRAID? 'Not yet. But, er...which way to the egress, please?' There was a pause. Then Death said, in a puzzled voice: ISN'T THAT A FEMALE EAGLE? I've been reading A Hat Full of Sky to my daughter these days, and there's a running joke that "supposedly intelligent people" don't know the meaning of the word "egress", mixing it up with things like egret, ogress or eagles. (See also the inspiration for the joke: https://unrealfacts.com/pt-barnum-would-trick-people-with-a-this-way-to-egress-sign/ https://unrealfacts.com/pt-barnum-would-trick-people-with-a-... )
- X6S1x6Okd1st 7y ago> NASA also knows that a torrent of petabytes is on the way. Oh that sounds like a potential solution. /s
- api 7y agoThis is exactly why the costs are set up that way. The first time I saw AWS pricing I chuckled and thought "roach motel." Data goes in but it doesn't come out. Its one of many soft lock in mechanisms cloud hosts use.
- Mave83 7y agojust build your own storage and save an incredible amount. It's hard you might think, but it's not. croit.io provides all you need to deploy a scalable cluster even on multiple geographic regions. Price for 1 PB sized cluster including everything from rack to hardware to license to labor for below 3€/TB/Month or at the Amazon Glacier price tag but with the S3-IA access.
- mister_hn 7y agoWhat about maintenance? Some forget about that... Broken drives, broken RAID, broken NAS. A 120TB SSD NAS might cost over 200k€ ..imagine a 250PB one
- RandomTisk 7y agoSeems like a poor choice. If they're getting an incredible deal with AWS, then fine, but I would be utterly shocked if most seasoned and competent IT professionals couldn't design and build a multi-region storage array for far less than Amazon will charge them.
- driverdan 7y agoAre you seriously suggesting that NASA didn't consider alternatives, like their current self-hosted solutions?
- GordonS 7y agoGiven they "forgot" about egress bandwidth costs, I think the parent's comment was fair.
- dna_polymerase 7y agoDo you want to add your contact details to your post so NASA can get in touch or what is going on here. Add a little disclaimer that you work for/are croit.io so people can instantaneously see why you would argue for the space agency of the U.S. to run their own data storage.
- mensetmanusman 7y agoThis seems like a good use of torrenting?
- maerF0x0 7y agoIt looks like they were aware of Bit torrent as recent as Oct 2011 https://web.archive.org/web/20111024223108/https://visibleearth.nasa.gov/faq.php https://web.archive.org/web/20111024223108/https://visibleea...
- caymanjim 7y agoTorrents are only helpful when there's a large number of people who download the data and are willing to share it. There's not a large userbase for the vast majority of NASA data. It wouldn't be distributed in any meaningful way.
- mensetmanusman 7y agoMaybe various world governments could bandwidth to accomplish these types of missions.
- anthonylukach 7y agoThis article seems short sighted. 1. Using the AWS cost calculator is pointless, naturally an entity the size of NASA would get heavily discounted rates. 2. As data volume grows, the complexities of working with that data expands. NASA appears to be embracing cloud computing by embracing a paradigm where scientists push computation to where the data rests rather than downloading data [1], [2], [3], thereby paying egress on only the higher order data products. 3. The report notes that NASA has tooling to rate limit and throttle access to data. This, in itself, proves that NASA didn't "[forget] about eye-watering cloudy egress costs before lift-off". People may scream about vendor lock in, which is a fair complaint; but acting like NASA just didn't think about egress is misleading. NASA is ultimately a science institution, I think diverting effort away from infrastructure management and towards studying data is likely a wise decision. [1: https://www.hec.nasa.gov/news/features/2018/cloud_computing_services.html https://www.hec.nasa.gov/news/features/2018/cloud_computing_...] [2: https://link.springer.com/article/10.1007/s10712-019-09541-z https://link.springer.com/article/10.1007/s10712-019-09541-z] [3: https://ui.adsabs.harvard.edu/abs/2017AGUFMIN21F..02P/abstract https://ui.adsabs.harvard.edu/abs/2017AGUFMIN21F..02P/abstra...]
- deleted 7y ago[deleted]
- tda 7y agoProbably there is even good competition between the cloud providers, because hosting the data means that you can sell a lot of compute time to all the users of the data. NASA choosing for AWS means that any IO intensive analysis on that data will run faster/better/cheaper on AWS.
- oh_hello 7y ago"The audit, meanwhile, suggests an increased cloud spend of around $30m a year by 2025" Isn't this a rounding error for NASA?
- tzm 7y ago$5,439,526.92 per month
- julienchastang 7y agoThis article is misleading. The entire point is to not move data out of the cloud. Instead bring your computing (analysis, visualization) to the data and pay for compute cycles on AWS. If your workflows are short/bursty, you will come out ahead. Moreover, you will be able to do big data-style computations that you cannot do in a local computing environment. This is bad journalism, IMO.
- pixelbath 7y agoUnless my numbers are way off, I got around $15.5 million per year using Backblaze's calculator: https://www.backblaze.com/b2/cloud-storage-pricing.html https://www.backblaze.com/b2/cloud-storage-pricing.html Numbers used: Initial upload: 258998272 GB (1024*1024*247) Monthly upload: 100 GB (default) Monthly delete: 5 GB (default) Monthly download: 1048576 GB (1 PB) Period of Time: 12 months (default)
- adtac 7y agoIt'll take 215,000 years to reach 247 petabytes if you averaged 100 GB of upload a month.
- bhandziuk 7y agoI think they're saying NASA would add ~100GB of new data to this dataset every month.
- adtac 7y agoI know. And I'm saying if that was the rate they've historically added data to their dataset, it would've taken them 200,000+ years to get here. Which is why 100GB/mo is virtually nothing for NASA -- it doesn't match with their historical throughput.
- bhandziuk 7y agoI see what you're saying. Yeah, I agree.
- pixelbath 7y agoThat's the default value on the calculator, but all that does is prorate the storage demand over the period. Backblaze is entirely usage-based (speaking from experience; I'm a customer).
- kylebarron 7y agoThe initial upload is 247 petabytes
- jka 7y agoWhat's the opposite of AWS Snowmobile[0]? [0] - https://aws.amazon.com/snowmobile/ https://aws.amazon.com/snowmobile/
- chickenpotpie 7y agoDownloading no data extremely fast
- deleted 7y ago[deleted]
- chx 7y agoIf you are facing similar problems you should know traffic via Cloudflare from B2 is free. I am not 100% CF would be happy if NASA picked the CF free tier but probably their quote would be magnitudes lower than Amazon's.
- movedx 7y agoIt's The Register, people. Don't take it seriously. It's practically The Onion of the IT industry, especially the comments sections. I've written two articles for them and the comments are a joke. They're all anti-Cloud, anti-progressive. Try selling them Kubernetes has a solution to their problems: they'll think you've come to steal their children. I know, I've tried. In short: this never happened. NASA didn't forget anything. It does, however, make for a great eye catching headline! Sorry to be bitter about this, but publications like The Register serve little purpose these days. It caters to a specific kind of IT personality that can't let go of their physical tin and they think public Cloud has no place or use at all. Again I know, I've tried convincing these people of such things.
- mturmon 7y agoSmartest comment so far in the thread. The issue of cloud egress has been known and worked at NASA for a decade now, and the article treats it like an OMG moment. Historically, data have been stored and processed on-premise but NASA has been migrating data and processing to the cloud where it makes sense. For instance, it makes a lot of sense to burst out to the cloud for near-real-time processing during and just after natural disasters like earthquakes and forest fires. The large missions they mention (SWOT, NISAR - big radars in Earth orbit) are drivers of the shift of more processing + data to the cloud, because they will generate an unprecedented amount of data. They are pathfinders. By percentage, very little of that data will ever egress - it's low-level and uncalibrated - so a cached strategy could be valuable. Here are some slides giving background on the SWOT/NISAR data system. They are from 2017, so more has happened in the meantime, but they touch on some of these issues: https://smd-prod.s3.amazonaws.com/science-red/s3fs-public/atoms/files/13%20-%20Day%202%20SDS%20Considerations%20for%20SWOT%20and%20NISAR%20-%20Hua.pdf https://smd-prod.s3.amazonaws.com/science-red/s3fs-public/at... Regarding the step function in data volume, see the humorous slide #4.
- turdnagel 7y agoRequester pays!
- szczepano 7y agoTo sum up no matter how big the hard drives or data center we produce we will always have problem with storage capacity.
- CKN23-ARIN 7y agoPutting a dataset into AWS is a lot like putting a satellite into orbit. You still need to pay later to get it down, or to safely destroy it.
- pontifier 7y agoCloud egress costs killed the business I'm now trying to save. I won't fall into that trap.
- yosito 7y ago> You don't need to be a rocket scientist to learn about and understand data egress costs. Which left The Register wondering how an agency capable of sending stuff into orbit or making marvelously long-lived Mars rovers could also make such a dumb mistake. I used to work very closely with this department at NASA. Without saying too much, the short answer is "tenured government employees more concerned about job security than the success of the project" is how an agency could make such dumb mistakes.