13 ms·
We should train AI in space [pdf]
- taspeotis 2y agoCompany that stands to make money from people sending stuff to space suggests people send stuff to space; news at 11.
- yieldcrv 2y agono conflict == no interest
- slyall 2y agoI saw a few posts about this and thought it was a joke. This tweet sums it up: YC company announces a giant data center in space Startup Twitter: Makes so much sense, it will be so easy to cool in space! Space Twitter: How the hell are they proposing to keep it cool?? https://x.com/mouthofmorrison/status/1831680658927669632 https://x.com/mouthofmorrison/status/1831680658927669632
- CobrastanJorji 2y agoWait, was this not a joke? I thought this was one of those "let's realistically think through the implications of trying to live on a space tortoise" kind of papers.
- throwup238 2y agoBut their video is so pretty! https://www.lumenorbit.com/ https://www.lumenorbit.com/ > As launch costs fall, orbital data centers will leverage 24/7 solar energy and _passive cooling_, rapidly deploying to gigawatt-scale, avoiding permitting constraints on Earth. It sounds like they want to use copper heatspreaders and radiators to dump gigawatts into space. From those little pods connected to the spine... without any active cooling components Oh dear.
- hollerith 2y agoWhat manner of active cooling do you imagine might work in vacuum? Fans?
- geysersam 2y agoPipes with a coolant?
- deleted 2y ago[deleted]
- dkbrk 2y agoYou use radiation for cooling in space. That obeys the Stefan-Boltzmann law with power scaling with T^4 (so you want the radiators as hot as possible). You can use passive elements, like heat-pipes to move heat to the radiators, but active elements like pumps for forced convection could make sense; and most importantly heat pumps are an active element that can boost the temperature of the radiators vs the thing you're keeping cool (thereby increasing the heat rejection capacity of a given radiator size).
- hollerith 2y agoThanks.
- deleted 2y ago[deleted]
- atoav 2y agoPicture of a radiator on the ISS in this thread: https://physics.stackexchange.com/questions/149832/cooling-a-satellite https://physics.stackexchange.com/questions/149832/cooling-a...
- hollerith 2y agoThanks. Since (like the answer by a "Fluid/Thermal Engineer" with a PhD says) the ISS uses mechanically-pumped fluid loops to transfer the heat to radiators, certainly a server farm in space (with its much higher energy density) should, too, rather than relying on a simpler, but less efficient passive-cooling design in which heat gradients cause the convection or movement of the fluid -- and ISTR reading that heat gradients don't even cause convection in the absence of gravity.
- causality0 2y agoSo did these people get into YC with a smart idea which they dropped in favor of this idiotic one, or did all the smart people at YC call in sick the day this company was accepted? Because dayum.
- ASalazarMX 2y ago"It was not impossible to build the Space Data Center (TM) in the vacuum of space. It was impossible to build it anywhere else."
- bbor 2y agoData centers in space use 24/7 solar power Maybe I’m just missing something obvious, but how exactly do you get 24/7 solar coverage in space? Surely they don’t propose launching all this stuff beyond LEO…? EDIT: they cover it later on; if you think about it you can orbit on the exact line that is perpendicular to both the equator and the direction of the sun, and then have the orbit progress perfectly in sync with the earth’s orbit (`365.25/360`° of longitude drift per day) so that it’s always on this special orbit. So it’s like GEO but (presumably…?) way harder to maintain, and slightly more exclusive (how many could you safely pack in a constellation?). Not to mention the need to account for all the other LEO satellites to dodge this 4km wide square, and the much higher (presumably??) risk of debris/dust collisions. This seems like a hat on top of a hat on top of a thousand hats, but Godspeed you mad geniuses. I would love to see this some day, perhaps if/when AGI trivializes some of the problems involved! PS if my napkin math is right (`tan(4/800)*60` = 0.005 arc minutes) it wouldn’t be visible to the human eye at typical LEO altitudes, a threshold that Wikipedia puts around 0.5 arc minutes. We can pump those numbers up! As-is it would certainly be a sight to behold every day at dawn and dusk, that’s for sure…
- reaperman 2y agoThey specify on that page a LEO-SSO (Low Earth Orbit Sun Synchronous Orbit). https://en.wikipedia.org/wiki/Sun-synchronous_orbit https://en.wikipedia.org/wiki/Sun-synchronous_orbit
- quaintdev 2y agoNice idea. Why not build the data center for general use in space as well? The companies that build data center certainly have enough resources to pull off something like this. What's stopping them?
- 317070 2y agoI imagine the requirements. For picking a location to train AI, the biggest constraint is legal, followed by cost of energy. For most regular datacenters, you want to run servers, and so you want low latency (proximity), sometimes high bandwidth, and reliability. Neither are really a factor for AI training. Say you have a datacenter which is offline for 15 seconds every 30 minutes, which only does gigabits per second with 100ms+ latencies. What else can you do with that? For training AI though, these are just fine. So, training AI is a type of compute that would benefit from space, especially for that legal constraint. (Not sure how they will cool the datacenter though)
- atoav 2y agoWhat are you talking about? The location of the data center is irrelevant unless you plan to train it on data that it creates itself and that data stays in space. If you want it to interface with the world, someone on ground has jurisdiction. I imagine if you run the multiple ground stations needed to maintain a data connection to an orbit that has 24/7 solar, those ground stations will be required to follow the jurisdiction of the (different) countries they are located at. Congrats you just multiplied your legal costs. Why do so many ideas nowadays radiate the energy of edgy weed smoking teenagers trying to game the system with half-baked ideas? Did Elon make this popular?
- 317070 2y agoFor clarity, the problem is legal _constraints_, not legal _costs_. A common one is that you can have a license to use a piece of data, also for training, but that license is only valid in a certain location. That is the kind of legal constraint that currently determines where large training runs happen, and indirectly, where datacenters are built. > Did Elon make this popular? I'm assuming good faith on the writers of the original article, based on my experience in the area. That is not something that was popularised by Elon Musk, as far as I am aware.
- reaperman 2y ago> Table 1: Cost comparison of a single 40MW cluster operated for 10 years in space vs on land… > Cooling (water usage): Land: 1.7m tons vs. Space: None They completely ignore the cost of 40MW of radiative cooling in the “Cost comparison” table. Then they have a section saying “we’re inventing something to dissipate gigawatts of energy into space.” So the cost comparison is entirely misleading, and that critical lie by omission makes it very hard to take them at face value about literally anything else.
- atoav 2y agoYeah and there are other things here. A terrestrial data center can be maintained. That is kinda hard in space. So if they just assume things run till they die, that space data center is a totally different kind of calculation compared to a terrestrial one. And that just scratches the first inches of the massive underbelly of this ice berg of hidden costs.
- kristjansson 2y agoNice, should also solve global warming when the radiators they propose to dissipate a GW of waste heat blot out the sun.
- FeepingCreature 2y ago> radiating towards deep space Literally ctrl-f heat in the pdf.
- sunbum 2y ago> radiators block out the sun. Nothing to do where they radiate but about the fact you need an absurd amount of them.
- FeepingCreature 2y agoOh I thought you meant outshine the sun from radiation. :) Um, isn't that a secondary benefit? They'd also act as solar shades.
- atoav 2y agoYeah, solar shades that are meant to cool the spacecraft, which is essentially a oven in need cooling already.
- atoav 2y agoYeah another poster did the back of the napkin math to find the area these radiators would have to have: twice the size of the pentagon. Launch cost: 30 billions. This kind of pdf is what you get when your genius startup doesn't have a single engineer with half a brain and everybody does enough coke to believe you can beat physics with sheer enthusiasm. That, or it is calculated fraud.
- reaperman 2y agoNote: In my napkin math I forgot they could maybe use both the front and the back of the panel so it would be “merely” the area of one pentagon. I’ve edited that. But the launch cost would remain the same as my original estimate because it was based on weight per “effective radiator area” of current best-practice space materials.
- phreeza 2y agoMaybe they can launch with SpinLaunch.
- TheAceOfHearts 2y agoLast month John Carmack was also suggesting that the idea of chip manufacturing in space might be worth exploring [0]. It would be interesting to hear from any experts if the idea has any merit; perhaps it could help unlock new manufacturing techniques? I only skimmed the linked PDF but did they write anything about the risks or challenges of deploying consumer hardware into space? Is space-hardened hardware not required? [0] https://x.com/ID_AA_Carmack/status/1821689022881636554 https://x.com/ID_AA_Carmack/status/1821689022881636554
- RockRobotRock 2y agoReminds me of the space exploration mod in factorio
- atoav 2y agoYou give them way too much credit I am afraid. They mention radiation shielding in their cost table. But that cost table omits huge costs. E.g. like the passive radiators the size of a small city they would need. Communications, Ground stations, RND, ... I think you could literally give this to a bunch of teenage nerds for an afternoon and get a pdf with more substance.
- atoav 2y ago> Cooling not required [in space] Either they are stupid or lying. Both doesn't make them look good. Also omitting costs like: - RND. You are unlikely to use off the shelve parts for cooling (radiators?), communications, fire safety, thrusters, you are basically building a satellite for a thing nobody has done before - maintenance mechanisms. You can't just send a guy to replace the radiation hardened SSD and if you do, congrats you have to build a space station. You could just let it die, but then the math will math different in comparison to a terrestrial data center - operational costs of satellite control personal and ground stations. Crunching AI in space is nice, but last time I checked the state of AI it needed tons of training data. Have fun paying for that connection into a orbit that has 24/7 solar, as earth is spinning below. Radio stations across the globe. And these are just unmentioned costs a non-space self-thought tech person came up with in 5 minutes. Don't invest into people this incompetent or malicicious.
- owenpalmer 2y agoLots of funny comments here, but does anyone actually have a good argument against this idea? It seems both ridiculous and obvious.
- yorick 2y agoTo passively radiate a gigawatt of heat in space at 100C, you'd need a radiator with a (visible) surface area of 1 million square meters.
- DiscourseFan 2y agoso you're saying its possible
- reaperman 2y agoNot for “$5 million in launch costs” per 40MW compute unit. Sure, many things are possible with enough money, but their “Table 1” is a pretty bald-faced lie. Just the weight of 40MW’s worth of H100 GPU’s alone would be $200 million in launch costs on a Falcon Heavy. That’s before you even add on launch costs for solar panels or “magic” not-invented-yet radiator panels. The first question isn’t whether it’s “possible” with enough money, it’s whether it could ever be within even an order of magnitude of “profitable”. The second question is why would you trust a company who provides these kinds of estimates with no reasonable explanation for these obvious massive discrepancies?
- atoav 2y ago1000000 m2 are a 1x1 km square. If we assume a 5mm copper sheet of that size it weighs 44512900 kg. A kg in a heavy falcon costs ~1500 USD. So we land at 66 billion USD in launch cost for the copper alone. I probably don't have to explain that building a radiator of that size in space isn't free either. And the stuff that gets the heat to those radiatoes is neither free nor lightweight either. Yet cooling somehow costs them zero dollars.
- Veedrac 2y agoIt's one of those ‘Is a space elevator a good idea?’ vibe questions. In theory, sure, inevitably civilization does this in passing if it continues on pace for another hundred or so years, because at minimum at some point you've already tiled the Earth and automation has brought cost of remote handling down to zero. But it's also, the proposal is to build a 4km x 4km array in space, and the main reason to do this now rather than say a data center in the Sahara is permitting, skipping storage, and solar utilization. The last two are rather silly reasons when the cost of solar panels is taking a nosedive and cheap energy enables dumb storage solutions, and permitting, well, let's just take as given that it's right and permitting really is that hard. That still only gives you an advantage after you scale to the point your permits would be denied, and whereas a Falcon 1 didn't really have competition so could get away with cost inefficiency, a mini compute cluster in space is competing against a terrestrial server rack. None of the arguments seem physically implausible. Launch can get to <$5m/Starship. You can build 20km² of solar and radiation. You can hook up 5 GW of compute with remote space vehicles. It's just really shockingly hard in an Apollo sort of way. Somehow I doubt the permitting will be easy either.
- extheat 2y agoNot talked about beyond heat is maintenance. When things fail, who are you going to send up to fix things? Is it never going to fail, or are they planning to throw away big components or are there going to need to be servicing missions?
- mosquitobiten 2y agoany rack that doesn't perform well gets spaced out:) , that will keep the AI on it's ... uhm, toes Most likely they'll just have a separate system to disconnect troublesome racks, once some treshhold is achieved the cluster will be replaced.
- atoav 2y agoThe answer is: they are either not competent enough to have had that thought or malicious enough to omit it. Neither would want me to invest money into their idea.
- Veedrac 2y agoThere is literally a header in the pdf titled ‘Maintenance’ that answers these questions.
- petabyt 2y agoKeeping cool in a vacuum would be easy, just stick a bunch of fans on it to blow the heat away :D
- ramraj07 2y agoNASA (used to) host a Space Settlement design competition for high school students, I submitted and won when I was in school. I am pretty sure I and other winners wrote proposals with fewer gaping scientific holes than this lol. Apparently that gets funding now.
- nxobject 2y agoJust imagine being the people judging that competition… a group of people with a pile of postgrad degrees between them, shaking their heads and going “kids think of the darndest things…”
- ramraj07 2y agoI went back and read mine and other winners and they’re really not that bad. Anything that’s not really standard idea is often very creative and interesting.
- ein0p 2y agoYeah, let’s put that megawatt grade data center into a perfect thermos, that’s the ticket
- DiscourseFan 2y agoThis is really stupid. All the resources going into this could be used for a million better things. But... Its pretty cool, and I wouldn't want to discourage anyone from thinking about the possibility of actualizing their wildest imaginations. Just, you know, think bigger.
- d00pl3d1 2y agoHow many people are required in the station to maintain the basics of operation?
- kkfx 2y agoRadiation and micrometeorites would wipe quickly very expensive systems. Not doable at sustainable costs.
- mike_hearn 2y agoHere's some thoughts trying to steelman this idea. Firstly, let's note that the founding team is certainly not inexperienced. One of them has worked at both SpaceX and Microsoft on the datacenter side, one claims to have 10 years of experience designing satellites at AirBus and he has a PhD in materials science. And the CEO has a business background mostly but also worked on US national security satellite projects (albeit, at McKinsey). They make a big deal of it being about AI training but it would seem like inference is a much better target to aim for. Training clusters are much harder than inference to build, the hardware obsoletes quicker, they have much higher inter-node connectivity, you need ultra-high bandwidth access to massive datasets and you therefore need a large cluster before anything is useful at all. Inference is a much easier problem. Nodes can work independently, the bandwidth needs are minimal and latency hardly matters either as there is an ever increasing number of customers who use inference in batch jobs. Offloading inference to the sky means the GPUs and power loads that do remain on earth can be fully dedicated to training instead of serving, which dodges power constraints on land for a while longer (potentially for as long as needed, as we don't know training will continue to scale up its compute needs whereas we can be fairly confident inference will). If you target inference then instead of needing square kilometers of solar and radiator you can get away with a constellation of much smaller craft that scale up horizontally instead of vertically. Component failures are also handled easily, just like for any other satellite. Just stop sending requests to those units and deorbit them when enough internal components have failed. Most GPU failures in the datacenter are silicon or transceiver failures due to high levels of thermal cycling anyway, and if you focus on batch inference you can keep the thermal load very steady by using buffering on the ground to smooth out submission spikes. Ionising radiation isn't necessarily a problem. As they note, AI is non-deterministic anyway and software architectures are designed to resilient to transient computation errors. Only the deterministic parts like regular CPUs need to be rad-hardened. Because you aren't going to maintain the components anyway you get possibilities that wouldn't make sense on earth, A focus on inference has yet another advantage w.r.t. heat management. For training right now the only game in town is either TPUs (not available to be put on a satellite) or Nvidia GPUs (designed for unlimited power/cooling availability). For inference though there is a wider range of chips available, some of which are designed for mobile use cases where energy and thermal efficiency is prime. You could even design your own ASICs that trade off latency vs energy usage to reduce your energy/cooling needs in space. Finally, although heat management in space is classically dealt with using radiators, if launch costs get really low you could potentially consider alternative approaches like droplet radiators or by concentrating heat into physical materials that are then ejected over the oceans. Because the satellites are unmanned it opens up the possibility of using dangerous materials for cooling that wouldn't normally be reasonable, like hydrogen or liquid sodium. This would mean regular "recooling runs" but if launch costs get low enough, maybe that's actually feasible to imagine.