8 ms·
MIMO and nonlinear control theories are probably some of the hardest topics in all of engineering. SpaceX control system also has to compensate for the fuel mov
by dtquad 2y ago
MIMO and nonlinear control theories are probably some of the hardest topics in all of engineering. SpaceX control system also has to compensate for the fuel moving inside their rockets so the control algorithms probably involve some kind of fast numerical fluid simulation.
Another interesting thing SpaceX is doing is to use consumer-grade chips in triple redundancy configurations instead of using $100,000+ radiation-hardened aerospace/defense grade chips.
- autonomousErwin 2y agoI always thought the liquid sloshing would be one of the hardest to simulate (considering how chaotic fluid mechanics is). Interestingly, I think this caused the 2nd Falcon launch to fail (the LOX sloshing).
- JorgeGT 2y agoIt is difficult, but there are modeling approaches that work, such as VoF (https://en.wikipedia.org/wiki/Volume_of_fluid_method https://en.wikipedia.org/wiki/Volume_of_fluid_method). Basically, in addition to velocity, pressure, temperature, etc., you store an additional scalar in each cell of your computational mesh representing the liquid's volume fraction. Then, you solve an additional equation to transport that scalar.
- s1291 2y agoSolving the Navier-Stokes equations numerically in 3D is very time-consuming, even on HPC clusters, not to mention the additional modeling required for multiphase flows. Your answer implies that the solutions are obtained almost instantaneously, which is not the case.
- nothercastle 2y agoThey probably pre solved a bunch of scenarios and interpolate between them known solutions
- j_maffe 2y agoThat usually doesn't work for chaotic systems.
- KeplerBoy 2y agoMaybe the tank is just not a large hollow structure but contains fins/compartments/whatever to restrict the sloshing motion and it's not that big a contribution to the overall motion. If it's no stronger than a sudden wind gust, it's just something the controller has to be able to take care of without a heads-up.
- TowerTall 2y agoThat is how they build the tank in Formula One Racing (and probably many other race cars, I guess)
- kaonwarb 2y agoThese are indeed part of the solution and are known as baffles. They have risks of their own, e.g.: https://wccftech.com/baffling-baffles-musk-explains-why-spacexs-mars-rocket-prototype-exploded/amp/ https://wccftech.com/baffling-baffles-musk-explains-why-spac...
- nuancebydefault 2y agoIn the first spacex rocket Musk thought that it was a good idea to not install baffles. He learned from experience that they are indeed needed.
- m4rtink 2y agoI remember a very similar anecdote about Von Braun & the early Juno/Jupiter rockets - with someone pointing out issues with sloshing on a press conference & Von Braun brushing it off as insignificant. Then the next launch crashed due to slosh induced oscillation - and the one rocket after that had anti-slosh baffles. ;-)
- vocram 2y agoThat’s how tanks in race cars are made. Another solution is fill the tank with some kind of sponge-like material.
- krisoft 2y agoI think the reason these kind of simulations are fast enough is because they are very coarse and approximate. Don't think of asking how exactly the foam swirls around the individual longerons, more like a very rough estimation of which side of the tank the liquid is slumped to. Remember it doesn't have to be "exact" just close enough to be useful. By their very nature model predictive controllers operate in a world where not everything is perfectly modelled. Engineers do their best and whatever is left is the "error" the controller is trying to deal with.
- Log_out_ 2y agoOr you compute variations ahead of time and do a situation based lookup, hoing through loops if a situation ressembles another one.
- keepamovin 2y agoMaybe they don't need to model the fluid dynamics, they just need to detect the mass movement / acceleration forces caused by it, and use those sensor inputs to inform a picture that's fed into their correction thursting. Sort of like how you can balance a few pitchers of beer on a tray in your hand by remaining aware of the weight, even when people remove one! hahaha :)
- m4rtink 2y agoStill if there indeed is "free" mass moving about, you need to make sure your control inputs don't make it slosh harder, so you compensate for that, so it sloshes even harder, etc - basically avoiding oscilation. :)
- keepamovin 2y agoYeah..ah, control theory. Heh :)
- jjk166 2y agoThey don't need to solve the Navier-Stokes equations, they don't care how the fluid is actually behaving, they just need to approximate how the mass is moving within a margin of error that the control system can handle.
- WalterBright 2y agoIf the computation is too difficult, another approach is build a test stand and try methods until it works. Which is why we use wind tunnels, for example.
- JorgeGT 2y agoOh no, apologies if that was the impression I gave!! I actually perform CFD simulations in HPC clusters, and in fact I'm an admin of the small cluster at my research institute =) These are indeed heavy computations. What I meant is that VoF is one additional equation to be solved besides the N-S equations (either filtered as in LES or Reynolds-averaged as in RANS), the energy equation, your turbulence model equations, and so on. Certainly, not instantaneous at all, but simply an additional "simple" model that we can hook into our current way of doing CFD. So, my point was, sloshing is a problem that we know how to simulate, although certainly you need HPC resources. Though, looking at those 100k NVIDIA H100 Elon has, I guess they have them! :P
- edm0nd 2y agoI'm curious, how long time wise do these type of "heavy computations" take on clusters of HPCs?
- JorgeGT 2y agoIt really depends on the problem to be solved (domain size, complexity of physical phenomena such as turbulence model, heat transfer, acoustics, multiphase flows, combustion, etc., number of time steps required...). In our case we perform for instance simulations of turbomachinery acoustics that can take 3-4 weeks running in a few hundreds of CPU cores, combustion acoustics simulations that can take a week or two running in 1k-2k cores...
- swader999 2y agoWouldn't it burn most of the fuel to mitigate the effect?
- dtquad 2y agoI think some Kerbel Space Program players have attempted to approximate the liquid sloshing as an inverted single or double pendulum problem inside the rocket that the control algorithm has to take into consideration in addition to the primary control of the rocket.
- sobellian 2y agoAt least for the retro-propulsive landing burn, I think the modeling problem is probably aided by the high G-forces that must keep the fuel very close to the bottom of the tank. Even before re-light the booster is falling near terminal velocity (I think?), so the fuel is likely sitting at the bottom. I think it's a huge problem when re-lighting the engines in orbit, though.
- chasd00 2y agoIirc cold gas thrusters are used before ignition to provide some acceleration to force the fuel to the bottom of the tank.
- baq 2y agoThe technical term for that is 'ullage'. See also: https://en.wikipedia.org/wiki/Ullage_motor https://en.wikipedia.org/wiki/Ullage_motor
- m4rtink 2y agoAlso IIRC the massive main tanks in Super Heavy should be basically empty at landing & the landings propelants come from a set of small header tabks that are near the central axis of the vehicle & arr completely full. This should reduce or even fully remove sloshing issues at landing time.
- dmitshur 2y agoHas it been considered to spin the fuel via some centrifuge mechanism as a way to remove sloshing from the equation, or is that more complex/expensive/error-prone than just predicting it via simulation?
- m4rtink 2y agoI'm thinking we will eventually end up with "active fuel management" techniques like this for in space vehicles. Bug tanks make sense there & they might not be always full. So I can imagine all kind of interesting ways you can work with the fuel in zero go to avoid not only slosh but also the need for ullage thrusters. Eq. some programmable nozzles using in-tank gas to nudge liquid fuel blobs to move in the right direction. Or even some nets or bags that herd in the fuel in the middle of the tank + prevent it from directly touching the side, reducing boil-off or refrigeration requirements. :)
- phkahler 2y agoMaybe they just use pressure sensors to tell where the fluid is within the tank. Even a real-time simulation should have some measurements to self correct to some degree. Otherwise it'll diverge.
- macinjosh 2y agoThere wouldn’t be a whole lot of fuel left by the time it’s back to land so likely an irrelevant factor.
- bob1029 2y agohttps://en.wikipedia.org/wiki/Model_predictive_control https://en.wikipedia.org/wiki/Model_predictive_control http://cse.lab.imtlucca.it/~bemporad/publications/papers/ecc15-descent.pdf http://cse.lab.imtlucca.it/~bemporad/publications/papers/ecc...
- sidewndr46 2y agohow much liquid fuel is present when it lands? I assumed all the clouds it was giving off was it dumping fuel to make sure it was empty on landing
- arijo 2y agoI never thought of using fluid dynamics in the rocket stabilization algorithm—maybe it's something that could be useful to prevent many of the accidents involving liquid-transport trucks
- alexpotato 2y agoReminds me of the early days of Google File System. They used trios of regular consumer grade disks/servers etc as a cluster and it looked like a single node. They had to replace a LOT of hardware but this was still cheaper than big iron industrial grades servers.
- moffkalast 2y agoInterestingly cheap redundancy is also how life does things for the most part. Most biological organisms just replicate a lot to guarantee success, so it's clearly a good strategy and an efficient use of energy.
- szundi 2y agoEfficient as in evolution or in business: you just need to be more efficient than most of the peers
- arijo 2y ago> Another interesting thing SpaceX is doing is to use consumer-grade chips in triple redundancy configurations instead of using $100,000+ radiation-hardened aerospace/defense grade chips. This has been known in the high availability and safety systems industry for a while and a good book to learn these reliability engineering techniques is "Reliability Evaluation of Engineering Systems". The book is available on amazon: https://a.co/d/1nH824K https://a.co/d/1nH824K
- wildzzz 2y agoOne downside of using non-rad hard parts is degradation from TID (gamma) and latch up effects. You can have chips monitoring other chips to reset whenever they latch up but TID is mostly permanent. The good thing is that TID in LEO, where SpaceX mostly operates, is relatively lower than GEO so they can get by with mostly commercial parts. It's not like the big defense contractors haven't figured out the same thing, they do fly stuff using commercial parts as well, they are just slower to adopt the same culture. SpaceX and the companies that built components using commercial parts are building the new-space industry.
- christophilus 2y agoCould you surround your components with gamma-shielding materials and get away with off-the-shelf parts deeper in space?
- gafferongames 2y ago"To block gamma rays completely, you need about 13.8 feet of water, 6.6 feet of concrete, or about 1.3 feet of lead." https://www.epa.gov/radiation/radiation-basics#:~:text=Gamma%20rays%20are%20a%20radiation,be%20required%20to%20stop%20them https://www.epa.gov/radiation/radiation-basics#:~:text=Gamma....
- arijo 2y agoYou don't need to block gamma radiation completely to increase the electronics reliability :) Maybe you could improve the system availability considerably by a bit of gamma radiation protection combined with some more parallelism of the components ..
- prox 2y agoJust as a note, Space Engineers has a mod that accounts for fuel in the tanks and also various orbital mods. If one feels inclined to try it for themselves ;)
- serf 2y agoas someone who absolutely loves SE -- please don't. the orbital and planetary mechanics kind of suck. They're meant to provide a decent 'arcade realism' for the sake of player/player interactions and pvp/pve. if you want to experience fuel slosh/weight during a vertical ascent/descent go with kerbal. It models a lot of that stuff without mods -- and mods can make the model even more accurate.
- wildzzz 2y agoI spent 2 semesters covering controls and I barely felt like I learned anything.
- WalterBright 2y agoReliability is not based on a system that cannot fail. It is based on a system that can survive failure.
- e_y_ 2y agoA system (whole) that can survive the failure of (some of the) individual parts. Up to a limit.
- szundi 2y ago… or cannot fail
- deleted 2y ago[deleted]
- mrdoops 2y agoThe canonical paper on handling software failures: https://erlang.org/download/armstrong_thesis_2003.pdf https://erlang.org/download/armstrong_thesis_2003.pdf
- WalterBright 2y agoThere's much more to it than the programming language. Algorithms can be faulty as well.
- mlyle 2y ago> There's much more to it than the programming language. Which was never claimed. That paper is a little bit about Erlang and a whole lot about OTP and other methodology and design technique. It is still, very much "the paper" for distributed systems, though its applicability to this particular problem is limited.
- mlsu 2y agoI know MPC takes a LOT of compute power. It's not like a finely tuned PID loop or even a cascade of PID loops, computationally. Does anyone know (or have educated speculation on) what kind of hardware is running these algorithms? Like, do they have a linux machine that's running the control loops? Are we talking megabytes, gigabytes of SRAM? I would think no -- you would definitely need hard real time for something like this. But my only experience with real time systems is in tiny MCUs with kb of SRAM. That's definitely too small for a controller like this. Really curious about the nuts and bolts of this.
- mlyle 2y agoMPC doesn't need to take a ton of compute power. It all comes down to how sophisticated the underlying model is. You can have a MPC with 20 variables and run it at multiple kilohertz on a tiny microcontroller. When you build something like this, you're torn between having a big model that represents everything and a smaller model that is easier to validate and reason about. Based on simulation, you might go for a smaller model that "knows" to stay away from operating areas where hidden variables (like really complicated tank slosh) invalidate the small model. I doubt the actual control loop is too much processing, but it's certainly possible to build controllers with SDRAM, millions of variables of state, and hard realtime processing, though I wouldn't build it on top of preempt-rt. ;)
- bgnn 2y agoIt would be a real time OS running it for sure. Which one I don't know.
- quailfarmer 2y agoThey have publicly discussed using Linux with RT kernel!
- erikerikson 2y agoI have been told by people who worked on them that you get radiation hardened aerospace/defence grade chips by backing off the clock speed about 20% to give signal stabilization slightly longer time. I can understand the population being confused about this but industry being confused seems to have more to do with regulatory capture and beaurocratic moats which SpaceX does seem to be bypassing.
- mensetmanusman 2y agoYou also have to add massive amounts (relatively) of static sink by approaches like ‘silicon on insulator’ to prevent energetic electrons from hopping into the transistor layer.
- m4rtink 2y agoI'm sure there are many hardening processes - from the modified COTS parts, all the way to presumably much more expensive custom substrates like: https://en.wikipedia.org/wiki/Silicon_on_sapphire https://en.wikipedia.org/wiki/Silicon_on_sapphire
- blurbleblurble 2y agoPeople don't realize how powerful applied math (especially in the areas you've mentioned) has become. Same tools can be applied to people in the ad tech/social media.
- schiffern 2y ago>probably involve some kind of fast numerical fluid simulation Sometimes even a simple approach can work. On Apollo they developed (at the time cutting-edge) passive RC filter networks, to avoid the control system "exciting" the rocket at frequencies of the slosh/bend/torsion modes. https://ntrs.nasa.gov/api/citations/19700023342/downloads/19700023342.pdf https://ntrs.nasa.gov/api/citations/19700023342/downloads/19... (search for "slosh" or "shaping network")
- gcanyon 2y ago> control system also has to compensate for the fuel moving inside their rockets My stepfather worked as a programmer on the Apollo program, and the thing he always talked about as his biggest accomplishment was working on the "slosh problem" -- so yeah, props to the SpaceX team for managing that landing. And props to my stepdad for managing it on hardware that was... a billion times less capable? :-)
- kbmr 2y agoDid he get a chance to see any of the recent SpaceX accomplishments?
- gcanyon 2y agoNope, died fifteen years ago.
- falconertc 2y agoI might be misremembering but I think slosh was the failure cause for one of the three failed Falcon 1 flights. It was number 11 out of a pre-flight list of top 10 most likely failure scenarios. Definitely a difficult problem.
- 3abiton 2y ago> MIMO and nonlinear control theories are probably some of the hardest topics in all of engineering. I am curious but clueless about these problems, can you expand more?
- mataslauzadis 2y agoGoogle is a good place to start. https://en.wikibooks.org/wiki/Control_Systems/MIMO_Systems https://en.wikibooks.org/wiki/Control_Systems/MIMO_Systems
- restalis 2y ago"instead of using $100,000+ radiation-hardened aerospace/defense grade chips" Well, that makes perfect sense considering that both the spaceport infrastructure, and the booster need to do their calculation on the ground level instead of the highly radiated environment that is space. However, for the rockets themselves, which happen to reach that harsh environment, they may use more resilient and expensive hardware in the future, after passing over the current "let it splash in the Indian Ocean" development and testing phases.
- debit-freak 2y ago> SpaceX control system also has to compensate for the fuel moving inside their rockets so the control algorithms probably involve some kind of fast numerical fluid simulation. Surely this isn't necessary with a small enough sensor granularity or whatever the terminology is. You can have very dumb software if it reacts quickly enough to changes in perception.
- noahbp 2y agoI'm not an engineer, but is there a reason why they can't french press (sans filter) the fuel toward the bottom of the tank as it empties?
- e_y_ 2y ago(Not an engineer either) My understanding is that it's been done before on smaller scales but having a giant piston in the tank requires a good seal, railings to keep it straight, and overall way too much mass and rigidity. Consider that the tank walls are only a few millimeters thick. Instead, it's more common to use gasses injected at the top of the tank to push the liquid to the bottom. Falcon 9 uses helium. Starship uses https://en.wikipedia.org/wiki/Autogenous_pressurization https://en.wikipedia.org/wiki/Autogenous_pressurization as well as small header tanks for the landing propellants.