16 ms·
SpaceX avoided a Russian jamming attack in Ukraine
- Thaxll 4y agoGovernment discovers modern CI/CD.
- mabbo 4y ago> “The next day [after reports about the Russian jamming effort hit the media], Starlink had slung a line of code and fixed it,” The notion that SpaceX is basically using a Full CI/CD pipeline on a massive satellite constellation is kind of amazing. Meanwhile most large government software requires months of contract negotiation before anyone is even hired to fix the problem.
- pc86 4y agoNot to discount how great SpaceX is but these are two completely unrelated things. Do you want the government to hire contractors without contracts?
- drc500free 4y agoI want the government to have the in-house expertise to just do things, especially things that can't be specified in a contract by a KO making $53k per year with no real-life experience.
- ajmurmann 4y agoI've read (need to dig up the source again) that high infrastructure costs in the US are in part due to the same problem. The government doesn't even have experts anymore who could write the request to take offers on. Maybe in some cases a slightly larger government can be a leaner government.
- ragebol 4y agoI think the issue with a lot of governments is: expertise and competence is expensive. If you don't want to pay for that because the government is then expensive, fine, but eventually you'll have an incompetent government that is slow and still expensive due to the mishaps it leads to, for itself and it's constituents.
- soco 4y agoSo option 1 is expensive and gets stuff done, and option 2 is expensive and shitty. And we have to go the way of option 2 because...
- null0ranje 4y agoCongress, mostly.
- ragebol 4y agoWell, option 2 is cheaper at first. Then everything breaks down eventually and that is costly, but later. Kind of like technical debt, but for a whole government. Also, there will be hell from the public when the govt. makes a mistake. So, there's all sorts of bureaucracy, checklists etc, to keep that from happening or to at least have people accountable. Which also slows stuff down.
- namelessoracle 4y agoThe idea that the federal government cant hire "competent engineers" is kind of odd to me. Federal government can pay enough to get software engineers. FAANG salaries are an outlier. They could easily target some midwestern state, put the headquarters for federal software agency there, find a feeder school, and get employees fed to an agency making a comfortable 6 figures in a low cost of living area. Will there be some attrition? Sure. But plenty will stay cause thats where family is, and where they have their house, and thats where there kids go to school and etc. I can guarantee you that the contractors they hire from Accenture and IBM are using devs that arent making FAANG salaries. Its kind of a meme how bad some of the government contractors are, its pretty well known you can just be a body that sits at a desk if you can get a security clearance and make easy money just manually running an SQL script once a week on some of these government contracts. The real reason nothing gets fixed is because the federal workers get free salaries and benefits packages that far outstrip their value that oversee these projects, and the contracting firms get easy money.
- thorin 4y agoObviously this used to the the case in most developed countries, but then more and more stuff started getting outsourced and pushed into the private sector to drive shareholder value. Suddenly, space, research, military, healthcare, education are all getting outsourced, so we can make more of a profit off it.
- soco 4y agoDefine "we".
- thorin 4y agoOK "they", but remember if you have a pension or any investments, you're also in on it!
- throwaway09223 4y agoThere are other problems. Currently, the US Federal government cannot hire most competent software engineers at all due to salary/comp caps. The maximum amount any federal employee can earn is $170k, with zero equity.
- thorin 4y agoAgreed, but this is largely an American problem. In Europe and where I am in the UK, salary isn't the problem, but there is still a drive to outsource. When I've been a contract developer, I've probably been making more than board members in some countries, but that seemed to be accepted over the short term. There's also an issue where enterprise companies wouldn't trust their IT departments, for historical reasons so the good people left or they got consulting businesses in and that made everything worse...
- HWR_14 4y agoWhile it's true that the government (obviously) cannot supply equity and is capped at 174k (by law capped at a senators salary), they can try to make it up other ways. Providing free room and board (or funds for the same), paying higher education costs and providing free healthcare. Also, allowing those benefits (and a healthy percentage of your income salary) to continue into retirement. There is also the fact that they have more take home pay then you might expect, because government salaries are exempt from FICA taxes
- jvanderbot 4y agoYou say that, but some senator will make his career talking about the 300k salaries paid to these teams and their managers during times when their expertise is not saving lives in a war zone. Now I agree the airforce or some branch could do this, but that would be a military intervention.
- vsareto 4y agoTech's churn and rapid rate of change prevent a slow government from making solid long-term choices in technologies that are both reliable and cutting edge. You used to hear a lot about how you'd become un-hireable if you didn't keep learning new things (not so much these days), so there's not a great incentive for experts to stick with older technologies and build years-to-decades long solutions.
- jvanderbot 4y agoI'm responsible for hiring exactly the kind of engineers we're talking about. We can't pay as much on gov dime, and equity is nonexistent, but we have some of the best, domestic engineering teams around doing amazing things. Our people go on to work at (eg) SpaceX and to found exactly the kind of startups that we're all dog whistling about. (Lars Blackmore?). Lean, focused, fast and providing valuable services to gov and private industries. This discussion about how 'the government' cannot 'do this' is a vague rorschach blot that reveals people's own motivations only, and ignores the completely different priorities gov has. Do you believe the deep technical expertise that gov-funded experts are surely providing to Ukraine would be posted on Twitter?
- shadowgovt 4y ago> We can't pay as much on gov dime It's just taxes, we definitely could. The fact that we don't has a lot more to do with government culture and a post-Reagan idea that the government should be a thin and highly reactive operation than the money not being there.
- mywittyname 4y agoI know several people working as engineers for government contractors and they all make well over the top of the GS Pay Scale, and that's before other benefits like profit sharing. Even if Congress corrected for that, they'd still be locked into slow moving pay raises, and they'd probably become a political target at some point, when some anti-spending congressperson begins harping on how there are so many federal employees make a quarter million dollars a year. Being a military contractor, for better or worse, inoculates you from a lot of politics. So, yes, the government could pay competitively, but I would be leery about accepting such a position.
- madamelic 4y agoContract =/= gov't. The way to make money working for the gov't is to get an established company to price gouge them through contracts so you can make a living wage. The gov't is wasting a huge amount of money paying a 100% premium for engineers because they aren't allowed to pay what the market demands. A "data entry specialist" and PhD scientist are paid off the same pay scale at different levels, which is great for equality but kind of difficult when you are trying to pay $31k (which is what someone with a BS in CompSci would get, GS-5) for someone who is going for $150k. If you have experience or a master's, they'll pay you $51k. It's absurdist. The GS scale caps out at $143k at GS-15 and requires a PhD and probably decades of work.
- IG_Semmelweiss 4y agoThe government fields a very well trained, powerful military. This is arguably within the mandate of government. Provide for the people's defense. However, the same military has been unable to complete its stated objectives in Iraq, Syria, Afghanistan, etc. 10 times out of 10, the situation has remained exactly as it was before, or worse. If the military fails at a core mandate that we mostly agree to be a reasonable one... what makes you think they could maintain complex code?
- greedo 4y agoI would argue that for the military to complete the stated objectives (which shift almost as much as a software project often does), is far more complex than maintaining complex code.
- fifticon 4y agoThat is a false argument. Usually US military solves their military targets quite effectively. The failing parts are usually tied to the flawed logic in the political goals they are activated to accomplish. I say this as a foreigner observing them for 50 years. The US military cannot compensate for flawed US political goals, eg bombing some democracy into appearing. And they dont set those goals. E.g. the balkan conflict in the 90s was not handled until the US bombed the serbs, and it worked.
- chasd00 4y ago> However, the same military has been unable to complete its stated objectives in Iraq, Syria, Afghanistan the military is under civilian control. The US military is very very good at killing people and breaking things. I have no doubt, given the orders, the US military could kill everyone and break everything they've been ordered to kill and break. Politicians direct the military, if objectives were failed to be met then politicians determined those objectives were no longer politically viable.
- dylan604 4y agoNo equity, but gov't could offer tax free jobs. That's like at least an instant 20% bump in salary.
- edgyquant 4y agoIMO the government shouldn’t be doing most things in house, they should have a handful of regulated contractors who they can delegate these things to.
- mabbo 4y agoI want solutions to problems quickly. The current system does not do that. You're presenting a false dichotomy, or lacking in imagination.
- pc86 4y agoI'm responding to the comment.
- whoisthemachine 4y agoI'm sure a contract can be developed that requires support for rapid response to such events.
- brianwawok 4y agoWill it be run by IBM or Deloitte?
- thedougd 4y agoDeloitte will collect the requirements. IBM will implement.
- chasd00 4y agoand then Accenture will have to fix their screwups and deal with an unhappy/angry client and repair that whole nest of relationships. sigh
- formerkrogemp 4y agoMcKinsey will explain the clustermess.
- randall 4y agoI think the issue was bureaucracy slowing down the ability to adapt, and using contracts as one example of dramatic buraucracy.
- afar 4y agoThere's a lot more to it than just contracts and contractors. There's a good book that talks about improving the military's ability to respond (The Kill Chain). The general point is that the face of war has evolved but the U.S. military's primary method of changing direction and decentralizing decisions (The Kill Chain) has not. Worth a read.
- danbruc 4y agoBut this also cuts both ways - really nice if it works but you will also look pretty stupid if you accidentally brick an entire satellite constellation or some malicious actor uses this capability against you. But I obviously have absolutely no clue what they are actually doing, so I will abstain from any judgement.
- jdrc 4y agoWhere is that pipeline described?
- jve 4y agohttps://www.reddit.com/r/spacex/comments/ncj4vz/comment/gy8yqsp/?utm_source=share&utm_medium=web2x&context=3 https://www.reddit.com/r/spacex/comments/ncj4vz/comment/gy8y...
- segmondy 4y ago"slung a line of code and fixed it" a vector of attack waiting to be exploited. A news article could read and the enemy, "slung a line of code and bricked it"
- p1mrx 4y agoDesigning an update mechanism with no remote exploits is straightforward, cryptographically speaking. The hard part is implementing it correctly.
- stickfigure 4y agoEvery functioning satellite orbiting the planet has the technical ability to update code remotely. That's a given.
- tyingq 4y agoIs it clear that it was a code fix for the satellite itself, versus the ground terminals?
- chasd00 4y agoin my experience dealing with governments (and i have a lot since the pandemic started) there's months of negotiation to make a decision that there _may_ be a problem in the first place. The next months are spent on identifying the person who will make the decision that there is an actual problem. Then they'll start negotiation on how to define the process that will pick the team responsible for qualifying the team responsible for fixing the problem. Then your "months of negotiation" start. edit: i'm being a bit dramatic but it's pretty bad. no one will make a decision because a culture of CYA means whoever dared raised their hand is the one on the cross if things go wrong.
- smcl 4y agoYou can celebrate SpaceX correcting their issue quickly, without also inventing a reality where the same issue wasn't fixable in a timely manner due to some worst-case-scenario government contract hell.
- mechanical_bear 4y agoWhat is your background with government contracting? This isn’t too far from reality.
- smcl 4y agoI have a solid background in Our Reality, where it didn’t happen. There’s no sense making something up just to get mad at it
- yellow_lead 4y agoThe director of electronic warfare for the Office of the Secretary of Defense said as much > The government, on the other hand, has a “significant timeline to make those types of corrections” as it muddles through analyses of what happened, decides how to fix it and gets a contract in place for the fix. > “We need to be able to have that agility,” Tremper said.
- nexuist 4y agoWorst case scenario? This is the most average case scenario you could hope for.
- mabbo 4y agoThe article literally has the government officials saying this would take months and contracts. That's the whole point of the article. > The government, on the other hand, has a “significant timeline to make those types of corrections” as it muddles through analyses of what happened, decides how to fix it and gets a contract in place for the fix. Or did you not even read it, but decided to come comment on what you presumed it said?
- ViViDboarder 4y agoGenerally there are contractors employed in some fashion at all times on a broad enough contract who would be available to jump on something like this. Whether pen or they’d get the approval to ship it quickly is another story.
- bcx 4y agoI think it's interesting to summarize this thread as: - Government can't offer competitive pay due to politics. - Government contracts big projects to 3rd parties who pay competitively, but lack the transparency that government is required to have. In a way it would seem to be that government contractor organizations are acting as an abstraction, and are in essence a way of government allocating funds without needing to provide the transparency and accountability of doing things in house. In essence, if you follow this line --> contractors are basically government. (i.e. Working for SpaceX is basically working for government without the page/scale political challenges of getting stuff done, SpaceX is more or less funded by Gov) Perhaps things are working how they should.
- whoisthemachine 4y agoI had never considered this before - but it seems to me that lean development methodologies and CI/CD (and other DevOps practices) will become a defense priority, at least a priority of the nations that effectively wage war.
- bezospen15 4y agoBetter hope not, you've seen how much bloat it has caused in most organizations. It's trash
- ivan_gammel 4y agoTBH such response does not require CD, only a proper support process where critical fixes can be delivered within hours or minutes. CD is for delivering everything this way, which is not always necessary from business perspective.
- brianwawok 4y agoCI/CD says that you can deliver any given bug fix in a few minutes. It doesn’t mean you use it. But IMO most people should.
- ivan_gammel 4y agoCI and CD are different things: you can use CI without CD, e.g. when you have manual process of deployment (if CD is continuous deployment) or long waterfall-like development cycles (if CD is continuous delivery). In highly secure environments CD may not be possible at all, because of formal processes of verification and manual processes of deployment (e.g. I once worked in a project, where deployment process involved copying build artifacts from CI server to an authorized flash drive and papers signed by customer).
- brianwawok 4y agoFair.
- 4y ago
- quirkot 4y agotl;dr : it's not an EMP and it's not a nuke
- totetsu 4y agoBut what was the attack and the change?
- notpachet 4y agoProbably much more banal than the article leads us to believe. Russia probably began jamming on <frequency x>, Starlink deployed a change to fail over the terminals to broadcast on <frequency y>. Would be my guess.
- Symmetry 4y agoProbably a bit more sophisticated than that involving patterns of frequency changing at certain intervals, but essentially.
- stefan_ 4y agoMy naive assumption was that since the Starlink terminals use phased arrays for beamforming, they were much less susceptible to brute force frequency jamming to begin with. Is there some truth to that?
- simne 4y agoAs I understand, it is prohibitively hard to make constant signal with high enough power, so SL will not work. So usually made trick - EW find when sync should be, and emit short high energy impulses, so snr become too low to receive sync part, and transmission effectively blocked. Spacex implemented new encoding, where sync is in random place in time-frequency, so now EW could emit, and this will lead to speed degradation, but not to full blocking.
- thrwaway9871 4y agoHow did it shut it down? What setbacks did it induce to Russians? The headline reads like sensationalized clickbait for satelites changing service frequencies.
- nexuist 4y agoIt's a war...they're not going to publicize exact methodology or characteristics used for national security purposes.
- deleted 4y ago[deleted]
- cyberpanther 4y agoThis is the kind of stuff we're building at https://www.cognitivespace.com/ https://www.cognitivespace.com/. If you're an engineer or developer and want to build this stuff. Come join us.
- simne 4y agoDo you accept remote workers from Ukraine? :)
- sschueller 4y agoAlthough his was a defensive move by SpaceX we have to be careful private industry doesn't pull us into WWIII. We are not sending heavy armament to Ukraine for a reason and the government needs to make sure that their policies also apply to companies like SpaceX. By the way how will we "feel" when the Chinese or some Russian billionaire launches their own "Starlink" which will also be operating over the US? From what I have read there are several such systems in the pipeline in China.
- plehoux 4y agoIf a private corporation releasing an update to its code base to allow its own service to keep working brings us into WWIII... the root cause of the WWIII is definitely not the code update.
- blenderdt 4y agoAnd that is the point. Elon sent starlink terminals to the Ukraine because of the war.
- pmorici 4y agoThey had been trying to launch Starlink service there since before the war and the government was dragging their feet on regulatory approval. The war caused the government to rethink that.
- blenderdt 4y agoRethink as in: "@elonmusk , while you try to colonize Mars — Russia try to occupy Ukraine! While your rockets successfully land from space — Russian rockets attack Ukrainian civil people! We ask you to provide Ukraine with Starlink stations and to address sane Russians to stand." https://mobile.twitter.com/elonmusk/status/1497701484003213317 https://mobile.twitter.com/elonmusk/status/14977014840032133...
- LeifCarrotson 4y agoRoot cause analysis in software postmortems and in business is important, but knowing the root cause that initiated WWIII is only really useful to understand in hopes of preventing WWIV. It's irrelevant whether a private corporation software update or the assassination of Archduke Ferdinand causes a world war to break out, it must be avoided.
- JaimeThompson 4y agoMarch 25, 2022 "Starlink, at least so far, has resisted all hacking & jamming attempts" It even more impressive that SpaceX was able to update it that quickly as well as a huge selling point for the military markets they are targeting so I can't quite understand why Musk has to make such a false statement. [1] https://twitter.com/elonmusk/status/1507505633259630599 https://twitter.com/elonmusk/status/1507505633259630599 Edited text below this line "Some Starlink terminals near conflict areas were being jammed for several hours at a time. Our latest software update bypasses the jamming." https://twitter.com/elonmusk/status/1500026380704178178 https://twitter.com/elonmusk/status/1500026380704178178
- fallingknife 4y agoSource that this is false?
- JaimeThompson 4y agoPlease see the edited post for the source, it was a Musk tweet.
- politician 4y agoCould you please explain why you believe that’s a false statement?
- JaimeThompson 4y agoI updated the original post with Musk's original tweet which I should have done originally.
- causality0 4y agoThis article uses incorrect terminology. What the Russians did was electronic warfare, not electromagnetic warfare. Electronic warfare jams, blocks, or otherwise disrupts electronic communication. Electromagnetic warfare damages or destroys communication equipment.
- peteradio 4y agoAny source for this, common search shows them as synonyms? Jamming would indicate intentional electromagnetic interference no?
- deleted 4y ago[deleted]
- dahfizz 4y ago> Electromagnetic Warfare, more commonly known as Electronic Warfare (EW), is defined by the U.S. Department of Defense (DoD) as, “any military action involving the use of electromagnetic and directed energy to control the electromagnetic spectrum or to attack the enemy.” Doesn't seem like your distinction is commonly used. https://www.baesystems.com/en-us/definition/what-is-electromagnetic-warfare https://www.baesystems.com/en-us/definition/what-is-electrom...
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- gambler 4y agoWhat is the evidence of this "electromagnetic warfare attack"? What is claimed to be attacked and how? How was it affecting SpaceX? Is SpaceX/Starlink really crucial to anything in Ukraine? Etc, etc. The only "explanation" this article offers is a link to a meandering video that doesn't answer any of these questions. After "establishing" the premise, the article immediately launches into demands for more resources: "We need to be able to have that agility" "We need to be able to change" "The U.S. needs to think a lot more innovatively" "This includes incorporating artificial intelligence and machine learning into next-generation systems to be able to respond faster" Over time articles like this one make people accumulate an extensive set of assumptions that basically aren't supported by anything except endless repetition.
- sterlind 4y agoagreed that the article is trash, but Starlink is important to Ukraine - the Ukrainian drones use Starlink for their uplinks (note that terminal roaming is enabled in Ukraine, you couldn't do this in the US!)
- Sebb767 4y agoHow did the update propagate to the ground terminals if the connection was jammed? I assume they switched frequencies, but it would still be interesting how it went exactly.
- deleted 4y ago[deleted]
- mempko 4y agoI feel like this article is over-representing how much SpaceX kept the country online. How much of Ukraine's traffic went over the SpaceX satellites? Even in the east where most of the destruction and fighting happened, people have cell reception. Article made it sound like without SpaceX, Ukraine would not have internet. But doesn't Ukraine have connections to Europe through Poland and it's other neighbors? Russia would need to literally cut the cables or destroy every cellphone tower and ISP and backbone providers in the country (Russia didn't do any of these things). This also kind of reveals that SpaceX and other radio based technology is more vulnerable to attacks than wired connections. Also shows how resilient the design of the internet is! Routing algorithms FTW! I call BS that it was SpaceX that kept the country online.
- agapon 4y ago> where it was touched they mostly hit military targets Eh? Destroying upwards of 50% of residential buildings (apartment blocks) apparently counts as hitting military targets? But I agree on the role of Space X.
- mempko 4y agoWhat? You believe 50% of residential buildings in Ukraine have been destroyed?
- londons_explore 4y agoMany areas are without power, and most of those same areas are without cell service. Look at a live twitter geolocation heatmap and it becomes pretty clear which areas are 'offline'.
- mempko 4y agoGreat idea! https://onemilliontweetmap.com/?center=48.1367666796927,33.49731445312501&zoom=6&search=&timeStep=0&timeSelector=0&hashtag1=&hashtag2=sad&sidebar=yes&hashtagBattle=0&timeRange=0&timeRange=25&heatmap=1&sun=0&cluster=1 https://onemilliontweetmap.com/?center=48.1367666796927,33.4... I found the site above. Is there a better map?
- photochemsyn 4y agoBetter article here: https://www.theinfographicsshow.com/musk-starship-starlink-cybersecurity/ https://www.theinfographicsshow.com/musk-starship-starlink-c... > "Jamming attacks are common threats to wireless communications and happen when a hostile device creates radio signals that disrupt communications by decreasing the Signal-to-Inference-plus-Noise ratio (SINR)." > "Possible ways that a software update can help bypass a jamming attack is if the jamming device is targeting a particular frequency. The software can implement FHSS or DSSS, Frequency Hopping / Spreading Sequences of frequencies for anti-jamming purposes. However, FHSS or DSSS, frequency distribution type of protection against jamming is flawed. It is ‘limited by a common assumption that the jammer can jam only part of the communication channels or has a limited transmit power.'" That article points to this report, which discusses all kinds of novel ways to defeat such attacks, like using counter-jamming as a means of delivering bits between sender and receiver, bit synchronization schemes, and others: https://apps.dtic.mil/sti/pdfs/AD1051122.pdf https://apps.dtic.mil/sti/pdfs/AD1051122.pdf
- masswerk 4y agoMind that frequency hopping for anti-jamming purpose is basically WWII-era technology, compare Hedy Lamarr.
- est 4y agoOTA a satellite like a gansta.
- sega_sai 4y agoI find this article dubuous (or lets say "thinly sourced"). I.e. this phase "its Starlink satellite broadband service, which was keeping Ukraine connected to the Internet" is just blatantly wrong. Yes they've send hundreds or maybe a few thousands of terminals ot a country with tens of millions of people. I have not seen any evidence that this plays any role for Ukraine. It's good that Spacex did send the equipment, but it is still more of advertisement as far as I can see, rather than something keeps the connectivity of Ukraine to the internet. (I'm happy to see reports that show otherwise).
- rtutz 4y agoIt was apparently used to reconnect local cellular networks [1] to the Internet. In such a case, a single antenna scales better than if it is used by only one person. Still, the overall benefit was probably limited, otherwise we would have seen more reports about it. [1] https://www.reddit.com/r/Starlink/comments/txh896/starlink_was_used_to_restore_mobile_network_in/ https://www.reddit.com/r/Starlink/comments/txh896/starlink_w...
- mempko 4y agoThank you, I have said the same thing but you said it better than me!
- ryzvonusef 4y agoi wonder if this counts? https://twitter.com/FedorovMykhailo/status/1516474487381897225 https://twitter.com/FedorovMykhailo/status/15164744873818972... > Is Starlink that important? It wouldn't be possible to restore 10 km of cable connection between villages in Chernigiv region after serious battles so quick. Normally it takes few months. Another case: provider Baryshivka-Net resumed the network for 5 villages by using only 1 .
- simne 4y agoThat's why I disappointed in high uptimes, which most tech persons love. What really need in modern environment, high availability, so it is not problem when one server rebooting, while others could continue deliver service.