7 ms·
Story time: I worked on Google Fiber. I believed in the project and it did a lot of good work but here, ultimately, was the problem: leadership couldn't decide
by cletus 3y ago
Story time: I worked on Google Fiber. I believed in the project and it did a lot of good work but here, ultimately, was the problem: leadership couldn't decide if the future of Internet delivery was wired or wireless. If it was wireless then an investment of billions of dollars might be made valueless. If it was wired and the company pursued wireless, then this would also lose.
But here's the thing: if you decide to do neither then you definitely lose. But, more importantly, no executive would lose their head from making a wrong decision. It's one of these situations where doing anything, even the wrong thing, is better than doing nothing because doing nothing will definitely lose.
Intel's 10nm process seemed like a similar kind of inflection point. Back in the mid-2010s it wasn't clear what the future of lithography would be. Was it EUV? Was in X-ray lithography? Something else? Intel seemed unable to commit. I bet no executive wanted to put their ass on the line and be wrong. So Intel loses to ASML and TSMC but it's OK because all the executives kept getting paid.
I forget the exact timelines but Intel's 10nm transition was first predicted in 2014 (?) and it got delayed at least 5 years. Prior to this, Intel's process improvements were its secret weapon. It constantly stayed ahead of the competition. There were hiccups though, most notably the Pentium 4 transition in the Gigahertz race (only saved by the Pentium 3 -> Centrino -> Core architecture transition) and pushing EPIC/Itanium where they got killed by Athlon 64 and it's x86_64 architecture.
I see the same problems at Boeing: once engineering-driven people companies get taken over by finance leeches. This usually follows an actual or virtual monopoly, just as Steve Jobs described [1].
[1]: https://www.youtube.com/watch?v=tGKsbt5wii0 https://www.youtube.com/watch?v=tGKsbt5wii0
- bastardoperator 3y agoI used to work with an old Sun Microsystems dude, he was an executive at the company I was at. We used to have these meetings every week and he ended up attending one. We had been trying to come to a conclusion for weeks on a specific piece of tech. He stopped the meeting dead in its tracks and said we're going to make a decision right now, if it's wrong, we'll learn from it, if it's right, awesome. Not making this decision is more costly than making the wrong decision. I just remember thinking, finally, someone with some authority is getting this ball moving.
- cogman10 3y agoReminds me of a story that happened at my company. We needed and had purchased a rather expensive database software license, however, we didn't have the hardware yet to run that database. The guys doing hardware spent MONTHS debating on which $10k piece of hardware they'd pick to run the DB. The DB license cost? Something like $0.5 mill. As one engineer said to me "I don't care what hardware you guys get, purchase them all! We are wasting god knows how much money on a license we can't use because we don't have the hardware to install it on!"
- cj 3y agoI've noticed a lot of people, even ones making $200k+ in salary, are really bad when it comes to making decisions that involve any amount of money. E.g. I've been in meetings with multiple developers who, if you add up everyone's salary, is well over $1 million/year, debating for way too much time on whether it's worth it to buy a $500/month service to help automate some aspect of devops. Maybe this wasn't the case for your specific anecdote, but in the scenario I'm describing I got the feeling that a lot of people think about business purchases in the context of their own personal finances rather than in the context of the business's finances. Leading people to be extremely cautious with things like a $10k purchase that would be "expensive" if purchased as an individual and "cheap" if purchased as a company. In those cases, getting an exec to come in and pull the trigger can help. The exec is used to looking at big picture budgets/strategy, which IC's aren't. (Although I'm sure someone here can come up with another anecdote proving that wrong)
- cogman10 3y agoLol, that reminds me of another fun one. Every so often my company would provide lunches for the developers. However, they didn't want to spend too much money doing this. So how did they resolve it? They put together a committee of devs to discuss lunch options/etc. Easily 1+million/year of salary in one room debating whether we do Jimmy Johns or McDonalds and how they'd get the food to the office. For $2000, you can get some pretty nice catering for 100 people. But like you said, people just seem bad at thinking of that sort of big picture.
- cogman10 3y ago> leadership couldn't decide if the future of Internet delivery was wired or wireless. Weird way to think of this problem (IMO). I'd think there would always be a mixed wired and wireless world. Even if customers don't end up using wired connections to their homes, you'd still need wired connection to the antennas servicing a home, neighborhood, apartment building. That's where a lot of Telcos today are making their money. Not to the customer, but to Tmobile or At&t as the put in a fiber line directly to the antenna towers. And even if google wanted to be the end to end ISP for someone, they'd benefit from a vast fiber network even if they later decided it wireless was the best, because they already have the fiber wherever they'd need their wireless antenna.
- cletus 3y agoThe last mile is expensive. Even hooking up the customer to the line running outside their house is expensive. I've seen different customers estimate this at anywhere between $2000 and $5000 per premises. This assumes ~40% customer take-up rate so with more competitors, the cost to each goes up. It's one reason why overbuilds make no sense and municipal broadband is the best model for last mile Internet delivery. Wireless bandwidth keeps going up. Wireless is already >1Gbps inside a building. What if instead of spending $5000 per house, you could use tightbeam wireless or highly cellular network with >1Gbps bandwidth? You may have spent billions on a network that it would take decades to amortize and have it be made worthless by wireless last mile delivery.
- cogman10 3y ago> Wireless bandwidth keeps going up. Wireless is already >1Gbps inside a building. What if instead of spending $5000 per house, you could use tightbeam wireless or highly cellular network with >1Gbps bandwidth? You may have spent billions on a network that it would take decades to amortize and have it be made worthless by wireless last mile delivery. I'd presume you'd not cut the existing wired customers over to wireless. So it's not like the $5000 spent is lost, it's just that you can do new customers for cheaper (if you expect a lot of growth in an area). Overbuilds is a weird one. They can make sense if it's a brand new community as you can get a BUNCH of homes done for cheap and can pretty much immediately turn on internet when someone moves in.
- altruios 3y ago> I see the same problems at Boeing: once engineering-driven people companies get taken over by finance leeches. This usually follows an actual or virtual monopoly, just as Steve Jobs described [1]. There is a nice mental framework for viewing such things. It has a bit of a religious origin, but it effectively explains and describes what you're seeing (I'm viewing it through an atheistic lens). I mean, the egregore. This is the natural life cycle of an egregore! Which is explained by having two groups, those that serve the purpose the egregore was created for (engineers, people that provide value), and those that serve the egregore itself (financials, people that extract value). Both these groups need to exist for a healthy entity to exist. But the balance (seems to) always tip - the egregore eventually chooses the group that serves the egregore to lead - when that happens, the original vision is often lost, and the company looses customer trust by altering the relation the customer has with the egregore (how much value the customer extracts from the egregore vs how much value the egregore extracts from the customer). https://en.wikipedia.org/wiki/Egregore#:~:text=Egregore%20(also%20spelled%20egregor%3B%20from,a%20distinct%20group%20of%20individuals https://en.wikipedia.org/wiki/Egregore#:~:text=Egregore%20(a.... This pattern comes up, an possible indication of this flip: when the original owners of a company are pushed out, or leave.
- 5- 3y agorelated: https://www.jerrypournelle.com/reports/jerryp/iron.html https://www.jerrypournelle.com/reports/jerryp/iron.html from this recent thread, which is rather relevant: https://news.ycombinator.com/item?id=39491863 https://news.ycombinator.com/item?id=39491863
- sahaj 3y agoThe Google Fiber project was always meant to push the carriers into competition. Google knew that if they didn't launch Google Fiber, none of their other ventures or the internet as a whole, could be as successful. Google paid big money for YouTube and the plan was always to turn it into the service it is today. At the time, there were also worries whether the carriers would restrict services (aka net neutrality) or if they would charge by GB. Launching Google Fiber made it such that the carriers had to start competing and upgrade their infrastructure. If it wasn't for Google Fiber, I'm certain that we'd be stuck with 20mbps speeds, the cable/DSL monopoly, and we wouldn't have the likes of the OTT services and the choices that we have today. Or at least it would have been delayed by quite a bit. I worked for a company that was an equipment vendor for Google Fiber and other service providers.
- swores 3y agoPlenty of countries have better (faster and/or cheaper) broadband options than most of the US, without having any Google involvement. Competition (or government enforced requirements and price caps) are what's needed, Google Fiber had a bit more of an incentive than most for aiming to undercut their competitors but ultimately I think you're overstating their importance.
- toast0 3y agoCompetition would be nice, but just the appearance of credible competition was enough to induce the incumbents to do better. Google Fiber deployed to the Kansas Cities, making themselves credible competition. Then, they announced 20 cities they would deploy to. Suddenly, incumbents in 20 cities had deployment plans and deployed before Google Fiber got anywhere, and then Google Fiber decided not to do any new deployments. Would the incumbents have deployed without Google Fiber's credible competitive announcements? Maybe? We'd need inside information to know for sure. It sure doesn't feel like they would have though.
- rangerelf 3y ago> Would the incumbents have deployed without Google Fiber's credible competitive announcements? Of COURSE they wouldn't! If google fiber hadn't happened, all providers would have continued sitting on their collective asses, soaking as much money as possible, doing the least possible legally permissible work, nickle-and-diming customers as much as possible.
- throw0101c 3y ago> Was it EUV? Was in X-ray lithography? Something else? Intel seemed unable to commit. Per the book Chip War, Intel put a lot of money into EUV (going back to the late 1990s): * https://en.wikipedia.org/wiki/Chip_War:_The_Fight_for_the_World%27s_Most_Critical_Technology https://en.wikipedia.org/wiki/Chip_War:_The_Fight_for_the_Wo... Per the book, and other sources: > Intel seemed primed to dominate the chip industry as it transitioned into the era of Extreme Ultraviolet Lithography (EUV). The company had played a pivotal role in the development of EUV technology, with Andy Grove’s early investment of $200 million in the 1990s being a crucial factor. * https://techovedas.com/intel-lost-decade-5-reasons-why-chip-giant-did-fall-behind/ https://techovedas.com/intel-lost-decade-5-reasons-why-chip-...
- agumonkey 3y agoI know nothing, but it felt like intel paid the price of being the first. They picked something hard and pricey.. and it didn't pan out in time, allowing other competitors to catch up and adapt to markets (mobile) nicely.
- throw0101c 3y ago> I know nothing, but it felt like intel paid the price of being the first. As the book goes into, there were other things in question: since TSMC only did fab, and did not design, they had more customers/opportunities to iterate the process and get good at it (more focus). There was internal-to-Intel stuff that led to lead loss as well. I'm only partially through the book currently, and there's a lot of chip history being described (going back to the 1950s), so I'm not going to retain all of it in a single pass.
- mdasen 3y agoOne difference I'd point to is that Intel was doing "fine" not committing to future lithography. I put that in quotes because clearly it wasn't a fine plan over the long run, but not spending money on the future is a fine plan in the short/medium term. AMD had been struggling for years and Intel continued to handily beat them. ARM processors weren't a threat at the time either. Intel certainly had the better part of a decade where they weren't committing to future lithography and doing fine. Before someone says, "but they lost mobile to ARM during that period," lithography isn't why they lost mobile to ARM. Apple was using TSMC's 16nm process in their September 2016 iPhone while Intel started shipping 14nm processors 2 years earlier. Mobile chose ARM when Intel wasn't behind on lithography. With Google Fiber, not choosing had immediate repercussions. With Intel, the repercussions took the better part of a decade to manifest. Google just decided it didn't really care about the home internet business. No one at Google could say "yea, we're not rolling out wired or wireless home internet and the business is booming." Intel didn't decide that they were exiting the processor business, but their processor business was doing "fine" without this decision being made. Intel could say, "we aren't investing in future lithography and the business is booming anyway. Maybe future lithography is just a big waste of money." You're correct that not choosing means you lose. However, sometimes it isn't obvious for a while. Google Fiber's lack of decision had obvious, immediate results and you couldn't delude yourself otherwise. Intel could delude itself. Execs could write reports about how they were still ahead of the competition (they were) and how they weren't wasting money on unproven technology. Fast forward a decade and they're not fine, but it took a while for that to manifest. Plus, if Apple hadn't helped push TSMC forward so much, would Intel be in quite as bad a situation? Qualcomm has been happy to just package together ARM reference designs with their modems and it's really just their poor performance compared to Apple really pushing them forward. While Android users on HN might be buying Snapdragon 8 series processors, the vast majority of Android devices aren't using high-end ARM cores. The vast majority of the market for high-end ARM cores is Apple. If Apple hadn't made a long-term commitment to TSMC for 2016-2021, would TSMC have pushed as hard on EUV? It's a lot easier to invest when you have a guaranteed customer like TSMC had in Apple. If Apple hadn't pushed performance so strongly, would we have seen as much EUV investment as quickly? It's unlikely it would be pushed by the Android ecosystem where most processors are low-end. TSMC serving Apple meant EUV investment. Once Apple was shipping extremely fast processors, Qualcomm and others wanted to be able to get to at least 50-70% of what Apple was offering (so there were more buyers). Once it was available, AMD could use it to push hard against Intel. Once there were more buyers, Samsung wanted to make sure that its fabrication business was at least in the ballpark. But if Apple hadn't been focused on taking a strong performance lead, it might have been another 5+ years before Intel's lack of decision came back to haunt it. If it had taken 12-17 years instead of 7-9 years for others to put the screws to Intel, they would have basked in its profits for a long time as its execs were touted as having amazing insight. Of course: you're right. Eventually, Intel would have gotten its comeuppance. But Intel could have pretended it didn't need to invest in the future for a long time. By contrast, when Google didn't make a decision on wireless or wired, that was just the end of expanding that business.
- WoahNoun 3y agoBoeing's previous CEO was an engineer.
- AnthonyMouse 3y ago> I believed in the project and it did a lot of good work but here, ultimately, was the problem: leadership couldn't decide if the future of Internet delivery was wired or wireless. If it was wireless then an investment of billions of dollars might be made valueless. If it was wired and the company pursued wireless, then this would also lose. This one is particularly amusing because the difference is primarily a business distinction and not a technical one. Here's how your tablet gets internet via fiber: There is a strand of fiber that comes near your house and then you attach an 802.11 wireless access point to it. Every few years the latter has to be replaced as new standards are created. Here's how your tablet gets internet via 5G: There is a strand of fiber that comes near your house and then the telco attaches a cellular wireless access point to it. Every few years the latter has to be replaced as new standards are created. They should have just built the fiber network and put cell sites on some of the poles. Then you sell fiber to anybody who buys it and cellular to anybody who buys it and you don't have to care which one wins.
- etempleton 3y agoYou start with product people, but if you do well enough making your product better doesn't really move the needle any more, so organizations tend to promote... marketing / sales /operations people. These people usually are pretty good at understanding what the customer wants and so has a decent feel for the product, perhaps innovation goes down, but the customer is getting what they want, but then once you saturate the market sales and marketing are no longer going to move the needle so you promote... Finance people. They usually don't have a great feel for product nor even what the customer wants, but they understand how to increase revenue and decrease costs and at this point in the company lifestyle that is what matters most. The risk is that you are in a competitive space where competitors are willing to jump on any product stumble. Often companies get stuck at this stage and stagnate, but usually they are so large and entrenched they keep doing just fine anyway.