11 ms·

LinkedIn shelved plan to migrate to Microsoft Azure cloud

187 points by helsinkiandrew 3y ago

miguelazo 3y ago
There are plenty of issues with Azure, but LinkedIn is hardly at the vanguard of innovation. And that was still the case before Microsoft vastly overpaid for it.
- joshhart 3y ago
  I left LinkedIn 1.5 years ago. I was there 12 years. I saw the revenue & profitability growth that occurred post acquisition. I am very very confident LinkedIn would be worth north of $100B on public markets today and Microsoft made the acquisition for $26B. You might argue that in the subsequent 6 years post acquisition that wasn't enough growth and they should have bought back shares instead but it was completely a debt financed acquisition and very high ROI for Microsoft.
  - manquer 3y ago
    Adjusting for inflation 26B from 2016 is worth 32B now. Going by just market returns 26B in S&P 500 in 2016, would be worth 70B today. Also 26B is just initial investment, MS surely invested more money in the division in last 8 or so years, Linkedin was not exactly highly profitable entity in 2016, while it was not burning a lot of money, growth experienced in last few years would have needed additional investments in the business. I don't have specific opinion on whether MS overpaid are not, just want to point out even 100B valuation today does not necessarily mean its high RoI for MS yet.
    - joshhart 3y ago
      LinkedIn was already very FCF positive. They tightly managed margins to get to net income positive (account for dilution and so on) but it took maybe 2 years after the acquisition.
      - manquer 3y ago
        Of course, they were and are healthy company, i was just trying to say they while very healthy it was not so profitable that just by using the free cash flow MS could have grown the business to today's size, it likely required external cash infusion and therefore $26 Billion is likely not the only money MS has spent on Linkedin. As far as M&As go, it is very successful outcome, vast majority of them fail spectacularly, RoI is not perhaps the right metric to judge a strategic acquisition.
  - miguelazo 3y ago
    Financially, perhaps the numbers made sense. It was (and still is) a very basic social networking site with poor UX. Microsoft paid for a user base and a rudimentary surveillance system. It would have probably been better for them to just start from scratch. I don’t know a single person who likes LinkedIn. Most see presence on it as an obligation for certain industries. Most of the commenters on there are far worse than what you see on Facebook. It’s usually just one long stream of retired or underemployed old men complaining about “wokeness” and environmentalists ruining the world.
helsinkiandrew 3y ago
4 years ago: "LinkedIn is moving to Microsoft’s Azure public cloud three years after $27 billion acquisition" https://www.cnbc.com/2019/07/23/linkedin-is-moving-to-microsoft-azure-three-years-after-acquisition.html https://www.cnbc.com/2019/07/23/linkedin-is-moving-to-micros...
RoyTyrell 3y ago
My company has invested in moving to Azure except where we need to stay on Google. Apparently MS gave us a package on all of their products if we use Azure and it was enough to sway the execs. We were then given the directive that everyone at my level would need to get some certifications so we could properly use Azure, assist the architects and more jr devs. It’s a good idea but my god the training is so poorly executed. I want to like Azure but it also seems like an uncoordinated mess. Maybe I’m just a grumpy dev. Anyone else have a better and more positive perspective? Who has good training for certs such as the Data Engineer or AI Engineer?
- whoknowsidont 3y ago
  >I want to like Azure but it also seems like an uncoordinated mess. It's literally a "me too" cloud. Azure is notorious for not being "finessed" out and god speed if you have to use AKS.
  - jtwaleson 3y ago
    What are your main problems with AKS? I’m setting up a new greenfield project and have to use it. Would be nice to avoid done pitfalls.
    - aryonoco 3y ago
      I second this. Seriously, don't use AKS. It looks so nice in the docs. In practice, you'll run into so many issues, I'm convinced no one actually uses AKS in production for anything major. If your apps are simple enough that you can get away with Azure Container Instances/Apps, use them. If you need proper managed k8, use OpenShift. Microsoft has a partnership with RedHat and RH provides full support.
      - ahoka 3y ago
        Some examples of those issues would be nice to back up your claim.
green-eclipse 3y ago
Microsoft bought Hotmail back in 1997. Hotmail was powered by Unix servers until 2004, despite MS's best efforts to transition to their own Wintel-powered backend [0]. These things take time. [0] https://news.softpedia.com/news/Windows-Live-Hotmail-Was-Powered-by-UNIX-Servers-until-2004-131323.shtml https://news.softpedia.com/news/Windows-Live-Hotmail-Was-Pow...
yellow_lead 3y ago
> LinkedIn was having a hard time taking advantage of the cloud provider's software. Sources told CNBC that issues arose when LinkedIn attempted to lift and shift its existing software tools to Azure rather than refactor them to run on the cloud provider's ready made tools. I think I need this translated back into tech-speak.
- aldarisbm 3y ago
  [flagged]
- asylteltine 3y ago
  That is tech speak. They tried to redeploy existing architecture into azure and it failed
  - ghaff 3y ago
    The headline notwithstanding, this doesn't seem like anything particularly Azure specific. They'd likely have had many of the same issues trying to mostly lift and shift to any of the big public cloud providers.
- eitally 3y ago
  ELI5: For any sufficiently complex enterprise system (e.g. LinkedIn, or Google), any plain vanilla architecture is infeasible for lift & shift. Moreover, the vanilla services may not comply with internal security requirements, or play nicely with internal CI/CD tools, or internal databases / data structures / data processing pipelines / analytics.
  - mecsred 3y ago
    No five year old is going to understand a single sentence of that.
    - calvin 3y ago
      ELI40YO Engineer
asylteltine 3y ago
[flagged]
- aldarisbm 3y ago
  having used azure after long stints with AWS and GCP. I found Resource Groups a much more logical bucket, moreso than anything on AWS/GCP
  - jen20 3y ago
    Resource groups are fine, but it is quite unclear why a group has a region when resources with in may be in other regions.
    - ahoka 3y ago
      The object has to be in a database in one of the regions I guess.
      - johngossman 3y ago
        That is correct. Though all that metadata is replicated
        jen20 3y ago
        Is there a case where it matters to the user which region their group is in if all of the resources live in a different group of regions?
        johngossman 3y ago
        For public regions, I don’t think so. At worst, there might be a tiny bit of extra latency as the orchestrator works across datacenters. But region is also used by Azure Stack and the rules might be different then. I worked on this a number of years ago and if you are really concerned, you should ask support.
upon_drumhead 3y ago
I wonder how the GitHub to Azure migration is going
- deleted 3y ago
  [deleted]
- redrove 3y ago
  I have it on good authority they’re trying a lift and shift too and it’s not going well, at least as of ~9mo ago.
duxup 3y ago
Any move to any cloud is going to depend on the environment you're coming from. I've been in on decisions not to use X, Y, Z ... doesn't mean there was anything wrong with them, we just weren't ready for that yet or had different priorities or the ever present weird deal-breaker issue / requirement.
- brodouevencode 3y ago
  Exactly. The cost to retool for the cloud is not insignificant.
- konschubert 3y ago
  You have to 1) Be comfortable routing traffic between on-prem and cloud over the internet or at least over a VPN, and 2) avoid the temptation to build your own platform (Terraform templates are a liability, not an asset!) and 3) Move tiny stuff first and bigger stuff later. It’s amazing how many companies fail at that.
  - bobthepanda 3y ago
    Doing things the right way and learning from before usually requires a certain degree of humility, and more often than not the person leading these projects is either required to be, or is deluding themselves to be a hotshot who can succeed in a bold new way.
    - konschubert 3y ago
      You mean somebody who posts 3 bullet points on the internet and claims that they solve everything ? ;)
      - bobthepanda 3y ago
        I mean, we invented the term Promotion Oriented Architecture for a reason. It’s like politics; the best person for the job is a sufficiently experienced person who does not want it.
mfer 3y ago
This doesn't appear to be about Microsoft's cloud but rather Public Cloud. The whole migration of LinkedIn from their own data centers to the public cloud (Microsofts) isn't going well. It appears they are still going to operate on-premise for many things. Some things moving or have moved to the public cloud. Isn't this more a shot at the public cloud for all the things than to any specific one?
- that_guy_iain 3y ago
  I don’t see anything that points to it being a general public cloud issue. And instead they talk about Azure software specifically as something that they couldn’t take advantage of, no?
  - mlhpdx 3y ago
    I would not assume that it is a specific Azure problem from that statement. Many, many teams struggle to take advantage of cloud infrastructure because of habits and knowledge retained for operating the existing systems. It’s possible given what they have, I t’s simply best to keep it on premise - at least to some degree. That would likely not be true with a successful re-architecture, but not everyone is up for that.
    - mfer 3y ago
      It may not be about the teams. For example, when you control the data center you can do certain things around performance and scale you can't do in a public cloud. There are so many unknowns about how things are setup that it's hard to know.
- carimura 3y ago
  Yes I came away with the same thing. It's The Register's modus operandi to use cheeky clickbait titles.
femiagbabiaka 3y ago
Most cloud migration projects at large companies fail. It usually takes 3 or 4 tries at least before all the necessary lessons are learned.
- bob1029 3y ago
  Our customers are really good at making a total shitshow of cloud and our jobs very easy as a result. Much sentiment over cloud looks to me like stick-in-bike-wheels meme. It's hard because you are making it hard. Strategically, I don't think you try to help someone with cloud until they've burned out their ego trying some grand delusion. Even one bad actor can throw the whole thing off. Cloud is about doing more with less, so committee bullshit is cursed. Once you've got your customer completely humbled by reality, they be willing to listen to reason and you will save so much frustration. We've got a huge one in the pipeline right now. They've been trying to "go to the cloud" for about 5 years now and executive management is ready to reset the entire tech org chart over the lack of progress. Cloud native is the solution but many technologists perceive it as the end of their careers. Anyone pitching a "cloud native" solution that still has container spam managed by the tenant owner is either incompetent or trying to protect their career at this point.
  - femiagbabiaka 3y ago
    Sad but very true. When careers have been crafted on the current architecture of a company, it's hard to shake.
ro_bit 3y ago
[flagged]
danielrhodes 3y ago
[flagged]
oooyay 3y ago
Maybe I'm a bit contrarian on this one but once I saw data center, Azure, and the phrase "lift and shift" it filled in a lot of context for me. I spent a lot of my early to mid career participating in these strategies. They don't work. VM images almost always are different in some way, there's something one vendor provides that another doesn't - in general there's enough minute details that add up to make a series of mini-mountains in terms of blockers.
- jeffbee 3y ago
  Yep, there are always differences. Just one thing I stumbled into recently was one of our program images that has long worked fine in AWS can't start in Azure because something their hypervisor does to the virtual address layout conflicts with the way that we remap .text to a huge page. It is both trivia and a showstopper.
- pphysch 3y ago
  Yeah, there is a vast gulf between "it works for us" and "every dependency was implemented strictly according to open standards and is therefore seamlessly portable". See also the joke of migrating between "SQL" databases.
lgkk 3y ago
How do you move that much data over to another cloud provider? Without losing data or disrupting the customer? Or do the databases just stay in the data center and not migrate.
- wharvle 3y ago
  Live replicas (perhaps initialized with a cold backup, initially, if the dataset's really huge), carving off parts of it for separate migration if that's at all feasible, and some expensive folks doing a lot of butt-clenching-worthy activity for an hour or two (unless it goes very poorly...) for the final cut-over, some evening.
- ThomasMoll 3y ago
  We (when I worked at LinkedIn) did it with ETL clusters, we already had built them out for moving data between datacenters nightly. They would mirror an HDFS cluster, then ran batch jobs to transfer either directly to the outbound cluster or to another ETL cluster in another DC. We used one of our ETL clusters to ship data to MSFT for various LinkedIn integrations, like seeing LinkedIn profile information in Outlook or Office products.
  - junto 3y ago
    Which tools were you using for ETL? Or were they completely custom?
- tqi 3y ago
  Roughly, pick a date and start writing all new data to both, while running etls to backfill data from old to new. Once that is done, you use a feature flag to do a small % of reads from the new system and wait and pray. If nothing major pops up, you slowly ramp up the % of reads until you're confident (as much as one can be) that everything is working, then you move 100% of reads to the new system. Finally, you turn off writes to the old system, clean up the feature flag and remove any old unneeded code. jk of course you never do any of the other stuff after turn off writes to the old system you just leave the FF turned on to 100% and never think about it again :-p
lumost 3y ago
It's incredibly difficult for a mature software business to justify infrastructure and tooling investments. This is why we think that startups are a haven for modern tooling and the largest legacy firms are ... well ... difficult. The last 15 years possibly broke this rule by virtue of low interest rates, enabling the justification of large internal teams focused on modernization efforts which sometimes went as far as moving the state of computing forward. I wouldn't be surprised to see legacy enterprises return to form now that interest rates are 7%
- cynicalsecurity 3y ago
  On premises hardware can hardly be called legacy. Clouds are way too expensive for startups; they are a milking machine for the big corpo.
  - lumost 3y ago
    Startups are typically capex constrained, at least until series C. Clouds are favorable to capex constraints. Having personally tried to be cheaper than the cloud in 2013 through large hardware buys, negotiated contracts etc. I found that the roi relative to a progressively discounting cloud weren’t there.
ksec 3y ago
I wonder what sort of scale do LinkedIn operate in terms of Server count. And Github also under Microsoft seems to be doing fine with on-prem as well. Why force LinkedIN to use Azure?
- wredue 3y ago
  If I had to guess, there are hordes of businesses out there that maintain operations on prem, and a large lift like this is great for the resume. Of course, I could also be entirely wrong, but I also am not going to pretend that IT resume padding then jumping ship and leaving a shart of an architecture behind doesn’t happen all the time in this industry.
- astockwell 3y ago
  $$$
- rdoherty 3y ago
  When I was there it was in the low hundreds of thousands. Probably more as growth was still in double digit percentages per year of user base.
  - ksec 3y ago
    >When I was there it was in the low hundreds of thousands. Blows my mind every time I see these kind of numbers.
    - fbdab103 3y ago
      Indeed. According to this in 2022[0], Stack Overflow is still running off of 9 web servers. [0] https://www.datacenterdynamics.com/en/news/stack-overflow-still-on-prem-runs-qa-platform-off-just-nine-servers/ https://www.datacenterdynamics.com/en/news/stack-overflow-st...
      - manquer 3y ago
        Social media org like Linkedin is hardly comparable with SO style KB the R/W workloads are much higher with Social media, than a KB where vast majority of the transactions can be cached, Wikipedia would be a better comparison to SO. Also Linkedin is 100x the size in users : 1B vs just around ~15M for StackOverflow.
thenewwazoo 3y ago
I'm obviously not going to comment on anything internal, and I'm obviously speaking for myself and not the company, but it's worth bearing in mind that this migration was not from "on-prem" in the traditional sense. LinkedIn has its own internal cloud, complete with all the abstractions you'd expect from a public cloud provider, except developed contemporaneously with all the rest of the "clouds" everyone is familiar with. It was designed for, and is tightly coupled to, LinkedIn's particular view on how to build a flexible infrastructure (for an extreme example, using Rest.Li[1], which includes client-side load balancing). There was no attempt to "lift-and-shift" anything. There are technologies that overlap and technologies that conflict and technologies that compliment one another. As with any huge layered stack, you have to figure out which from the "LinkedIn" column marry well with those in the "Azure" column. I personally appreciate LI management's ability to be clear-eyed about whether the ROI was there. [1] https://linkedin.github.io/rest.li/ https://linkedin.github.io/rest.li/
- dfxm12 3y ago
  Yeah, based on my own experience with AWS and Azure (that has nothing to do with Linked In), my immediate reaction to the headline was, "well, you can be keen on Azure, but "stuck" on AWS for a myriad of other reasons". Reading the article pretty confirmed it.
  - whoknew1122 3y ago
    I'm not sure how this has anything to do with AWS. But I guess you can use it to confirm your idea about vendor lock-in if you're bound and determined to convince yourself you're right. For what it's worth, working at AWS, I see a lot of companies who use Entra for AD and then federate into AWS using that.
    - ajcp 3y ago
      I think you misunderstood OPs anecdote and took it a little personally? They were simply saying that much as you can love one platform there are many valid reasons why your deployment on another makes it hard to shift that and recapture that configuration. Nowhere did they intimate it was a fault of the current platforms capabilities...
tjpnz 3y ago
Sounds like they would've faced a similar set of issues moving to AWS or GCP.
joshhart 3y ago
This was cancelled over a year ago - which the articles notes and is old news. It was clear the effort would have needed a very significant push that would have required a large halt in product development and management wasn't willing to stomach it due to high growth in 2020/2021. Which made sense. But LinkedIn revenue growth has heavily slowed with the pullback in tech hiring and they had the space to do it and consider it optimization time. Also as part of Blueshift the plan was to do batch processing first but LinkedIn had a culture belief in colocation of batch compute & storage, which is against the disaggregated storage paradigm we see now. IMO this led to some dragging of feet. Source: Worked at LinkedIn 12 years, am a director at Databricks now.
- ThomasMoll 3y ago
  Not only that but the Hadoop team literally had the guy who wrote the original HDFS whitepaper. Moving a service with that much in house expertise first never made sense. I worked on one of the original Azure PoCs for Hadoop, even before Blueshift and it was immediately clear that we operated at a scale that Azure couldn't handle at the time. Our biggest cluster had over 500PB and total we had over an exabyte as of 2021 [1]. It was exorbitantly expensive to run a similar setup on VMs, and at the scale that we had I think it would have taken over 4,000 - 5,000 separate Azure Data Lake namespaces to support one of our R&D clusters. I believe most of this "make the biggest cluster you can" mentality was a hold over from the Yahoo! days. [1] https://engineering.linkedin.com/blog/2021/the-exabyte-club--linkedin-s-journey-of-scaling-the-hadoop-distr https://engineering.linkedin.com/blog/2021/the-exabyte-club-...
hasty_pudding 3y ago
As someone who worked at LI. They spent years and god knows how many millions TRYING to move to Azure with the Blueshift project..before pulling the plug. They hired armies of contractors. They didn't stop by choice. They stopped because their tech stack is a giant over engineered unmovable turd.
- bbkane 3y ago
  As a current employee, there's things I don't like, but the infrastructure is more custom than bad (far better than my last job)
  - vasco 3y ago
    Custom is usually bad, just takes longer to reveal the problems. Sometimes companies need to create custom things, the mistake is continuing to invest in them when a community project appears and doubling down for years until nobody in your org knows how it's done "normally" and nobody that you hire knows anything about your stack.
    - deleted 3y ago
      [deleted]
  - hasty_pudding 3y ago
    The entire company has no Q.A team. The amount of bugs I troubleshot in that tech stack was staggering just for basic day-to-day stuff. promotion based engineering means an engineer shits something out that looks cool in a demo then bails to the next cool demo project while the rest of us are stuck with the reality of the turd.
- bushbaba 3y ago
  ^ it’s not an azure issue but a LinkedIn issue.
- carlivar 3y ago
  Didn't LinkedIn create Kafka? Was that some of the overengineering?
jabiko 3y ago
Since LinkedIn is using Kafka quite extensively, I'm wondering whether Azures poor support of Apache Kafka is at least partially responsible for this. Azure is developing its own inhouse message broker that they call EventHubs. And while it is able to speak the Kafka protocol it has some weird limitations and quirks¹. I'm wondering why they are so hellbent on developing their own inferior version of something when they could just use the real deal and host it as SaaS. ¹ No support for KIP-71 (Enable log compaction and deletion to co-exist), limited to 10 topics (Azure lingo: EventHubs) per broker (Azure lingo: EventHubs namespace), mandatory client config parameters that you must not forget to set, even though the documentation just calls it "recommended" (https://learn.microsoft.com/en-us/azure/event-hubs/apache-kafka-configurations https://learn.microsoft.com/en-us/azure/event-hubs/apache-ka...), some weird protocol level incompatibilities that are nowhere documented, for example a consumer group state transitions to "Dead" after the last consumer left instead of "Empty", and auto scaling that only ever scales up, but never down.
- jameshart 3y ago
  > Azure is developing its own inhouse message broker that they call EventHubs This seems to imply Event hubs is some sort of upcoming beta or something. Event Hubs are one of the oldest parts of Azure IaaS - I think it was launched in 2014?
  - jabiko 3y ago
    I didn't want to imply that its brand new or beta. Just that it is still being actively developed. For example the compacted topic support did get into GA in May 2023.
- TideAd 3y ago
  Lots of Azure stuff is like that. Some big services internally at Microsoft refused to use their Service Fabric product, it couldn't handle the scale & complex deployment needs.
  - jiggawatts 3y ago
devaiops9001 3y ago
LinkedIn doesn't have competent people. Anyone who has peaked behind that curtain sees they struggle with very simple things.
- dboreham 3y ago
  I don't have deep intel, but all the ex-LI people I've worked with or met seemed pretty competent.
risfriend 3y ago
My company migrated to Azure few years back. They can give good bulk discounts but on the flip side the experience with some of their infra and AKS has been choppy at times, with the support team taking time to fix or rca h/w issues. They do come back though. Would love to know how does it compare to long term experiences with other major cloud providers.
- bob1029 3y ago
  I've been with all 3 major and Azure is the only one that even feels like it has support. Even so, unless you are a big customer on some special agreement, you aren't going to get any red carpet treatment. We pay for the $100/m support plan right now and it's pretty goddamn mediocre. Maybe submitting tickets for "legacy" AD domain services outages doesn't touch the rockstar support team these days. Support quality is probably variable across product team at some level.
FpUser 3y ago
If their stack works fine and I assume it it why the fuck do they need to move it at all?
- seanhunter 3y ago
  Well they decided they didn't need to. That being said, you can't necessarily wait until things no longer work fine before deciding to move. If you can reasonably anticipate a problem in the future, sometimes even if things work fine today you might be best off moving in order to avoid future pain.
  - FpUser 3y ago
    >"you can't necessarily wait until things no longer work fine before deciding to move" You mean cloud is the holy grail where everything works and everything else is a failure waiting to happen? I claim BS.
    - seanhunter 3y ago
      No I don’t mean that at all, which is why I didn’t say that. What I mean is that in planning anything in tech you can’t always wait until problems arise before you take action to deal with them, particularly if the consequences of failure are extreme. The specifics of what is the right setup are going to depend heavily on the circumstances of the use case. Definitely being on cloud isn’t right for everyone, nor is trying to maintain on-prem hardware the right choice for everyone.
- nyokodo 3y ago
  > why … do they need to move it at all? Because they’re owned by Microsoft and it looks bad if Microsoft doesn’t dogfood its own cloud infrastructure. Plus it’d be more economical long term if they didn’t invest in a parallel cloud infrastructure. They will migrate eventually, just apparently it’s not ready for them yet.
  - FpUser 3y ago
Sparkyte 3y ago
Only really benefits a company if they lack the technologies or cloud solution. Not like they are paying anyone to host something besides themselves.
personalityson 3y ago
Isn't LinkedIn owned by Microsoft?
mgaunard 3y ago
why would you ever want to move to cloud if you have a functional non-cloud setup? Cloud platforms charge you much more than what the hardware is worth, the only advantages are that they provide a simplified resource management system (which is mostly an illusion, as any non-trivial system will require you to build in-house tech regardless) and the ability to scale more easily (which is done at a prohibitive cost, so a good reason to switch away if you need to scale up).
JOnAgain 3y ago
This is why AWS is better. They dogfood when it is hard. I remember when S3 and EC2 were coming out. They tried to make us working on the retail website move year after year. Our excuses became their roadmap. We (developers working on not AWS) really didn’t want to move to AWS. It was 100% worse than what we had before. “Network is slow”, “it’s not PCI compliant”, “the drive is unreliable”, “the load balancer is missing a feature”. It took years and years. But they did it. Google and Microsoft don’t have the willpower to force this, and its why they will always be behind AWS. When Google tries to sell you Google Cloud, remember they don’t use it for anything internally. They don’t think it’s good enough. So why should I?
- seper8 3y ago
  ... You think Microsoft doesn't dogfood? Where do you think Teams, Bing, OpenAI, the billion internal systems they use and Xbox are hosted
  - hyperliner 3y ago
    I think the point is that the scale of requirement intake that came into AWS from the website is orders of magnitude more than whatever level of dogfooding that Azure might undergo from internal solutions. Not that they don’t dogfood at all, only that the massive amount of engineers moving one of the most complex websites in world’s history into AWS, and AWS taking them in as real input into the product, made AWS a lot better than the old AWS.
  - alternatex 3y ago
    Yeah reading that comment felt like the twilight zone. Microsoft is foremost a software company and they host everything on Azure. Amazon's software services offering is a speck of dust in comparison. Amazon cannot dogfood AWS more than Microsoft dogfoods Azure.
    - carlivar 3y ago
      Well, to take it full circle, they don't dogfood LinkedIn on Azure. How about GitHub for that matter? Wouldn't that indicate Amazon can dogfood more?
chucke1992 3y ago
I guess Azure has run out of hardware. AI requires too much computation power.

There are plenty of issues with Azure, but LinkedIn is hardly at the vanguard of innovation. And that was still the case before Microsoft vastly overpaid for it.

joshhart 3y ago
I left LinkedIn 1.5 years ago. I was there 12 years. I saw the revenue & profitability growth that occurred post acquisition. I am very very confident LinkedIn would be worth north of $100B on public markets today and Microsoft made the acquisition for $26B. You might argue that in the subsequent 6 years post acquisition that wasn't enough growth and they should have bought back shares instead but it was completely a debt financed acquisition and very high ROI for Microsoft.
- manquer 3y ago
  Adjusting for inflation 26B from 2016 is worth 32B now. Going by just market returns 26B in S&P 500 in 2016, would be worth 70B today. Also 26B is just initial investment, MS surely invested more money in the division in last 8 or so years, Linkedin was not exactly highly profitable entity in 2016, while it was not burning a lot of money, growth experienced in last few years would have needed additional investments in the business. I don't have specific opinion on whether MS overpaid are not, just want to point out even 100B valuation today does not necessarily mean its high RoI for MS yet.
  - joshhart 3y ago
    LinkedIn was already very FCF positive. They tightly managed margins to get to net income positive (account for dilution and so on) but it took maybe 2 years after the acquisition.
    - manquer 3y ago
      Of course, they were and are healthy company, i was just trying to say they while very healthy it was not so profitable that just by using the free cash flow MS could have grown the business to today's size, it likely required external cash infusion and therefore $26 Billion is likely not the only money MS has spent on Linkedin. As far as M&As go, it is very successful outcome, vast majority of them fail spectacularly, RoI is not perhaps the right metric to judge a strategic acquisition.
- miguelazo 3y ago
  Financially, perhaps the numbers made sense. It was (and still is) a very basic social networking site with poor UX. Microsoft paid for a user base and a rudimentary surveillance system. It would have probably been better for them to just start from scratch. I don’t know a single person who likes LinkedIn. Most see presence on it as an obligation for certain industries. Most of the commenters on there are far worse than what you see on Facebook. It’s usually just one long stream of retired or underemployed old men complaining about “wokeness” and environmentalists ruining the world.

helsinkiandrew 3y ago

4 years ago: "LinkedIn is moving to Microsoft’s Azure public cloud three years after $27 billion acquisition" https://www.cnbc.com/2019/07/23/linkedin-is-moving-to-microsoft-azure-three-years-after-acquisition.html https://www.cnbc.com/2019/07/23/linkedin-is-moving-to-micros...

RoyTyrell 3y ago

My company has invested in moving to Azure except where we need to stay on Google. Apparently MS gave us a package on all of their products if we use Azure and it was enough to sway the execs. We were then given the directive that everyone at my level would need to get some certifications so we could properly use Azure, assist the architects and more jr devs. It’s a good idea but my god the training is so poorly executed. I want to like Azure but it also seems like an uncoordinated mess. Maybe I’m just a grumpy dev. Anyone else have a better and more positive perspective? Who has good training for certs such as the Data Engineer or AI Engineer?

whoknowsidont 3y ago
>I want to like Azure but it also seems like an uncoordinated mess. It's literally a "me too" cloud. Azure is notorious for not being "finessed" out and god speed if you have to use AKS.
- jtwaleson 3y ago
  What are your main problems with AKS? I’m setting up a new greenfield project and have to use it. Would be nice to avoid done pitfalls.
  - aryonoco 3y ago
    I second this. Seriously, don't use AKS. It looks so nice in the docs. In practice, you'll run into so many issues, I'm convinced no one actually uses AKS in production for anything major. If your apps are simple enough that you can get away with Azure Container Instances/Apps, use them. If you need proper managed k8, use OpenShift. Microsoft has a partnership with RedHat and RH provides full support.
    - ahoka 3y ago
      Some examples of those issues would be nice to back up your claim.

green-eclipse 3y ago

Microsoft bought Hotmail back in 1997. Hotmail was powered by Unix servers until 2004, despite MS's best efforts to transition to their own Wintel-powered backend [0]. These things take time. [0] https://news.softpedia.com/news/Windows-Live-Hotmail-Was-Powered-by-UNIX-Servers-until-2004-131323.shtml https://news.softpedia.com/news/Windows-Live-Hotmail-Was-Pow...

yellow_lead 3y ago

> LinkedIn was having a hard time taking advantage of the cloud provider's software. Sources told CNBC that issues arose when LinkedIn attempted to lift and shift its existing software tools to Azure rather than refactor them to run on the cloud provider's ready made tools. I think I need this translated back into tech-speak.

aldarisbm 3y ago
[flagged]
asylteltine 3y ago
That is tech speak. They tried to redeploy existing architecture into azure and it failed
- ghaff 3y ago
  The headline notwithstanding, this doesn't seem like anything particularly Azure specific. They'd likely have had many of the same issues trying to mostly lift and shift to any of the big public cloud providers.
eitally 3y ago
ELI5: For any sufficiently complex enterprise system (e.g. LinkedIn, or Google), any plain vanilla architecture is infeasible for lift & shift. Moreover, the vanilla services may not comply with internal security requirements, or play nicely with internal CI/CD tools, or internal databases / data structures / data processing pipelines / analytics.
- mecsred 3y ago
  No five year old is going to understand a single sentence of that.
  - calvin 3y ago
    ELI40YO Engineer

asylteltine 3y ago

[flagged]

aldarisbm 3y ago
having used azure after long stints with AWS and GCP. I found Resource Groups a much more logical bucket, moreso than anything on AWS/GCP
- jen20 3y ago
  Resource groups are fine, but it is quite unclear why a group has a region when resources with in may be in other regions.
  - ahoka 3y ago
    The object has to be in a database in one of the regions I guess.
    - johngossman 3y ago
      That is correct. Though all that metadata is replicated
      - jen20 3y ago
        Is there a case where it matters to the user which region their group is in if all of the resources live in a different group of regions?
        johngossman 3y ago
        For public regions, I don’t think so. At worst, there might be a tiny bit of extra latency as the orchestrator works across datacenters. But region is also used by Azure Stack and the rules might be different then. I worked on this a number of years ago and if you are really concerned, you should ask support.

upon_drumhead 3y ago

I wonder how the GitHub to Azure migration is going

deleted 3y ago
[deleted]
redrove 3y ago
I have it on good authority they’re trying a lift and shift too and it’s not going well, at least as of ~9mo ago.

duxup 3y ago

Any move to any cloud is going to depend on the environment you're coming from. I've been in on decisions not to use X, Y, Z ... doesn't mean there was anything wrong with them, we just weren't ready for that yet or had different priorities or the ever present weird deal-breaker issue / requirement.

brodouevencode 3y ago
Exactly. The cost to retool for the cloud is not insignificant.
konschubert 3y ago
You have to 1) Be comfortable routing traffic between on-prem and cloud over the internet or at least over a VPN, and 2) avoid the temptation to build your own platform (Terraform templates are a liability, not an asset!) and 3) Move tiny stuff first and bigger stuff later. It’s amazing how many companies fail at that.
- bobthepanda 3y ago
  Doing things the right way and learning from before usually requires a certain degree of humility, and more often than not the person leading these projects is either required to be, or is deluding themselves to be a hotshot who can succeed in a bold new way.
  - konschubert 3y ago
    You mean somebody who posts 3 bullet points on the internet and claims that they solve everything ? ;)
    - bobthepanda 3y ago
      I mean, we invented the term Promotion Oriented Architecture for a reason. It’s like politics; the best person for the job is a sufficiently experienced person who does not want it.

mfer 3y ago

This doesn't appear to be about Microsoft's cloud but rather Public Cloud. The whole migration of LinkedIn from their own data centers to the public cloud (Microsofts) isn't going well. It appears they are still going to operate on-premise for many things. Some things moving or have moved to the public cloud. Isn't this more a shot at the public cloud for all the things than to any specific one?

that_guy_iain 3y ago
I don’t see anything that points to it being a general public cloud issue. And instead they talk about Azure software specifically as something that they couldn’t take advantage of, no?
- mlhpdx 3y ago
  I would not assume that it is a specific Azure problem from that statement. Many, many teams struggle to take advantage of cloud infrastructure because of habits and knowledge retained for operating the existing systems. It’s possible given what they have, I t’s simply best to keep it on premise - at least to some degree. That would likely not be true with a successful re-architecture, but not everyone is up for that.
  - mfer 3y ago
    It may not be about the teams. For example, when you control the data center you can do certain things around performance and scale you can't do in a public cloud. There are so many unknowns about how things are setup that it's hard to know.
carimura 3y ago
Yes I came away with the same thing. It's The Register's modus operandi to use cheeky clickbait titles.

femiagbabiaka 3y ago

Most cloud migration projects at large companies fail. It usually takes 3 or 4 tries at least before all the necessary lessons are learned.

bob1029 3y ago
Our customers are really good at making a total shitshow of cloud and our jobs very easy as a result. Much sentiment over cloud looks to me like stick-in-bike-wheels meme. It's hard because you are making it hard. Strategically, I don't think you try to help someone with cloud until they've burned out their ego trying some grand delusion. Even one bad actor can throw the whole thing off. Cloud is about doing more with less, so committee bullshit is cursed. Once you've got your customer completely humbled by reality, they be willing to listen to reason and you will save so much frustration. We've got a huge one in the pipeline right now. They've been trying to "go to the cloud" for about 5 years now and executive management is ready to reset the entire tech org chart over the lack of progress. Cloud native is the solution but many technologists perceive it as the end of their careers. Anyone pitching a "cloud native" solution that still has container spam managed by the tenant owner is either incompetent or trying to protect their career at this point.
- femiagbabiaka 3y ago
  Sad but very true. When careers have been crafted on the current architecture of a company, it's hard to shake.

ro_bit 3y ago

[flagged]

danielrhodes 3y ago

[flagged]

oooyay 3y ago

Maybe I'm a bit contrarian on this one but once I saw data center, Azure, and the phrase "lift and shift" it filled in a lot of context for me. I spent a lot of my early to mid career participating in these strategies. They don't work. VM images almost always are different in some way, there's something one vendor provides that another doesn't - in general there's enough minute details that add up to make a series of mini-mountains in terms of blockers.

jeffbee 3y ago
Yep, there are always differences. Just one thing I stumbled into recently was one of our program images that has long worked fine in AWS can't start in Azure because something their hypervisor does to the virtual address layout conflicts with the way that we remap .text to a huge page. It is both trivia and a showstopper.
pphysch 3y ago
Yeah, there is a vast gulf between "it works for us" and "every dependency was implemented strictly according to open standards and is therefore seamlessly portable". See also the joke of migrating between "SQL" databases.

lgkk 3y ago

How do you move that much data over to another cloud provider? Without losing data or disrupting the customer? Or do the databases just stay in the data center and not migrate.

wharvle 3y ago
Live replicas (perhaps initialized with a cold backup, initially, if the dataset's really huge), carving off parts of it for separate migration if that's at all feasible, and some expensive folks doing a lot of butt-clenching-worthy activity for an hour or two (unless it goes very poorly...) for the final cut-over, some evening.
ThomasMoll 3y ago
We (when I worked at LinkedIn) did it with ETL clusters, we already had built them out for moving data between datacenters nightly. They would mirror an HDFS cluster, then ran batch jobs to transfer either directly to the outbound cluster or to another ETL cluster in another DC. We used one of our ETL clusters to ship data to MSFT for various LinkedIn integrations, like seeing LinkedIn profile information in Outlook or Office products.
- junto 3y ago
  Which tools were you using for ETL? Or were they completely custom?
tqi 3y ago
Roughly, pick a date and start writing all new data to both, while running etls to backfill data from old to new. Once that is done, you use a feature flag to do a small % of reads from the new system and wait and pray. If nothing major pops up, you slowly ramp up the % of reads until you're confident (as much as one can be) that everything is working, then you move 100% of reads to the new system. Finally, you turn off writes to the old system, clean up the feature flag and remove any old unneeded code. jk of course you never do any of the other stuff after turn off writes to the old system you just leave the FF turned on to 100% and never think about it again :-p

lumost 3y ago

It's incredibly difficult for a mature software business to justify infrastructure and tooling investments. This is why we think that startups are a haven for modern tooling and the largest legacy firms are ... well ... difficult. The last 15 years possibly broke this rule by virtue of low interest rates, enabling the justification of large internal teams focused on modernization efforts which sometimes went as far as moving the state of computing forward. I wouldn't be surprised to see legacy enterprises return to form now that interest rates are 7%

cynicalsecurity 3y ago
On premises hardware can hardly be called legacy. Clouds are way too expensive for startups; they are a milking machine for the big corpo.
- lumost 3y ago
  Startups are typically capex constrained, at least until series C. Clouds are favorable to capex constraints. Having personally tried to be cheaper than the cloud in 2013 through large hardware buys, negotiated contracts etc. I found that the roi relative to a progressively discounting cloud weren’t there.

ksec 3y ago

I wonder what sort of scale do LinkedIn operate in terms of Server count. And Github also under Microsoft seems to be doing fine with on-prem as well. Why force LinkedIN to use Azure?

wredue 3y ago
If I had to guess, there are hordes of businesses out there that maintain operations on prem, and a large lift like this is great for the resume. Of course, I could also be entirely wrong, but I also am not going to pretend that IT resume padding then jumping ship and leaving a shart of an architecture behind doesn’t happen all the time in this industry.
astockwell 3y ago
$$$
rdoherty 3y ago
When I was there it was in the low hundreds of thousands. Probably more as growth was still in double digit percentages per year of user base.
- ksec 3y ago
  >When I was there it was in the low hundreds of thousands. Blows my mind every time I see these kind of numbers.
  - fbdab103 3y ago
    Indeed. According to this in 2022[0], Stack Overflow is still running off of 9 web servers. [0] https://www.datacenterdynamics.com/en/news/stack-overflow-still-on-prem-runs-qa-platform-off-just-nine-servers/ https://www.datacenterdynamics.com/en/news/stack-overflow-st...
    - manquer 3y ago
      Social media org like Linkedin is hardly comparable with SO style KB the R/W workloads are much higher with Social media, than a KB where vast majority of the transactions can be cached, Wikipedia would be a better comparison to SO. Also Linkedin is 100x the size in users : 1B vs just around ~15M for StackOverflow.

thenewwazoo 3y ago

I'm obviously not going to comment on anything internal, and I'm obviously speaking for myself and not the company, but it's worth bearing in mind that this migration was not from "on-prem" in the traditional sense. LinkedIn has its own internal cloud, complete with all the abstractions you'd expect from a public cloud provider, except developed contemporaneously with all the rest of the "clouds" everyone is familiar with. It was designed for, and is tightly coupled to, LinkedIn's particular view on how to build a flexible infrastructure (for an extreme example, using Rest.Li[1], which includes client-side load balancing). There was no attempt to "lift-and-shift" anything. There are technologies that overlap and technologies that conflict and technologies that compliment one another. As with any huge layered stack, you have to figure out which from the "LinkedIn" column marry well with those in the "Azure" column. I personally appreciate LI management's ability to be clear-eyed about whether the ROI was there. [1] https://linkedin.github.io/rest.li/ https://linkedin.github.io/rest.li/

dfxm12 3y ago
Yeah, based on my own experience with AWS and Azure (that has nothing to do with Linked In), my immediate reaction to the headline was, "well, you can be keen on Azure, but "stuck" on AWS for a myriad of other reasons". Reading the article pretty confirmed it.
- whoknew1122 3y ago
  I'm not sure how this has anything to do with AWS. But I guess you can use it to confirm your idea about vendor lock-in if you're bound and determined to convince yourself you're right. For what it's worth, working at AWS, I see a lot of companies who use Entra for AD and then federate into AWS using that.
  - ajcp 3y ago
    I think you misunderstood OPs anecdote and took it a little personally? They were simply saying that much as you can love one platform there are many valid reasons why your deployment on another makes it hard to shift that and recapture that configuration. Nowhere did they intimate it was a fault of the current platforms capabilities...

tjpnz 3y ago

Sounds like they would've faced a similar set of issues moving to AWS or GCP.

joshhart 3y ago

This was cancelled over a year ago - which the articles notes and is old news. It was clear the effort would have needed a very significant push that would have required a large halt in product development and management wasn't willing to stomach it due to high growth in 2020/2021. Which made sense. But LinkedIn revenue growth has heavily slowed with the pullback in tech hiring and they had the space to do it and consider it optimization time. Also as part of Blueshift the plan was to do batch processing first but LinkedIn had a culture belief in colocation of batch compute & storage, which is against the disaggregated storage paradigm we see now. IMO this led to some dragging of feet. Source: Worked at LinkedIn 12 years, am a director at Databricks now.

ThomasMoll 3y ago
Not only that but the Hadoop team literally had the guy who wrote the original HDFS whitepaper. Moving a service with that much in house expertise first never made sense. I worked on one of the original Azure PoCs for Hadoop, even before Blueshift and it was immediately clear that we operated at a scale that Azure couldn't handle at the time. Our biggest cluster had over 500PB and total we had over an exabyte as of 2021 [1]. It was exorbitantly expensive to run a similar setup on VMs, and at the scale that we had I think it would have taken over 4,000 - 5,000 separate Azure Data Lake namespaces to support one of our R&D clusters. I believe most of this "make the biggest cluster you can" mentality was a hold over from the Yahoo! days. [1] https://engineering.linkedin.com/blog/2021/the-exabyte-club--linkedin-s-journey-of-scaling-the-hadoop-distr https://engineering.linkedin.com/blog/2021/the-exabyte-club-...

hasty_pudding 3y ago

As someone who worked at LI. They spent years and god knows how many millions TRYING to move to Azure with the Blueshift project..before pulling the plug. They hired armies of contractors. They didn't stop by choice. They stopped because their tech stack is a giant over engineered unmovable turd.

bbkane 3y ago
As a current employee, there's things I don't like, but the infrastructure is more custom than bad (far better than my last job)
- vasco 3y ago
  Custom is usually bad, just takes longer to reveal the problems. Sometimes companies need to create custom things, the mistake is continuing to invest in them when a community project appears and doubling down for years until nobody in your org knows how it's done "normally" and nobody that you hire knows anything about your stack.
  - deleted 3y ago
    [deleted]
- hasty_pudding 3y ago
  The entire company has no Q.A team. The amount of bugs I troubleshot in that tech stack was staggering just for basic day-to-day stuff. promotion based engineering means an engineer shits something out that looks cool in a demo then bails to the next cool demo project while the rest of us are stuck with the reality of the turd.
bushbaba 3y ago
^ it’s not an azure issue but a LinkedIn issue.
carlivar 3y ago
Didn't LinkedIn create Kafka? Was that some of the overengineering?

jabiko 3y ago

Since LinkedIn is using Kafka quite extensively, I'm wondering whether Azures poor support of Apache Kafka is at least partially responsible for this. Azure is developing its own inhouse message broker that they call EventHubs. And while it is able to speak the Kafka protocol it has some weird limitations and quirks¹. I'm wondering why they are so hellbent on developing their own inferior version of something when they could just use the real deal and host it as SaaS. ¹ No support for KIP-71 (Enable log compaction and deletion to co-exist), limited to 10 topics (Azure lingo: EventHubs) per broker (Azure lingo: EventHubs namespace), mandatory client config parameters that you must not forget to set, even though the documentation just calls it "recommended" (https://learn.microsoft.com/en-us/azure/event-hubs/apache-kafka-configurations https://learn.microsoft.com/en-us/azure/event-hubs/apache-ka...), some weird protocol level incompatibilities that are nowhere documented, for example a consumer group state transitions to "Dead" after the last consumer left instead of "Empty", and auto scaling that only ever scales up, but never down.

jameshart 3y ago
> Azure is developing its own inhouse message broker that they call EventHubs This seems to imply Event hubs is some sort of upcoming beta or something. Event Hubs are one of the oldest parts of Azure IaaS - I think it was launched in 2014?
- jabiko 3y ago
  I didn't want to imply that its brand new or beta. Just that it is still being actively developed. For example the compacted topic support did get into GA in May 2023.
TideAd 3y ago
Lots of Azure stuff is like that. Some big services internally at Microsoft refused to use their Service Fabric product, it couldn't handle the scale & complex deployment needs.
- jiggawatts 3y ago

devaiops9001 3y ago

LinkedIn doesn't have competent people. Anyone who has peaked behind that curtain sees they struggle with very simple things.

dboreham 3y ago
I don't have deep intel, but all the ex-LI people I've worked with or met seemed pretty competent.

risfriend 3y ago

My company migrated to Azure few years back. They can give good bulk discounts but on the flip side the experience with some of their infra and AKS has been choppy at times, with the support team taking time to fix or rca h/w issues. They do come back though. Would love to know how does it compare to long term experiences with other major cloud providers.

bob1029 3y ago
I've been with all 3 major and Azure is the only one that even feels like it has support. Even so, unless you are a big customer on some special agreement, you aren't going to get any red carpet treatment. We pay for the $100/m support plan right now and it's pretty goddamn mediocre. Maybe submitting tickets for "legacy" AD domain services outages doesn't touch the rockstar support team these days. Support quality is probably variable across product team at some level.

FpUser 3y ago

If their stack works fine and I assume it it why the fuck do they need to move it at all?

seanhunter 3y ago
Well they decided they didn't need to. That being said, you can't necessarily wait until things no longer work fine before deciding to move. If you can reasonably anticipate a problem in the future, sometimes even if things work fine today you might be best off moving in order to avoid future pain.
- FpUser 3y ago
  >"you can't necessarily wait until things no longer work fine before deciding to move" You mean cloud is the holy grail where everything works and everything else is a failure waiting to happen? I claim BS.
  - seanhunter 3y ago
    No I don’t mean that at all, which is why I didn’t say that. What I mean is that in planning anything in tech you can’t always wait until problems arise before you take action to deal with them, particularly if the consequences of failure are extreme. The specifics of what is the right setup are going to depend heavily on the circumstances of the use case. Definitely being on cloud isn’t right for everyone, nor is trying to maintain on-prem hardware the right choice for everyone.
nyokodo 3y ago
> why … do they need to move it at all? Because they’re owned by Microsoft and it looks bad if Microsoft doesn’t dogfood its own cloud infrastructure. Plus it’d be more economical long term if they didn’t invest in a parallel cloud infrastructure. They will migrate eventually, just apparently it’s not ready for them yet.
- FpUser 3y ago

Sparkyte 3y ago

Only really benefits a company if they lack the technologies or cloud solution. Not like they are paying anyone to host something besides themselves.

personalityson 3y ago

Isn't LinkedIn owned by Microsoft?

mgaunard 3y ago

why would you ever want to move to cloud if you have a functional non-cloud setup? Cloud platforms charge you much more than what the hardware is worth, the only advantages are that they provide a simplified resource management system (which is mostly an illusion, as any non-trivial system will require you to build in-house tech regardless) and the ability to scale more easily (which is done at a prohibitive cost, so a good reason to switch away if you need to scale up).

JOnAgain 3y ago

This is why AWS is better. They dogfood when it is hard. I remember when S3 and EC2 were coming out. They tried to make us working on the retail website move year after year. Our excuses became their roadmap. We (developers working on not AWS) really didn’t want to move to AWS. It was 100% worse than what we had before. “Network is slow”, “it’s not PCI compliant”, “the drive is unreliable”, “the load balancer is missing a feature”. It took years and years. But they did it. Google and Microsoft don’t have the willpower to force this, and its why they will always be behind AWS. When Google tries to sell you Google Cloud, remember they don’t use it for anything internally. They don’t think it’s good enough. So why should I?

seper8 3y ago
... You think Microsoft doesn't dogfood? Where do you think Teams, Bing, OpenAI, the billion internal systems they use and Xbox are hosted
- hyperliner 3y ago
  I think the point is that the scale of requirement intake that came into AWS from the website is orders of magnitude more than whatever level of dogfooding that Azure might undergo from internal solutions. Not that they don’t dogfood at all, only that the massive amount of engineers moving one of the most complex websites in world’s history into AWS, and AWS taking them in as real input into the product, made AWS a lot better than the old AWS.
- alternatex 3y ago
  Yeah reading that comment felt like the twilight zone. Microsoft is foremost a software company and they host everything on Azure. Amazon's software services offering is a speck of dust in comparison. Amazon cannot dogfood AWS more than Microsoft dogfoods Azure.
  - carlivar 3y ago
    Well, to take it full circle, they don't dogfood LinkedIn on Azure. How about GitHub for that matter? Wouldn't that indicate Amazon can dogfood more?

chucke1992 3y ago

I guess Azure has run out of hardware. AI requires too much computation power.