11 ms·
LinkedIn adopts protocol buffers and reduces latency up to 60%
- rekttrader 3y agoIn other news binary serialization is smaller.
- deleted 3y ago[deleted]
- euos 3y agoJust use gRPC.
- LatticeAnimal 3y agoHow is gRPC related? You send JSON or Protobuf over gRPC
- ShadowBanThis01 3y agoNo Swift support.
- upon_drumhead 3y agohttps://github.com/grpc/grpc-swift https://github.com/grpc/grpc-swift
- ShadowBanThis01 3y agoThanks. It's not in their list of supported languages: https://grpc.io/docs/languages/ https://grpc.io/docs/languages/
- gcampbell 3y agofrom the article: > Based on the learnings from the Protocol Buffers rollout, the team is planning to follow up with migration from Rest.li to gRPC, which also uses Protocol Buffers but additionally supports streaming and has a large community behind it.
- DedlySnek 3y agoFrom rest.li's github page[0] - At LinkedIn, we are focusing our efforts on advanced automation to enable a seamless, LinkedIn-wide migration from Rest.li to gRPC. gRPC will offer better performance, support for more programming languages, streaming, and a robust open source community. There is no active development at LinkedIn on new features for Rest.li. The repository will also be deprecated soon once we have migrated services to use gRPC. Refer to this blog[1] for more details on why we are moving to gRPC. [0] - https://github.com/linkedin/rest.li https://github.com/linkedin/rest.li [1] - https://engineering.linkedin.com/blog/2023/linkedin-integrates-protocol-buffers-with-rest-li-for-improved-m https://engineering.linkedin.com/blog/2023/linkedin-integrat...
- Manjuuu 3y agoYep, REST make no sense between microservices, for what are basically RPCs.
- WWLink 3y agolol. at some point in time I ran protocol buffers over kafka. Fun times. I liked avro better though.
- MarkMarine 3y agoI’ve seen protbufs as a custom encoding in the database before. Don’t do this, it’s a terrible terrible idea.
- malkia 3y agoIt depends. I've seen it at Google - in bigtable, megastore and spanner - there are quirks - for example you (may) need a registry with all proto(s) the db may contain. In our case (Ads team), any other team using us could've placed their proto in our db, and during the ETL process (bigtable -> capacitor columnn db) you had to know the protos down to the leaf, as each leaf field gets made into a column (based on allow/disallow lists as we had at some point over million columns). It's crazy, but worked.
- GeneralMayhem 3y agoUnfortunately, it really only works ergonomically at Google, because you need the global protodb and a proto-aware SQL dialect to make it usable for humans, and you need proto-aware indexing to make the retrieval speed usable for machines if you ever query by the contents of a proto field. Externally, the closest I've been able to get is using JSON-encoded protos as values, but that comes with its own problems.
- ShadowBanThis01 3y ago"They also didn’t want to limit the number of supported language stacks" But that's exactly the problem with protocol buffers. In typical Google fashion, they never bothered to create support for the native language of their platform: Kotlin. And that was just the one example we encountered on a project. If your language isn't supported by protoc, then what? Also touted was protocol buffers' "future-proofing" or resistance to version changes. We never realized this alleged benefit, and it wasn't clear how it was supposed to be delivered. I dreaded the opacity of the format, and expected decodes to start failing in mysterious ways at some point... wasting days or weeks of my time. But I have to admit that this rarely if ever happened. In our case we were using C++ on one end and Swift on the other. I don't get this rationale in the article: "While the size can be optimized using standard compression algorithms like gzip, compression and decompression consumes additional hardware resources." And unpacking protobufs doesn't?
- skybrian 3y agoCould you use the Java implementation of protocol buffers from Kotlin?
- joe_fishfish 3y agoYou absolutely can. You can also use Square’s excellent Wire library.
- MarkMarine 3y agoYou can but you’re going to have ? EVERYWHERE, which does drive the search for a native implementation that doesn’t make you deal with null everywhere
- billjings 3y agoIs there another way you're supposed to represent optionals in Kotlin?
- drdvehn 3y agoGood. latency has always been a problem at LinkedIn. for me, as a recruiter, I want to send friend requests and job vacancy to a bunch of random users in the blink of an eye. This helps. Thank you LinkedIn.
- jfghi 3y agoShhh, don’t want this to be a new ai-supported feature as a service
- agilob 3y agoDon't worry this gap will be filled by another microservice!
- heavenlyblue 3y agoIndeed, with LinkedIn and come browser console features I can accept as many friend requests from as many recruiters I want in a blink of an eye.
- YetAnotherNick 3y agoLinkedin didn't reduce client latency by 60%. Instead they reduced latency of some microservice, which used to take ~15ms, by <10ms, which is not bad but the claim is flat out wrong.
- nahseeu 3y agoIt’s incomplete, not wrong. You inferred more than you should have from the title.
- noveltyaccount 3y agoAgreed. And "by up to" is a poor metric. What about P90 or even average?
- iamflimflam1 3y ago“50 thousand API endpoints exposed by microservices” Feels that their problem may just be they went a bit too far on the microservices band wagon.
- cbb330 3y agoSpeaking as a current engineer at LinkedIn, Big Tech companies will often conflate micro service with large distributed system. These services are by no means at all micro.
- bingemaker 3y agoCurious to know how many macroservices or monoliths LinkedIn has currently
- whatshisface 3y agoOur interlinked system of distributed monoliths includes one in AWS region af-south-1, another on the Moon, and a third orbiting Jupiter. All of these services are yours, except for the fourth on Europa. Attempt no POSTing there.
- jraph 3y agoOk for Jupiter and Europa, you need some redundancy and to be close to your users, but AWS would is a bit unexpected with LinkedIn belonging to MS. Then again, GitHub still uses AWS IIRC.
- midnitewarrior 3y agoConvert all of my JSON to protobuf to get 60% performance improvement? I'm sorry Dave, I'm afraid I can't do that.
- olliej 3y agoI feel that wholly depends on how you’re classifying those. What makes a service micro vs macro?
- cmckn 3y agoI wonder why they didn’t use MS’ protobuf clone, bond: https://github.com/microsoft/bond https://github.com/microsoft/bond When I was at MS, it was definitely preferred over protobuf. IIRC it was also used internally as the wire format for gRPC. I guess LinkedIn is still kind of doing their own thing.
- wilg 3y agoA likely reason is that Bond does not support their target languages, listed in the article as "Java, Kotlin, Scala, ObjC, Swift, JavaScript, Python, Go". Of those, Bond seems to support only Java and Python.
- kaashif 3y agoIf it's available in Java, then it surely is usable from Kotlin and Scala. The others I have no idea about though.
- basementcat 3y agoBond over gRPC appears to have been deprecated. https://github.com/microsoft/bond/issues/1131 https://github.com/microsoft/bond/issues/1131
- cmckn 3y agoI think that’s referring to some specific libraries by MS that provided a batteries-included gRPC + bond setup. The standard gRPC libraries still allow you to plug in whatever serialization format you want, including bond. But based on the complete lack of activity on the issue, gRPC + bond doesn’t seem like a popular choice. :)
- duckmysick 3y agoHow is it different from Google's protobuf?
- throwaway914 3y agoWish they'd adopt a code of conduct but at least they'll be able to mine my phone contacts 60% faster. I always feel held hostage by LinkedIn. I have to write a resume for every posting, and update LinkedIn and connect with every working person I've ever met on there. Otherwise I don't exist. What a plague of a company.
- hdjjhhvvhga 3y agoA piece of Anecdata: I managed to live around and still have no problems finding a new job. Yes, some job application sites have a mandatory "link to linkedin" field but I just ignore these.
- Raed667 3y agoYes you can live without it. But you'll be missing you whole lot of jobs that are "linkedin only" in the form of posts, DMs and people spontaneously discovering your profile and reaching out. Some people will be fine finding jobs without it, a lot won't be that fortunate.
- olliej 3y agoI feel “LinkedIn only” is a class of job that many people aren’t interested in (biz/fintech maybe?). I’ve literally never seen a LinkedIn only job for a field I was interested in let alone a job - but I don’t work biz/fintech.
- rusticpenn 3y agoI am not in either of the fields. I get "confidential" offers from headhunters usually on Linkedin or Xing.
- berkes 3y agoI've found my most interesting jobs on LI. Sure, several of them, I could have found elsewhere, e.g. on a dedicated job board. But I wasn't at those other job boards, nor was the recruiter who brought them onto my radar. If I had avoided LI, I would have missed out on some of my best gigs.
- achalkumar 3y agogreat
- raminf 3y agoWhat happens if you need to dynamically discover a service and invoke it? Or does it only work if both ends are known and you can push out change to both end of the wire? Also, wouldn't something like MessagePack, ION, RION, or CBOR give you payload compression over JSON?
- AtNightWeCode 3y agoI always try to avoid these special formats. Most people have not used them and even if there are claims that the tools work on different platforms there are often some quirks. Another approach is to add compression and tweak it for performance.
- EspressoGPT 3y agoThe target service must be known and compatible. Looser coupling is sacrificed for higher performance.
- fnordpiglet 3y agohttps://www.tbray.org/ongoing/When/201x/2019/11/17/Bits-On-the-Wire https://www.tbray.org/ongoing/When/201x/2019/11/17/Bits-On-t... One thing about protobufs in a highly interconnected ball of mess, good luck reving the protocol in any non trivial way. Schema-less encodings such as those you mention (as well as bson, etc) are really advantageous for loosely coupled interfaces and graceful message format migrations. IMO (opinion!) protobufs is popular simply because of the Google cargo cult where anything Google tech is slavishly adopted without a great deal of introspection on applicability to the specific situation.
- bruce343434 3y agoWhat happens when you don't need to do that?
- rkagerer 3y agoWhy did it take this long? Real question.
- nottorp 3y agoStill feels extremely bloated though. Maybe their internal latency?
- hknmtt 3y agofor intra-service communication gRPC makes the most sense. the only problem with it is the code it generates. personally I am speaking from Go point of view. am not sure how other languages fare but the Go code is as far from optimal as humanly possible(essentially everything is a pointer and schema is a gzipped blob). not to mention the issues of empty map or slice vs nil that is big pain-point in Go serialization in general.
- dlahoda 3y agowhy it makes sence more than for example using http3 with avro payload and data structures generated from json schema like swagger?
- AtNightWeCode 3y agoAvro does not even come with deserialize to object in the libs. A terrible choice for interoperability.
- gazpacho 3y agoThe Python code is much, much worse.
- wg0 3y ago> 50 thousand API endpoints exposed by microservices. So feature wise, there's profile page of each user, there's searching page with filters, there's inbox, some recruitment tools I'd imagine, job posting, apply to the job and such along with search and recommendations for that too plus analytics for each of the component above certainly there would be some internal advertisement management platform. What else I'm missing? So 50 thousand endpoints for all that?
- ThalesX 3y agoStartup CTOs after reading the title: we gotta build from the ground up with Protobuf or we won't have any customers!
- rewmie 3y agoThe article itself illustrates this nonsense attitude by mentioning 60% latency drops but a) what does that translate in ms, b) were those savings relevant at all? A cursory glance over the article leads to some references to P99 latencies dropping from around mid 20ms to around mid 10ms. Those are irrelevant. Does the 1% care if their requests take 10ms more to fulfill? From the surface this sounds like meaningless micro-optimization. I'm glad someone got to promote themselves with this bullshit though.
- hellcow 3y agoLatencies add up. 1 end-user request may have lots of requests in the backend which run in serial… especially when they have “50,000 services.”
- daxfohl 3y agoThe article acts like this was an atomic switch, but certainly it was rolled out service by service over more than a year, in which case the comparison is grain of salt.
- Dwedit 3y agoNice to see people realize that unnecessary conversion to and from text is unnecessary.
- jonathankoren 3y agoWell this is bizarre. They were using a binary JSON and protobufs 9 or 10 years ago, so I don’t know what the hell happened at that company.
- mkl95 3y agoSome "rockstar engineers" who write tons of logic in their controllers are shivering right now.
- 19h 3y agoTalked to a few LinkedIn engineers a while ago and was astonished at the size of some payloads they’re pushing over Kafka.. in almost all production deployments I’ve done with different teams in the past once we got rid of the big payloads and stored them on block storage and instead used pointers in the messages most pressure issues disappeared — switching to protobuf sounds like an attempt to decrease the payload sizes without offloading the huge payloads… which almost never works because hey if we’re faster now why not send more messages? That’s how it always goes…
- not_enoch_wise 3y ago"We switched to $BUZZWORD and reduced $LOOSELY_UNDERSTOOD_METRIC by $STIMULATING_NUMBER !" Never tell them it's just because of all the technical debt you finally had organizational will to pay off.
- ivix 3y agoI think this is the correct answer. To fix technical debt you often need a sexy sounding cover story.
- meisel 3y agoTypically, the difference between equivalent JSON and Protobuf payloads that have been gzipped is quite small. It can even be that the gzipped Protobuf payload is larger than gzipped JSON one. Granted, there's less data to decompress out of it, which may mean a faster decompression stage, but I'm not sure that's a bottleneck in the first place. So, do they not compress requests between microservices? Otherwise, how did they see such a reduction?
- deleted 3y ago[deleted]
- dilap 3y agowhy not flatbuffers, if youre looking to minimize wasted cpu?
- the-alchemist 3y agoFirst instinct is: Duh! Of course going from a verbose text format like JSON to _any_ kind of binary format will lead to decreased everything. Second instinct was: Wait, were they using HTTP/2, HTTP/3? What JSON parser were they using? One of the fast ones, or one of the slow ones? Third instinct was: There is not enough information in the article to conclude "60%" of anything. It's not an apples to apples comparison. It's just an attention-seeking headline. Then again, if it wasn't an attention seeking headline, it probably wouldn't have made it to HN. ;)
- randito 3y agoAt a company that large and that diverse, the biggest gain in using protocol buffers would be having an official schema and versioning for the messages. Serialization is nice (but not 60% improvement nice). But having some structure around messages is the big win.