12 ms·
How Netflix really uses Java
- deleted 3y ago[deleted]
- dphuang2 3y agoalways wanted to work on a system that needs hystrix, I haven't needed such strict uptimes just yet.
- jedberg 3y agoHystrix isn't only for uptime. It also provides some really great observability. Even if you don't have the uptime requirements, the observability can help with debugging (and having better than expected uptime can never hurt!).
- paulddraper 3y agoIt's in maintenance mode now, FYI. https://github.com/Netflix/Hystrix https://github.com/Netflix/Hystrix
- Solvency 3y agoIn theory... would Go be (1) as performant for Netflix's scale/speed needs, and (2) be practical in 2024 from a platform/ecosystem perspective?
- ludovicianul 3y agoJava reached somehow parity with Go performance when using GraalVM and virtual threads. For a Java shop, this will minimize the appeal to move everything to Go as the gains are marginal.
- lbhdc 3y agoI think that depends. One of the advantages that the jdk offers is jit. I haven't seen the inside of netflix, but I am guessing it is similar to other big tech companies where they have a large portion of there instances on all the time, and receiving a lot of traffic. This can be an advantage for the jit. I think there is a great ecosystem for building large systems with go (check out cncf for a ton of go projects targeting the cloud), but a lot of big tech companies open source their java frameworks, tools, and systems. If you are looking for something really specific there may not be an equivalent in go. I think go is great, and it is close enough for me to continue to choose it for this kind of work.
- mseepgood 3y agoNetflix uses that, too.
- paulddraper 3y agoI mean...PHP/Hack works for Facebook's scale, so it's not really a black+white question. Go will give you lower memory usage, less abstractions, and the ecosystem has fewer observability tools.
- iammiles 3y ago…let's say, your TV, or your iOS device will just do 10 network calls to these different microservices. It will just not scale at all. You would have a very bad customer experience. It would feel like using the Disney app. It's just not ideal. That’s some shade.
- kagakuninja 3y agoI'm interviewing at Disney Streaming, lol
- __xor_eax_eax 3y agoFix it!
- abhinavk 3y agoAnd then quit. Reason: My work is done here.
- rco8786 3y agoThere is a bug in the Roku app where if you close it and then re-open quickly it will hang on the profile login screen every, single time. Have to go back to Home again and re-open the app. Seems dumb, but I have 2 little kids and they both like to "pause" whatever show when it's time to turn the tv off. So after the first one does it, I have to do this dumb little dance to get the app to load again so the second one can do it.
- rightbyte 3y ago> they both like to "pause" My son also want to do this. Pause. Then turn the TV off. Maybe they believe the show will keep running with the TV off unless paused?
- 01HNNWZ0MV43FF 3y agoI have sometimes opened YouTube on my Roku to find it skipped part of a video I was watching, maybe because the Roku took time to close down after I turned off the TV by IR. Better safe than sorry. ^S
- rickette 3y agoPrev: https://news.ycombinator.com/item?id=37829395 https://news.ycombinator.com/item?id=37829395
- dang 3y agoThanks! Macroexpanded: How Netflix uses Java - https://news.ycombinator.com/item?id=37829395 https://news.ycombinator.com/item?id=37829395 - Oct 2023 (196 comments) (Reposts are fine after a year or so, but when a story has had significant discussion within the last year or so, that's what counts as a dupe on HN. This is in the FAQ: https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html.)
- torcete 3y agoI never worked with micro services, but I have this question. Do micro services require more and better management? I can imagine that you need to keep track of all these services, along with information about dependencies, code version, API version and others. Of course, monolithic services also require these this sort of management, but it should be less complex.
- frfl 3y agoYou may find this video interesting, https://www.youtube.com/watch?v=5IUj1EZwpJY https://www.youtube.com/watch?v=5IUj1EZwpJY It's about Conway's Law. I think that's worth looking into based on your question. You can look it up on Wikipedia if you don't wanna watch the video and would rather read about it.
- jedberg 3y agoThe short answer is yes. The longer answer is that you will spend a bunch of engineering time on building systems to help manage these things, but if you do it well and you're at a large enough scale, the investment pays off. Some companies spend a bunch of engineering time making their monolith easier to manage (Google/Facebook) and some spend it making microservices work (Netflix/Amazon). Neither one is better, just different paths. Where you get into trouble is trying to go either route without spending the resources to make management better. > I can imagine that you need to keep track of all these services, along with information about dependencies, code version, API version and others. This is the software Netflix wrote to do that: https://github.com/Netflix/eureka https://github.com/Netflix/eureka
- paulddraper 3y ago> making their monolith easier to manage (Google/Facebook) Google has Borg. They literally invented Kubernetees to make their microservices easier to manage. --- Don't confuse mono/microservices with mono/micro repositories.
- jedberg 3y agoI'm aware of that. I was talking about their modified version of Subversion.
- throwitaway222 3y agoI bought into the "RX Java/JS/etc.." years back. Everyone I showed the code couldn't handle it, and we just backed off to other methods, and things worked just fine. RX has some interesting ideas, but from a practical standpoint, at companies not netflix, it just doesn't work
- jedberg 3y agoRX definitely requires some non-standard thinking to do well, but any competent engineer can get it if they want to dedicate the time to it. But you're right, the same things can be accomplished in other ways with less up front effort. But where it really shines is when you hit massive scale. Having RX and its concepts in place ahead of growth makes the growth easier.
- Jcampuzano2 3y agoI think this is where things go wrong. I used RxJS plenty and tried even spearheading its usage at times in some of our apps since it could cleanly solve some problems in what I thought were elegant solutions. I quickly abandoned this after realizing that the upfront ask in terms of dedicating time to learning it for some people is just too much for them to handle. I'd say a decent chunk of engineers just were not competent enough to get it in the first place and then there were those who just refused to put in the time to learn. i.e. there are many engineers who I truly believe do not even want to put in the effort to be competent in the first place. That or false competence takes precedent as well.
- John23832 3y agoEven at Netflix RxJava is not fully accepted. When I was there I wrote a service which called many downstreams. It orchestrated test user creation. I used RxJava because it was being pushed by the platform team, and coupled well with our internal GRPC api handlers. I shared with the team and people complained about the complexity… so I ended up ripping it out. What people wanted was a test user creation service that worked well and could be maintained… not one that was the most efficient.
- mtlmtlmtlmtl 3y agoThis is mostly off-topic I suppose but I recently noticed that Netflix, and Netflix on my TV(WebOS) specifically is absolutely spamming my router with DNS requests sometimes several times a minute, most of them for nrdp-ipv6.prod.ftl.netflix.com. I'm not blocking them btw, so it's not some buggy retry thing going on. Even if I do a hard reboot of my TV and don't start the app, my TV is still happily resolving away. I'm gonna have to set up a local cache or something just to save on my NextDNS quota. Edit: I just asked Netflix support about it. Worse than useless, just kept telling me to either try a different internet connection(no idea why), then they told me to take it up with my ISP and immediately closed the chat. I'm gonna wait an hour or so, open another chat, and tell them my ISP said this was purely a Netflix issue, I suppose.
- wesleytodd 3y agoNot my area of work, but pretty sure that is expected behavior to resolve an ever changing set of services being called out to. Better to do dns resolution than for a call to fail and have to re-resolve while you wait on a loading screen right?
- grepfru_it 3y agoIt looks like the TTL on that address is 24 hours, so likely a problem of the dns resolver, which would point to webos. Have you tried a factory reset recently? (Or root and diagnose the system internally)
- mtlmtlmtlmtl 3y agoMy system is rooted actually, but no idea how to disgnose this. I can SSH in but it's a very limited busybox setup and everything app-related seems to be encrypted.
- NavinF 3y ago> absolutely spamming my router Ah, how many k/second? > sometimes several times a minute WTF? We're talking 0.00003 thousand requests/second? Most apps do several DNS queries per minute. Otherwise every DNS update would require >1 minute to propagate. Slow DNS updates is a massive pain in the ass. FWIW, ISP DNS servers and cheap home routers are both capable of caching DNS responses, updating the cache when they get a query for something that's about to expire, and serving thousands of queries per second. > then they told me to take it up with my ISP and immediately closed the chat This is the correct response. DNS is usually provided by the ISP via DHCP. If their server can't handle "several times a minute", the ISP would be at fault 100% of the time
- Nican 3y agoI have been out of the loop with Java. Is Virtual Threads the answer to asynchronous I/O? (Much like Go/C# async/node.js?) That looks like an interesting solution to support asynchronous I/O without breaking all the APIs, and having the async/await mess that C# created.
- CharlieDigital 3y ago> ...the async/await mess that C# created What do you find messy about it? Seems fairly straight forward, IME.
- jakewins 3y agoNot sure what the poster above was thinking of, but it seems kinda the same as every other language that’s adopted it - powerful, but footguns abound. I ran some async C# in a debugger - in Rider - the other month, and the debugger just goes off the deep end. Does C# have the same issue Python does with accidentally calling blocking code? In async Python - which I mostly quite like actually - it’s terrifying to bring in a library because if you’re unlucky it does some blocking network call deep inside that stalls your whole app..
- Kwpolska 3y agoPython's async works on a single thread, C# uses a thread pool. Calling a blocking method is not ideal, but doesn't ruin everything, and it's easy to hand that work off to a separate thread by using Task.Run.
- CharlieDigital 3y ago> Does C# have the same issue Python does with accidentally calling blocking code? This can't really happen in C# except maybe if you are working on a GUI thread and you make the mistake of running blocking code on a GUI thread. For APIs, console apps, and such, it's not a concern for actual parallel-concurrent code. Of course, if you write a standard non-parallel `for` loop that has blocking code, it's going to block in a console app as well if you don't run it on a thread. But I think that once you do enough JS/TS or C#, `async/await` doesn't feel very onerous if you have the basic concept.
- swozey 3y agoI've deployed multiple open sourced netflix java services and each one of them was archived and no longer maintained according to the repos. Are they still being developed internally and the foss repos are basically without the secret sauce? Or are these projects "done"? Did Netflix move on to replacement tools? I'm referring to things like hystrix and .. a queue system I can't recall right now. This was a long time ago but that queue system was a bit of a rough roll out and IIRC I had to make a fair amount of code changes to get going. Not biting the hand that feeds me, just curious about how netflix treats projects like those. I'm usually extremely apprehensive about deploying archived things but a few teams wanted to use them. I don't know if things are archived because they're done, perfect, terrible, cooked, deprecated, insecure.. edit: Oh I hadn't looked at it in a long time, the hystrix repo explains it well on the readme. I'm not sure if the other project I used did that.
- wiradikusuma 3y agoMeanwhile, if you're building something smaller than Netflix, I'm writing a book just about that (https://opinionatedlaunch.com/ https://opinionatedlaunch.com/). It's about mobile apps, but I talk about backend at great length, especially since my background is Java. The book is called "opinionated" because I cover Quarkus (https://quarkus.io/ https://quarkus.io/), monolith, Fly.io, and no K8s. No fancy stuff to pad your CV, just to get things done the simplest way possible, with the least headache in maintenance (I'm lazy).
- taffer 3y agoWhy Fly.io if you want to build a monolith? Compared to Heroku or Render, to me Fly.io seems to be more aimed at edge computing.
- wiradikusuma 3y agoI want Heroku experience without paying Heroku price, so Render and Fly.io were an option. But when I evaluated them, Render didn't support loading my own docker image, so it's incompatible with my build process and CI/CD. The book is a living document, and I always try to find easier/cheaper/faster/better ways of doing things, so I'll definitely check Render again!
- anurag 3y agoRender supports loading your own Docker image now: https://docs.render.com/deploy-an-image https://docs.render.com/deploy-an-image
- geodel 3y ago[flagged]
- Analemma_ 3y agoI mean, I upvoted the story because Netflix is pretty well-known for being a high-intensity dev environment where people are regularly PIP'd and dismissed if their engineering output doesn't measure up - of all the companies which do the eye-rolling claim of "we only hire 10x engineers", Netflix is one of the only ones for which it kinda holds water. And yet, despite this, their stack is boring old Java+Spring. That seems like a useful lesson for all the people who demand that high-performance teams must use the New Shiny.
- kagakuninja 3y agoI'm a Scala developer, so we get the shiny on top of boring old Java.
- deleted 3y ago[deleted]
- elwell 3y ago[flagged]
- deleted 3y ago[deleted]
- dventimihasura 3y agoHas anyone else used Netflix DGS outside of Netflix to build GraphQL servers? I'm wondering what the experience was like. Full disclosure: I work at a GraphQL company. But still, I'm genuinely curious. I was a longtime back-end Java developer, and I'm wondering what the experience is like these days.
- TYPE_FASTER 3y agoNetflix Hollow is pretty great: https://hollow.how/ https://hollow.how/