8 ms·
If your reasoning is productivity above everything, therefore the only solution you can think of is Python, I call BS. As a Googler, I could relate countless s
by titzer 10y ago
If your reasoning is productivity above everything, therefore the only solution you can think of is Python, I call BS.
As a Googler, I could relate countless stories of systems that were cobbled together in Python and then scaled up with massive investment, then later rewritten in C++ or Go, and generally ran with 100x fewer resources. Hell, from what I can tell, almost all of YouTube was written in Python and has or is still being rewritten in C++, Java, and Go, saving XX millions of dollars per year of CPU time.
Sure, prototype with Python. But by the time you get to scaling out a huge service, it's probably time that you rearchitected the system anyway, and you should spend the engineering effort to make it efficient.
- pbreit 10y agoI don't think there's much sense in using Google as a proxy for anything really. The scale at which Google operates is in another universe.
- nhumrich 10y agoIn no ways am I saying Python is the only solution. Moreso that Python is A solution. Sure, other languages and things exist. After all, no silver bullet. Also, you are saying that google hit a scale where it really WAS expensive to use python. Twitter had a similar issue with Ruby. That makes sense for a huge company like google, twitter, etc. But for most of us, our companies/projects will never get the scale that youtube has.
- rakoo 10y ago> then later rewritten in C++ or Go, and generally ran with 100x fewer resources This kind of story comes again all the time, but the most important detail is always missing: how much of that 100x can be attributed to the language, and how much can be attributed to now I know the domain better, I know where the pitfalls are, how to model the problem, and when I need to optimize ? Because arguably the second one can bring a lot.
- jackmott 10y agoIn my own deep experiments with simplex noise generation in c++ and C#, C++ allowed me to go ~10x faster and it was 100% due to the 'language' (and associated ecosystem). Domain knowledge for me is higher in C#, both were optimized as far as I could figure out. There are dozens of cpu instructions you can't even make happen in most runtime GC languages. It is banananastown.
- thesmallestcat 10y agoYou could get similar performance with a C or C++ extension, or by using Cython. 99% of your application is probably not generating simplex noise or doing anything CPU-bounded like that.
- std_throwaway 10y agoIt's about half and half, 50x comes from the language and 50x comes from experience. The important part is to add those factors together by good management.
- poikniok 10y agoThe math here doesn't really make sense, factors multiply, so if there were two 50x factors the total improvement would be 2500x. In other words if one factor is 50x, the other one must be 2x to get 100x total.
- deleted 10y ago[deleted]
- khedoros1 10y agoThis is useless pedantry. You know what they were trying to say: The improvement can be attributed equally to using a more suitable technology and to better knowledge of the problem domain.
- staticassertion 10y agoI've written 1 to 1 (algorithm) code from Python to Rust and seen orders of magnitude performance/ memory difference. This was not a 'rewrite it with lessons learned' this was 'rewrite it as similarly as possible'. It is the language (and runtime).
- sfilargi 10y ago> almost all of YouTube was written in Python and has or is still being rewritten in C++, Java, and Go, saving XX millions of dollars per year of CPU time. If you reach YouTube scale, maybe the CPU savings > developer time. But I don't think you will reach that scale if you start with C++. Completion will outpace you in no time.
- jackmott 10y agoThere are middle grounds between c++ and interpreted Python that have the same productivity as python and better performance.
- nhumrich 10y agoCorrect. Like writting in python, but optimizing the "bottlenecks" in C/C++/Cython
- kajecounterhack 10y agoOr choosing a language that's inherently somewhere in the middle (go).
- jackmott 10y agoWhy not just compile/JIT the python, always? I feel like I'm taking crazy pills! :) Even when you write the hot bits in C++ you still have a python interpreter thrashing the CPU caches and using up ram. and now you have to context switch in your head more often, and the build process is more complex, and usually there is a cost when you do the FFI.
- poikniok 10y agoIf you are talking about PyPy (I presume you are?), then two issues are that it can in some cases be slower, and also C libraries such as Numpy / SciPy are not simple to use with PyPy (I think some of Numpy has been translated, but there are many useful C libraries for Python).
- 10y ago
- rdtsc 10y agoBut if those system were developed in another language, maybe by the time they were released there would not be any market demand for it anymore. I've seen it happen where everything is meticulously performance optimized in C and by the time it is released it is fast, snappy, scalable but by then the market has moved on and nobody wants to use it anymore. Often "We moved from X to Y because X is slow" is seen as a failure of X, but it is not. It could mean the feature was deployed quickly, caught on and has customers which want to use it. So it can be a testament of how productive X is.
- kajecounterhack 10y ago+1, not speaking for Google With large systems you want assurances at compile time, something that Python can't give you much of. When you have a huge system with many modules that interact in ways you can't expect anyone to map out without staring at a lot of code, it pays to have a stronger type system and fewer runtime errors. For experimentation and small-scale work, Python's great. Single client application? Fantastic (e.g Dropbox). Giant web service or distributed application? Starts to show its ugly sides quickly.
- shas3 10y agoWell, there are two ways of looking at this: 1. Decision to use Python created unnecessary extra work (your point); 2. Executing (not just prototype) with Python, then re-writing and re-optimizing your code in a 'leaner'/lower-level language is a valid approach given that's how Youtube and other highly successful Google products and services evolved. I think both are valid approaches depending on your application, available resources, nature of incentives, and the speed at which your market moves.
- lawnchair_larry 10y agoI bet the youtube founders would disagree with you. They probably do not regret launching it quickly in python, selling it to Google for over a billion dollars, and letting Google spend the next decade rewriting it.
- kajecounterhack 10y agoIf they didn't sell, they would have had to do the rewrite. There are plenty of future YouTubes who don't want to be acquired and instead are looking to be long term companies. They'd do well to avoid the pain and cost. There are ways to achieve velocity and maintainability, they're not mutually exclusive.
- guohaochuan 10y agoAs others' comments, it's totally different maintaining a successful business from launching one. The decision should be based on what value the language can provide. I'm in a very python heavy company, and maintains the python architecture that are being used on hundreds of servers. The ugly part(lack of restriction, multiprocessing, distribution) quickly shows why it's not suited for very large system. But I'm curious at which point, we would be so annoyed as to decide to rewrite the whole stuff.