16 ms·

It's extremely hard to agree with Linus on that. One problem in his argument is that he believes that everybody has a kernel hacker mindset: most today's develo

It's extremely hard to agree with Linus on that. One problem in his argument is that he believes that everybody has a kernel hacker mindset: most today's developers don't care about environment reproducibility at architecture level. The second problem is that he believes that every kind of development is as platform sensitive as kernel hacking, and he even makes the example of Perl scripts. The reality is that one year ago I started the effort to support ARM as a primary architecture for Redis, and all I had to do is to fix the unaligned accesses, that are anyway fixed in ARM64 almost entirely, and almost fixed also in ARM >= v7 if I remember correctly, but for a subset of instructions (double words loads/stores). Other than that, Redis, that happens to be a low level piece of code, just worked on ARM, with all the tests passing and no stability problems at all. I can't relate to what Linus says there. If a low level piece of code written in C, developed for many years without caring about ARM, just worked almost out of the box, I can't see the Ruby or Node application to fail once uploaded to an ARM server.

mcv 8y ago
I'm a web developer, and for years now I've been developing on either Mac or Windows, but deploying to Linux servers. Or possibly totally different kinds of servers. I don't care much about the architecture of the server as long as it runs my code. Give me Apache, JVM, node, and the necessary build and deploy tools, and there's nothing I can't run on it.
fmajid 8y ago
Linux' point is that there is hardly any developer-class ARM machines available, just RaspberryPi-class SBCs that use mobile SOCs with the performance of a 2012 vintage smartphone, nothing with the grunt of a Qualcomm Centriq or Cavium ThunderX.
013a 8y ago
It seems likely that, simply, times have changed. There was a time when being on the same platform as the deployment environment was super important, but nowadays the tooling has gotten so much better that it matters a lot less. The proportion of people still writing C code on a day to day basis has dropped... well to pretty much a rounding error. The bigger issue is really that ARM servers aren't that much cheaper than x86 servers today, and its very likely a lot of that difference in cost is just Intel's synthetic market advantage that would disappear if ARM actually started becoming a threat (which has already started happening due to AMD becoming a threat). Phoronix did a synthetic benchmark of AWS's A1 instances, versus the C5 Intel and C5A AMD instances [1]; they're nothing special at all, even with price taken into account. Maybe that'll change in the future, but now that AMD is in a competitive state, that's pushing Intel into high-gear and its hard to say that ARM will have any effect on the server market in the short-term. [1] https://www.phoronix.com/scan.php?page=article&item=ec2-graviton-performance&num=5 https://www.phoronix.com/scan.php?page=article&item=ec2-grav...
- antirez 8y ago
  > but nowadays the tooling has gotten so much better that it matters a lot less I think that that's the point. Portability to platforms with a strong tooling and usage base even in a different sector is ok and safe. The problem is when you try to do something like x86 -> Itanium or alike, that could take some time to stabilize.
- WorldMaker 8y ago
  > There was a time when being on the same platform as the deployment environment was super important Which is also interesting because there was a time before that where being on the same platform as the deployment environment was sometimes considered nigh impossible, such as the early days of the "microcomputer" revolution where a lot of software was written on big iron mainframes to run on much more constrained devices (C64, Apple II, etc). It's interesting to compare the IDEs and architectures of that era and how much cross-compilation has always happened. There doesn't seem to be a lot of computing history where the machine used to build the software was the same machine intended to run the software, it's the modern PC era that seems the unique inflection point where so much of our software are built and run on the same architectures. (A lot of the modern tools such as VMs like the JVM and CLR are because of the dreams and imaginations of those developers that directly experienced those earlier eras.) It's interesting how that tide shifts from time to time, and we so easily forget what that was like, forget to notice the high water marks of previous generations. (Even as we take advantage of it in other ways, we cross-compile to mobile and IoT devices today we'd have no way to run IDEs on, and would rather not try to run compilers directly on them.)
  - rbanffy 8y ago
    I know some software was written on minis to run on 8-bit computers, but I have a hard time imagining that as the norm. My Apple II dev rig was two computers, one running development tools and one to test and they were two because running my software wasn't possible on the development machine without rebooting and loading all the tools took 30 seconds - a painful eternity in Apple II terms.
    - 8y ago
otterley 8y ago
I agree with you. For many users of the cloud -- especially serverless application developers -- the actual hardware on which their software runs is a black box. In fact, if the serverless application consists solely of scripts or precompiled bytecode and doesn't contain architecture-specific binary code, it could likely run on arbitrary hardware with any supported ISA and users wouldn't know the difference.
mattnewport 8y ago
A low level piece of code written in C seems likely to have less portability issues than something sitting on top of many layers of abstraction. The thorny problems that show up deploying on an environment that's not identical to the development environment are often the result of unexpected interactions in the stack of dependencies. This is why containerization is a thing. Platform portability issues have got easier with better adherence to standards and where you have largely the same code running across different ISAs (and no endianness issues between x86 and Arm) but the popularity of things like Docker suggest many devs do care about reproducible production environments.
boris 8y ago
What hardware do you use for development and for testing? Can you elaborate on your setup, ARM-wise?
iveqy 8y ago
I once had a bug developing an IP phone. Connecting to one server worked but once on site with the customer connection to their server didn't worked, even though the servers where identical. It turned out that the power supply on their server was malfunctioning sometimes delivering too little power. Specially when taking the code paths my IP phone triggered. The server software was built in php. It's not often you start looking for a bug in PHP but end up switching a capacitor in the PSU. My point is that even if your write in php, php is running C libraries, that is running ASM that is running hardware and every part of this chain is important. There's no such thing as "works everywhere", it's just "have a very high chance of working everywhere". (off-topic) thanks for the sds library. I'm a heavy user of it.
- antirez 8y ago
  Yes but the platform is just one of the unknowns at the lower level. If the tooling is fine, the C compiler is very unlikely to emit things that will run in a different way, it is much simpler to see software breaking because of higher level parts that don't have anything to do with the platform like: libc version, kernel upgrade, ... About SDS: glad it is useful for you!
- noah256 8y ago
  I’d be really interested to hear more of this story! How you isolated the problem down to the level of the PSU is going deeper into the machine than I’ve personally been, so this story could be a great teacher.
  - iveqy 8y ago
    It sounds way cooler than it was. For some reason I openened up the server box and during a reboot I saw the LED on the motherboard flickering in a way I didn't expect. So we tried to change PSU and then everything worked.
sipos 8y ago
On the other hand, it's easier to not have to think about it, even when it doesn't matter 99% of the time, and run the exact same container as you tested locally on the server. I feel like this shouldn't matter really, but people are amazingly lazy/developer time valued highly.
asadkn 8y ago
Depends on general confidence level and how mission critical your deployed software is. For instance, I will always prefer to run PHP, Ruby or Python on Linux servers. But on the client side, I have faith in Electron Apps to run cross-platform without issues. I wouldn't be comfortable with an underlying architecture change to ARM for at least years to come and the usage decision would be based on general consensus on reliability that follows.
- antirez 8y ago
  I would more comfortably run my code developed on x86/Linux on ARM/Linux than on x86/FreeBSD for instance... Platform is just one unknown and not the worst if the tooling is good IMHO. Consider that complex software under Linux/ARM now has a ten years history at least, with numbers (mobile) that are not approached by any other thing on earth.
readittwice 8y ago
I guess the situation is similar to containers and static linking that got popular for server deployments quite recently. They allow to develop, test and deploy your application with its dependencies in the exact same version - the exact same binary. Although usually this works just fine even with minor differences in the dependencies, a lot of developers deem it worthwhile to preclude such issues. If you now use ARM in production, you can't use the same binary file anymore. There might be different implementations depending on the architecture in some library you use. Also even with higher-level languages like Java it is possible to observe ISA differences: e.g. memory ordering.
AnimalMuppet 8y ago
I'm with Linus on this one. I had the pleasure (?) of working on a C/C++ codebase that compiled on Windows and ten different flavors of Unix. It was all "portable", but all over the place there was stuff like #if defined AIX || defined OSF1 short var; #else int var; #endif And to get it right, you had to compile it on all the platforms and fix all the errors (and preferably all the warnings). Yeah, cross platform is never as simple as same platform.
- musicale 8y ago
  Single platform cross-development (e.g. iOS) is a lot easier.
- DerekL 8y ago
  Yes, cross-platform development is more work, but that example you gave is the wrong way to do it. It's better to abstract the differences using typedefs, functions, macros, etc., and keep the platform switches in a few isolated places in the code.
- wahern 8y ago
  How many years ago was that? POSIX compliance has come a long way and most of the proprietary vendors (the ones with all the corner cases) are gone. These days not only do platforms like AIX and Solaris have much fewer corner cases, they're even adopting Linux and GNU extensions wholesale. Anyhow, most people can ignore these altogether. Portability between Linux and the BSDs is much easier. macOS is the biggest outlier in terms of corner cases yet in many ways the best supported thanks to the popularity of Homebrew. C++ is a different matter, but C++ portability is a headache even if you stay on Linux. Likewise, trying to maintain OS-level portability of monolithic codebases between Windows and Unix is a fools errand, which is why Windows Subsystem for Linux (WSL) is likely to only get better.
  - AnimalMuppet 8y ago
    How long ago? 15 years - and it was at least a decade-old code base. But I had to take some new code that was something like Windows-and-Sun-only, and port it to run on all of the other architectures.
sangnoir 8y ago
I don't think you are not really disagreeing with Linus - he's not saying ARM is not viable - he is saying it will not win. With your current setup (cross-compiling), are your ARM executables more performant than x86? Or do they have any other advantage at all over x86? Without an advantage, ARM can't possibly win. Having a cheap, viable ARM-native development platform drastically increases the chances of ARM-only killer apps to exist, this would be an advantage over the currently dominant x86 (just as there were Windows-only and Linux-only killer apps that cemented their ascent). However, if everyone is cross-compiling due to the cost, it means ARM will always be a secondary platform (at most)- it can't win by being the Windows Phone of platforms. [edited for clarity]
- bryanlarsen 8y ago
  He's not saying it won't win, either. He's just saying that for it to win, it needs a viable dev platform. Which, if you reverse cause and effect is blatantly obvious. If ARM comes anywhere close to viable enough to be "winning", there will be a good market for dev platforms, and somebody will step in and fill the need. Heck, some are even arguing here that the Pine64 already meets that need.
  - jeffdavis 8y ago
    It's not simple cause and effect. Servers cause compatible dev boxes and dev boxes cause compatible servers.
  - mntmoss 8y ago
    I'm definitely on the side of ARM(and RISC-V and other new architectures for that matter) getting "wins", because the modern environment is displaying the signs of a low-layer shakeup: * New systems languages with promising levels of adoption * Stablization and commodification of the existing platforms, weakening lock-in effects * Emphasis on virtualization in contemporary server architectures * "The browser is the OS" reaching its natural conclusion in the form of WASM, driving web devs towards systems languages All of that produces an environment where development could become much more portable in a relatively short timeframe. It's the high friction of the full-service, C-based, multitasking development ecosystem that keeps systems development centralized within a few megaprojects like Linux. But what is actually needed for development is leaner than that, and the project of making these lower layers modernized, portable, and available to virtualization will have the inevitable effect of uprooting the tooling from its existing hardware dependencies, even when no one of the resulting environments does "as much" as a Linux box. The classic disruption story.
devit 8y ago
I think it ultimately depends on how much ARM/RISC-V's price/performance ends up being better than x86. If it's not much better then people will not switch due to these small annoyances, and there doesn't seem to be any fundamental reason for it being much better (Intel and AMD are perfectly capable of producing top-performing x86 CPUs, and the architecture should not matter much).
shitgoose 8y ago
Linus said "the cross-development model is so relatively painful". What you described is a relatively painful effort of chasing the word alignments. Wouldn't it be better not to worry about alignments at all and spend your time on something more productive? I think this is his point - if non-productive effort can be avoided, people will avoid it.
mark-r 8y ago
Don't you see that his answer has nothing to do with a hacker mindset? It's an assertion that making your development and production environments as close as possible will save you from unexpected grief, coupled with an observation that this has driven server architectures historically. Especially with subtle problems like performance issues. I find it a very sensible conclusion. Of course it didn't hurt that x86 quickly became the price/performance leader for servers, but he makes a good case that this will continue for at least the near future.
- kstrauser 8y ago
  The NetBSD people vehemently disagree. By ensuring your software works on various architectures, you expose subtle bugs in the ones you actually care about. Lots of 32-bit x86 code was improved during the migration to 64-bit, not because the move created new bugs, as because existing ones (i.e. code that relied on undefined behavior) couldn't get away with it anymore.
  - geff82 8y ago
    Even on NetBSD, my old love, you can not take you program on the x86 machine, pack it and then run it on arm. You will have to crosscompile and hope it works.
  - mehrdadn 8y ago
    I wouldn't call their bugs. If the binaries worked correctly on x86 due to compiler specific guarantees then the code wasn't buggy. It just wasn't written for a generic C or C++ compiler.
    - pertymcpert 8y ago
      Undefined behavior is not a compiler specific guarantee. UB can change based on almost random factors, especially between newer releases of the same compiler. They are bugs, they were just masked.
      - 8y ago
Androider 8y ago
Node and Ruby applications do fail on ARM though, when it comes to native libraries and extensions. And now your whole distro is different than your development machine, which adds complexity. Do I really want to be debugging why node-gyp fails to compile scrypt on the ARM distro on the new Amazon A1 ARM instance (which it did in my case)? And if I solve that, what about the other 2451 dependencies? Let's pessimistically say there's a 1% failure rate, I'll be stuck doing that forever! Nah, I'll just go back to my comfy x86 instance, life's short and there's much code to write :) I think I'll side with Linus on this one. I saw first-hand how non-existent the x86 Android market was, despite Intel pouring megabucks into the project. If the developers don't run the same platform, it's not going to happen, no matter how great the cross platform story is in theory. Even if it's as simple as checking a box during upload that "yes, this game can run on X86", a huge chunk of the developers will simply never do that.
- Androider 8y ago
  On a related note, if Apple does switch to ARM chips for their laptops, that will make mainstream ARM server-side development more viable than any cross-platform story ever can. Or kill the Mac desktop. One or the other :)
  - rbanffy 8y ago
    If that happens, a lot of low level programming will be done ARM-first. A lot of Swift and Objective-C code will be built, tested and run primarily on ARM.
  - vbezhenar 8y ago
    I don’t think that it would kill desktop, but I’m sure that unless ARM will be much faster, developers will use x86 macs for a long time.
    - nicoburns 8y ago
      Apple's latest iPad processors are competitive with low-end laptop x86 processors. And they have a much stricter power budget than a laptop. If Apple wants to go this route, then they probably have the capability to build the chips to support it.
jdsully 8y ago
Redis is a popular program, it makes sense to spend the time to port it. But what your missing is the long tail, thousands of little programs scattered around - or edge cases in bigger ones. Working on a non-x86 platform makes you a second class citizen, you will experience issues that others have already ironed out on x86. Software has a long tail of niche code not actively maintained but still heavily used. It doesn't make sense to switch to ARM there.
musicale 8y ago
"I can pretty much guarantee that as long as everybody does cross-development, the platform won't be all that stable. Or successful." iOS seems like a huge counterexample (as you note.)
- Androider 8y ago
  When developing for iOS you typically interact with the iOS simulator on your desktop, which natively compiles your app against x86 versions of the mobile frameworks. True native iOS development is pretty rare, and more painful. Overall, iOS development is a delightful experience because there's a singular hardware target and Apple pretty much nails the execution. For Android development on the other, you don't have a good simulator, and the out-of-the-box dev experience relies on an x86 emulator of the ARM environment. In practice this means that in your day-to-day Android development, you're running the compile-run-test cycle by looking at your actual ARM device all the time, because the emulator is dogshit. I wouldn't really call it cross-platform development in any traditional sense, it's more like remote development, and a bad experience.
  - pcwalton 8y ago
    > When developing for iOS you typically interact with the iOS simulator on your desktop, which natively compiles your app against x86 versions of the mobile frameworks. And the fact that the "develop on x86, test on ARM" workflow works so smoothly on iOS is strong evidence that Linus is wrong.
    - MBCook 8y ago
      Is it? Or is it evidence that Apple has worked REALLY DAMN HARD to make it work decently? I’ve certainly heard of bugs that the simulator doesn’t reproduce because it’s not ARM.
      - pcwalton 8y ago
        > I’ve certainly heard of bugs that the simulator doesn’t reproduce because it’s not ARM. And that isn't enough to get people to demand an ARM emulator. In fact, Android developers hate the ARM emulator and prefer the x86 simulator—more evidence against Linus' assertion.
pennaMan 8y ago
>one year ago I started the effort to support ARM as a primary architecture for Redis I'm looking forward to embedding Redis my Android app :)
supermatt 8y ago
i think that they don't care until they do. I think it's awesome that you ported redis to ARM, but that is your software. If my node/rails app has modules with native libs unsupported on ARM, do I fix all those modules that "almost" work, replace them with other ones that already work, or just deploy to an environment I already know works? And once I've hit that issue once, will I even try it again?
jdietrich 8y ago
>It's extremely hard to agree with Linus on that. It's very easy to disagree with him, because the server market doesn't work the way he think it does. Google, Amazon, Microsoft and Facebook collectively purchased 31% of all the servers sold in 2018. The market for server hardware is dominated by a handful of hyperscale operators. The "long tail" is made up of a few dozen companies like SAP, Oracle, Alibaba and Tencent, with the rest of the market practically representing a rounding error. These customers are extraordinarily sensitive to performance-per-watt; for their core services, they can readily afford to employ thousands of engineers at SV wages to eke out small efficiency improvements in their core services. They aren't buying Xeon chips on the open market - they're telling Intel what kind of chips they need, Intel are building those chips and everyone else gets whatever is left over. If someone else has a better architecture and can deliver meaningful efficiency savings, they'll throw a couple of billion dollars in their direction without blinking. This is not theoretical - Google are on the third generation of the TPU, Amazon are making their own ARM-based Graviton chips and Facebook now have a silicon design team led by the guy who designed the Pixel Visual Core. It's looking increasingly certain that Apple are moving to ARM on the desktop, which further undermines the "develop at home" argument. ARM won't win the server space, because nobody will win the server space. With Moore's Law grinding to a halt, the future of computing clearly involves an increasing number of specialised architectures and instruction sets. When you're spending billions of dollars a year on server hardware and using as much electricity as a small country, using a range of application-specific processors becomes a no-brainer.
- sasavilic 8y ago
  All of this will be useless, if there is no customers that are interested in new platform. There is a big difference between making existing used platform more efficient and offering new efficient platform for which only few customers are interested.
  - jdietrich 8y ago
    "Run your code on our boxes" is a very, very small subset of cloud services. Does anyone other than Facebook care what instruction set they're using to ingest images? Does anyone other than Amazon care what instruction set they're using to serve S3 requests? Does anyone other than Google care what architecture they're using to crawl the web or serve ads or do something creepy with neural nets?
jecxjo 8y ago
I think more people need to have a kernel hacker mindset. We are getting to the point where everyone is working so high level that they just assume you can swap out the Distro/Kernel/Arch and it will not only "just work" but will work exactly the same was as your home system. While your simple applications (not attempting to optimize or really push the limits of your system) you might not run into issues initially but when you try push things you'll inevitably run into the problem. The fact that we're talking about ARM makes this even more important. You're having to compete against x86, which requires increased core counts, and a lot more optimization and potentially even redesigns of your software to make your higher level environment to be perceived equal to x86. Businesses will need this, your boss will ask if ARM is as fast as x86 and they won't care to quibble about technological differences if you can't just get the same output speeds as their old, trusted hardware. There is only so much your language can do to cover your butt. At some point you'll have to be aware of your environment to compete.
kakwa_ 8y ago
It's likely that ARM will find a place in a cloud. But, at, least at first, it will be in some specific parts of a cloud offering. I don't see ARM displacing X86 on VMs offering like EC2 any time soon, an ARM offering will exists (and it already exists in fact), but it will remain a small portion. However, some parts of a cloud offering are completely abstracted from the hardware: DNS, object stores, Load Balancers, queues, CDNs... for these, from the point of view of a developer, CPU architecture doesn't matter at all and if the cloud provider find it more interesting to use ARM (maybe with some custom extensions), it will probably switch to them. From there, it can gradually go to services where architecture kind of matters, but not necessarily, like serverless, or Postgres/MySQL as a service. And while it grows, ARM CPUs will improve for other use cases, and maybe overtake X86 VMs. The other possibility is a massive cost reduction like 3 to 4 times cheaper for equal performances, but it's not really the case right now. Also, given the all the wasted money I've seen on AWS ("app is leaking memory? just use a 64GB instance"), I'm not sure it's a good enough incentive. However we are specialist at being penny wise and pound foolish.
jfrn 8y ago
Redis is not a very large scale piece of software or software system.
coldtea 8y ago
It might be easy to do that in a well written single C/C++ codebase like Redis. It's not simple nor something devs will ever want or care to do in a big web app with several binary dependencies. Just consider that a single Node app's binary deps could trivially include the entirety of Chrome itself, not just in the form of Node's v8 engine, but e.g. as the PDF rendering "headless chrome" wrapper Puppeteer. And that's just the tip of the iceberg, add DBs, extensions, Python backend scripts, etc etc, and few will bother.
justicezyx 8y ago
Actually, he is right.
Solar19 8y ago
Good work on Redis. Ruby, however, strikes me as pretty fragile. They can't get it running on Windows, officially, and the third-party Windows installer is hit or miss. Ruby also seems to depend on a specific compiler, gcc. I wouldn't be surprised if it has trouble on ARM.
computerex 8y ago
> most today's developers don't care about environment reproducibility at architecture level. Source for this? Seems like pure speculation.
think_free 8y ago
"I often find black-and-white people a bit stupid, truth be told" - Linus Torvalds, 2005 I think maybe Linus T. is getting old, out of touch, and closed-minded, and I think we should be open to change and care less about every random thought he blows off. Quote source: https://www.linux.com/news/linus-compares-linux-and-bsds https://www.linux.com/news/linus-compares-linux-and-bsds
- Chyzwar 8y ago
  he at least he does not create an accounts to shit on someone.
ronreiter 8y ago
Yes!

mcv 8y ago

I'm a web developer, and for years now I've been developing on either Mac or Windows, but deploying to Linux servers. Or possibly totally different kinds of servers. I don't care much about the architecture of the server as long as it runs my code. Give me Apache, JVM, node, and the necessary build and deploy tools, and there's nothing I can't run on it.

fmajid 8y ago

Linux' point is that there is hardly any developer-class ARM machines available, just RaspberryPi-class SBCs that use mobile SOCs with the performance of a 2012 vintage smartphone, nothing with the grunt of a Qualcomm Centriq or Cavium ThunderX.

013a 8y ago

It seems likely that, simply, times have changed. There was a time when being on the same platform as the deployment environment was super important, but nowadays the tooling has gotten so much better that it matters a lot less. The proportion of people still writing C code on a day to day basis has dropped... well to pretty much a rounding error. The bigger issue is really that ARM servers aren't that much cheaper than x86 servers today, and its very likely a lot of that difference in cost is just Intel's synthetic market advantage that would disappear if ARM actually started becoming a threat (which has already started happening due to AMD becoming a threat). Phoronix did a synthetic benchmark of AWS's A1 instances, versus the C5 Intel and C5A AMD instances [1]; they're nothing special at all, even with price taken into account. Maybe that'll change in the future, but now that AMD is in a competitive state, that's pushing Intel into high-gear and its hard to say that ARM will have any effect on the server market in the short-term. [1] https://www.phoronix.com/scan.php?page=article&item=ec2-graviton-performance&num=5 https://www.phoronix.com/scan.php?page=article&item=ec2-grav...

antirez 8y ago
> but nowadays the tooling has gotten so much better that it matters a lot less I think that that's the point. Portability to platforms with a strong tooling and usage base even in a different sector is ok and safe. The problem is when you try to do something like x86 -> Itanium or alike, that could take some time to stabilize.
WorldMaker 8y ago
> There was a time when being on the same platform as the deployment environment was super important Which is also interesting because there was a time before that where being on the same platform as the deployment environment was sometimes considered nigh impossible, such as the early days of the "microcomputer" revolution where a lot of software was written on big iron mainframes to run on much more constrained devices (C64, Apple II, etc). It's interesting to compare the IDEs and architectures of that era and how much cross-compilation has always happened. There doesn't seem to be a lot of computing history where the machine used to build the software was the same machine intended to run the software, it's the modern PC era that seems the unique inflection point where so much of our software are built and run on the same architectures. (A lot of the modern tools such as VMs like the JVM and CLR are because of the dreams and imaginations of those developers that directly experienced those earlier eras.) It's interesting how that tide shifts from time to time, and we so easily forget what that was like, forget to notice the high water marks of previous generations. (Even as we take advantage of it in other ways, we cross-compile to mobile and IoT devices today we'd have no way to run IDEs on, and would rather not try to run compilers directly on them.)
- rbanffy 8y ago
  I know some software was written on minis to run on 8-bit computers, but I have a hard time imagining that as the norm. My Apple II dev rig was two computers, one running development tools and one to test and they were two because running my software wasn't possible on the development machine without rebooting and loading all the tools took 30 seconds - a painful eternity in Apple II terms.
  - 8y ago

otterley 8y ago

I agree with you. For many users of the cloud -- especially serverless application developers -- the actual hardware on which their software runs is a black box. In fact, if the serverless application consists solely of scripts or precompiled bytecode and doesn't contain architecture-specific binary code, it could likely run on arbitrary hardware with any supported ISA and users wouldn't know the difference.

mattnewport 8y ago

A low level piece of code written in C seems likely to have less portability issues than something sitting on top of many layers of abstraction. The thorny problems that show up deploying on an environment that's not identical to the development environment are often the result of unexpected interactions in the stack of dependencies. This is why containerization is a thing. Platform portability issues have got easier with better adherence to standards and where you have largely the same code running across different ISAs (and no endianness issues between x86 and Arm) but the popularity of things like Docker suggest many devs do care about reproducible production environments.

boris 8y ago

What hardware do you use for development and for testing? Can you elaborate on your setup, ARM-wise?

iveqy 8y ago

I once had a bug developing an IP phone. Connecting to one server worked but once on site with the customer connection to their server didn't worked, even though the servers where identical. It turned out that the power supply on their server was malfunctioning sometimes delivering too little power. Specially when taking the code paths my IP phone triggered. The server software was built in php. It's not often you start looking for a bug in PHP but end up switching a capacitor in the PSU. My point is that even if your write in php, php is running C libraries, that is running ASM that is running hardware and every part of this chain is important. There's no such thing as "works everywhere", it's just "have a very high chance of working everywhere". (off-topic) thanks for the sds library. I'm a heavy user of it.

antirez 8y ago
Yes but the platform is just one of the unknowns at the lower level. If the tooling is fine, the C compiler is very unlikely to emit things that will run in a different way, it is much simpler to see software breaking because of higher level parts that don't have anything to do with the platform like: libc version, kernel upgrade, ... About SDS: glad it is useful for you!
noah256 8y ago
I’d be really interested to hear more of this story! How you isolated the problem down to the level of the PSU is going deeper into the machine than I’ve personally been, so this story could be a great teacher.
- iveqy 8y ago
  It sounds way cooler than it was. For some reason I openened up the server box and during a reboot I saw the LED on the motherboard flickering in a way I didn't expect. So we tried to change PSU and then everything worked.

sipos 8y ago

On the other hand, it's easier to not have to think about it, even when it doesn't matter 99% of the time, and run the exact same container as you tested locally on the server. I feel like this shouldn't matter really, but people are amazingly lazy/developer time valued highly.

asadkn 8y ago

Depends on general confidence level and how mission critical your deployed software is. For instance, I will always prefer to run PHP, Ruby or Python on Linux servers. But on the client side, I have faith in Electron Apps to run cross-platform without issues. I wouldn't be comfortable with an underlying architecture change to ARM for at least years to come and the usage decision would be based on general consensus on reliability that follows.

antirez 8y ago
I would more comfortably run my code developed on x86/Linux on ARM/Linux than on x86/FreeBSD for instance... Platform is just one unknown and not the worst if the tooling is good IMHO. Consider that complex software under Linux/ARM now has a ten years history at least, with numbers (mobile) that are not approached by any other thing on earth.

readittwice 8y ago

I guess the situation is similar to containers and static linking that got popular for server deployments quite recently. They allow to develop, test and deploy your application with its dependencies in the exact same version - the exact same binary. Although usually this works just fine even with minor differences in the dependencies, a lot of developers deem it worthwhile to preclude such issues. If you now use ARM in production, you can't use the same binary file anymore. There might be different implementations depending on the architecture in some library you use. Also even with higher-level languages like Java it is possible to observe ISA differences: e.g. memory ordering.

AnimalMuppet 8y ago

I'm with Linus on this one. I had the pleasure (?) of working on a C/C++ codebase that compiled on Windows and ten different flavors of Unix. It was all "portable", but all over the place there was stuff like #if defined AIX || defined OSF1 short var; #else int var; #endif And to get it right, you had to compile it on all the platforms and fix all the errors (and preferably all the warnings). Yeah, cross platform is never as simple as same platform.

musicale 8y ago
Single platform cross-development (e.g. iOS) is a lot easier.
DerekL 8y ago
Yes, cross-platform development is more work, but that example you gave is the wrong way to do it. It's better to abstract the differences using typedefs, functions, macros, etc., and keep the platform switches in a few isolated places in the code.
wahern 8y ago
How many years ago was that? POSIX compliance has come a long way and most of the proprietary vendors (the ones with all the corner cases) are gone. These days not only do platforms like AIX and Solaris have much fewer corner cases, they're even adopting Linux and GNU extensions wholesale. Anyhow, most people can ignore these altogether. Portability between Linux and the BSDs is much easier. macOS is the biggest outlier in terms of corner cases yet in many ways the best supported thanks to the popularity of Homebrew. C++ is a different matter, but C++ portability is a headache even if you stay on Linux. Likewise, trying to maintain OS-level portability of monolithic codebases between Windows and Unix is a fools errand, which is why Windows Subsystem for Linux (WSL) is likely to only get better.
- AnimalMuppet 8y ago
  How long ago? 15 years - and it was at least a decade-old code base. But I had to take some new code that was something like Windows-and-Sun-only, and port it to run on all of the other architectures.

sangnoir 8y ago

I don't think you are not really disagreeing with Linus - he's not saying ARM is not viable - he is saying it will not win. With your current setup (cross-compiling), are your ARM executables more performant than x86? Or do they have any other advantage at all over x86? Without an advantage, ARM can't possibly win. Having a cheap, viable ARM-native development platform drastically increases the chances of ARM-only killer apps to exist, this would be an advantage over the currently dominant x86 (just as there were Windows-only and Linux-only killer apps that cemented their ascent). However, if everyone is cross-compiling due to the cost, it means ARM will always be a secondary platform (at most)- it can't win by being the Windows Phone of platforms. [edited for clarity]

bryanlarsen 8y ago
He's not saying it won't win, either. He's just saying that for it to win, it needs a viable dev platform. Which, if you reverse cause and effect is blatantly obvious. If ARM comes anywhere close to viable enough to be "winning", there will be a good market for dev platforms, and somebody will step in and fill the need. Heck, some are even arguing here that the Pine64 already meets that need.
- jeffdavis 8y ago
  It's not simple cause and effect. Servers cause compatible dev boxes and dev boxes cause compatible servers.
- mntmoss 8y ago
  I'm definitely on the side of ARM(and RISC-V and other new architectures for that matter) getting "wins", because the modern environment is displaying the signs of a low-layer shakeup: * New systems languages with promising levels of adoption * Stablization and commodification of the existing platforms, weakening lock-in effects * Emphasis on virtualization in contemporary server architectures * "The browser is the OS" reaching its natural conclusion in the form of WASM, driving web devs towards systems languages All of that produces an environment where development could become much more portable in a relatively short timeframe. It's the high friction of the full-service, C-based, multitasking development ecosystem that keeps systems development centralized within a few megaprojects like Linux. But what is actually needed for development is leaner than that, and the project of making these lower layers modernized, portable, and available to virtualization will have the inevitable effect of uprooting the tooling from its existing hardware dependencies, even when no one of the resulting environments does "as much" as a Linux box. The classic disruption story.

devit 8y ago

I think it ultimately depends on how much ARM/RISC-V's price/performance ends up being better than x86. If it's not much better then people will not switch due to these small annoyances, and there doesn't seem to be any fundamental reason for it being much better (Intel and AMD are perfectly capable of producing top-performing x86 CPUs, and the architecture should not matter much).

shitgoose 8y ago

Linus said "the cross-development model is so relatively painful". What you described is a relatively painful effort of chasing the word alignments. Wouldn't it be better not to worry about alignments at all and spend your time on something more productive? I think this is his point - if non-productive effort can be avoided, people will avoid it.

mark-r 8y ago

Don't you see that his answer has nothing to do with a hacker mindset? It's an assertion that making your development and production environments as close as possible will save you from unexpected grief, coupled with an observation that this has driven server architectures historically. Especially with subtle problems like performance issues. I find it a very sensible conclusion. Of course it didn't hurt that x86 quickly became the price/performance leader for servers, but he makes a good case that this will continue for at least the near future.

kstrauser 8y ago
The NetBSD people vehemently disagree. By ensuring your software works on various architectures, you expose subtle bugs in the ones you actually care about. Lots of 32-bit x86 code was improved during the migration to 64-bit, not because the move created new bugs, as because existing ones (i.e. code that relied on undefined behavior) couldn't get away with it anymore.
- geff82 8y ago
  Even on NetBSD, my old love, you can not take you program on the x86 machine, pack it and then run it on arm. You will have to crosscompile and hope it works.
- mehrdadn 8y ago
  I wouldn't call their bugs. If the binaries worked correctly on x86 due to compiler specific guarantees then the code wasn't buggy. It just wasn't written for a generic C or C++ compiler.
  - pertymcpert 8y ago
    Undefined behavior is not a compiler specific guarantee. UB can change based on almost random factors, especially between newer releases of the same compiler. They are bugs, they were just masked.
    - 8y ago

Androider 8y ago

Node and Ruby applications do fail on ARM though, when it comes to native libraries and extensions. And now your whole distro is different than your development machine, which adds complexity. Do I really want to be debugging why node-gyp fails to compile scrypt on the ARM distro on the new Amazon A1 ARM instance (which it did in my case)? And if I solve that, what about the other 2451 dependencies? Let's pessimistically say there's a 1% failure rate, I'll be stuck doing that forever! Nah, I'll just go back to my comfy x86 instance, life's short and there's much code to write :) I think I'll side with Linus on this one. I saw first-hand how non-existent the x86 Android market was, despite Intel pouring megabucks into the project. If the developers don't run the same platform, it's not going to happen, no matter how great the cross platform story is in theory. Even if it's as simple as checking a box during upload that "yes, this game can run on X86", a huge chunk of the developers will simply never do that.

Androider 8y ago
On a related note, if Apple does switch to ARM chips for their laptops, that will make mainstream ARM server-side development more viable than any cross-platform story ever can. Or kill the Mac desktop. One or the other :)
- rbanffy 8y ago
  If that happens, a lot of low level programming will be done ARM-first. A lot of Swift and Objective-C code will be built, tested and run primarily on ARM.
- vbezhenar 8y ago
  I don’t think that it would kill desktop, but I’m sure that unless ARM will be much faster, developers will use x86 macs for a long time.
  - nicoburns 8y ago
    Apple's latest iPad processors are competitive with low-end laptop x86 processors. And they have a much stricter power budget than a laptop. If Apple wants to go this route, then they probably have the capability to build the chips to support it.

jdsully 8y ago

Redis is a popular program, it makes sense to spend the time to port it. But what your missing is the long tail, thousands of little programs scattered around - or edge cases in bigger ones. Working on a non-x86 platform makes you a second class citizen, you will experience issues that others have already ironed out on x86. Software has a long tail of niche code not actively maintained but still heavily used. It doesn't make sense to switch to ARM there.

musicale 8y ago

"I can pretty much guarantee that as long as everybody does cross-development, the platform won't be all that stable. Or successful." iOS seems like a huge counterexample (as you note.)

Androider 8y ago
When developing for iOS you typically interact with the iOS simulator on your desktop, which natively compiles your app against x86 versions of the mobile frameworks. True native iOS development is pretty rare, and more painful. Overall, iOS development is a delightful experience because there's a singular hardware target and Apple pretty much nails the execution. For Android development on the other, you don't have a good simulator, and the out-of-the-box dev experience relies on an x86 emulator of the ARM environment. In practice this means that in your day-to-day Android development, you're running the compile-run-test cycle by looking at your actual ARM device all the time, because the emulator is dogshit. I wouldn't really call it cross-platform development in any traditional sense, it's more like remote development, and a bad experience.
- pcwalton 8y ago
  > When developing for iOS you typically interact with the iOS simulator on your desktop, which natively compiles your app against x86 versions of the mobile frameworks. And the fact that the "develop on x86, test on ARM" workflow works so smoothly on iOS is strong evidence that Linus is wrong.
  - MBCook 8y ago
    Is it? Or is it evidence that Apple has worked REALLY DAMN HARD to make it work decently? I’ve certainly heard of bugs that the simulator doesn’t reproduce because it’s not ARM.
    - pcwalton 8y ago
      > I’ve certainly heard of bugs that the simulator doesn’t reproduce because it’s not ARM. And that isn't enough to get people to demand an ARM emulator. In fact, Android developers hate the ARM emulator and prefer the x86 simulator—more evidence against Linus' assertion.

pennaMan 8y ago

>one year ago I started the effort to support ARM as a primary architecture for Redis I'm looking forward to embedding Redis my Android app :)

supermatt 8y ago

i think that they don't care until they do. I think it's awesome that you ported redis to ARM, but that is your software. If my node/rails app has modules with native libs unsupported on ARM, do I fix all those modules that "almost" work, replace them with other ones that already work, or just deploy to an environment I already know works? And once I've hit that issue once, will I even try it again?

jdietrich 8y ago

>It's extremely hard to agree with Linus on that. It's very easy to disagree with him, because the server market doesn't work the way he think it does. Google, Amazon, Microsoft and Facebook collectively purchased 31% of all the servers sold in 2018. The market for server hardware is dominated by a handful of hyperscale operators. The "long tail" is made up of a few dozen companies like SAP, Oracle, Alibaba and Tencent, with the rest of the market practically representing a rounding error. These customers are extraordinarily sensitive to performance-per-watt; for their core services, they can readily afford to employ thousands of engineers at SV wages to eke out small efficiency improvements in their core services. They aren't buying Xeon chips on the open market - they're telling Intel what kind of chips they need, Intel are building those chips and everyone else gets whatever is left over. If someone else has a better architecture and can deliver meaningful efficiency savings, they'll throw a couple of billion dollars in their direction without blinking. This is not theoretical - Google are on the third generation of the TPU, Amazon are making their own ARM-based Graviton chips and Facebook now have a silicon design team led by the guy who designed the Pixel Visual Core. It's looking increasingly certain that Apple are moving to ARM on the desktop, which further undermines the "develop at home" argument. ARM won't win the server space, because nobody will win the server space. With Moore's Law grinding to a halt, the future of computing clearly involves an increasing number of specialised architectures and instruction sets. When you're spending billions of dollars a year on server hardware and using as much electricity as a small country, using a range of application-specific processors becomes a no-brainer.

sasavilic 8y ago
All of this will be useless, if there is no customers that are interested in new platform. There is a big difference between making existing used platform more efficient and offering new efficient platform for which only few customers are interested.
- jdietrich 8y ago
  "Run your code on our boxes" is a very, very small subset of cloud services. Does anyone other than Facebook care what instruction set they're using to ingest images? Does anyone other than Amazon care what instruction set they're using to serve S3 requests? Does anyone other than Google care what architecture they're using to crawl the web or serve ads or do something creepy with neural nets?

jecxjo 8y ago

I think more people need to have a kernel hacker mindset. We are getting to the point where everyone is working so high level that they just assume you can swap out the Distro/Kernel/Arch and it will not only "just work" but will work exactly the same was as your home system. While your simple applications (not attempting to optimize or really push the limits of your system) you might not run into issues initially but when you try push things you'll inevitably run into the problem. The fact that we're talking about ARM makes this even more important. You're having to compete against x86, which requires increased core counts, and a lot more optimization and potentially even redesigns of your software to make your higher level environment to be perceived equal to x86. Businesses will need this, your boss will ask if ARM is as fast as x86 and they won't care to quibble about technological differences if you can't just get the same output speeds as their old, trusted hardware. There is only so much your language can do to cover your butt. At some point you'll have to be aware of your environment to compete.

kakwa_ 8y ago

It's likely that ARM will find a place in a cloud. But, at, least at first, it will be in some specific parts of a cloud offering. I don't see ARM displacing X86 on VMs offering like EC2 any time soon, an ARM offering will exists (and it already exists in fact), but it will remain a small portion. However, some parts of a cloud offering are completely abstracted from the hardware: DNS, object stores, Load Balancers, queues, CDNs... for these, from the point of view of a developer, CPU architecture doesn't matter at all and if the cloud provider find it more interesting to use ARM (maybe with some custom extensions), it will probably switch to them. From there, it can gradually go to services where architecture kind of matters, but not necessarily, like serverless, or Postgres/MySQL as a service. And while it grows, ARM CPUs will improve for other use cases, and maybe overtake X86 VMs. The other possibility is a massive cost reduction like 3 to 4 times cheaper for equal performances, but it's not really the case right now. Also, given the all the wasted money I've seen on AWS ("app is leaking memory? just use a 64GB instance"), I'm not sure it's a good enough incentive. However we are specialist at being penny wise and pound foolish.

jfrn 8y ago

Redis is not a very large scale piece of software or software system.

coldtea 8y ago

It might be easy to do that in a well written single C/C++ codebase like Redis. It's not simple nor something devs will ever want or care to do in a big web app with several binary dependencies. Just consider that a single Node app's binary deps could trivially include the entirety of Chrome itself, not just in the form of Node's v8 engine, but e.g. as the PDF rendering "headless chrome" wrapper Puppeteer. And that's just the tip of the iceberg, add DBs, extensions, Python backend scripts, etc etc, and few will bother.

justicezyx 8y ago

Actually, he is right.

Solar19 8y ago

Good work on Redis. Ruby, however, strikes me as pretty fragile. They can't get it running on Windows, officially, and the third-party Windows installer is hit or miss. Ruby also seems to depend on a specific compiler, gcc. I wouldn't be surprised if it has trouble on ARM.

computerex 8y ago

> most today's developers don't care about environment reproducibility at architecture level. Source for this? Seems like pure speculation.

think_free 8y ago

"I often find black-and-white people a bit stupid, truth be told" - Linus Torvalds, 2005 I think maybe Linus T. is getting old, out of touch, and closed-minded, and I think we should be open to change and care less about every random thought he blows off. Quote source: https://www.linux.com/news/linus-compares-linux-and-bsds https://www.linux.com/news/linus-compares-linux-and-bsds

Chyzwar 8y ago
he at least he does not create an accounts to shit on someone.

ronreiter 8y ago

Yes!