21 ms·
Will R Work on Apple Silicon?
- Thaxll 6y agoThe question is why in 2020 R still uses a Fortran compiler?
- _Wintermute 6y agoBecause it's R, it doesn't even have 64 bit integers yet.
- rodonn 6y agoIt does with the bit64 package, but agree that I wish it was directly supported.
- anbende 6y agoThis is answered in the article. Chunks of R are written in Fortran 90, which can’t be converted to C easily right now. You might ask why it’s written in Fortran at all. Probably has something to do with its history coming out of the S language at Bell labs in the 70s and 80s.
- mmrezaie 6y agoFortran is domninant in HPC (Maybe not dominant, but there are a lot of software in HPC written in Fortran). R uses some performance oriented libraries which most likely implemented in Fortran.
- uberman 6y agoFortan is berserkly fast.
- bregma 6y agoI'm sure someone will come along with a Rust rewrite any day now.
- mike_ivanov 6y agoI admire your sense of humor.
- gh02t 6y agoTons of software are still using Fortran in the scientific/numerical world. NumPy and SciPy, for example, also make extremely heavy use of Fortran. As would many things that rely on BLAS/LAPACK. It's not just legacy code, either... Fortran is still very active in its own little niche in the numerical world.
- danpalmer 6y agoYep, I think the issue with R is that they're using a customised version of BLAS/LAPACK – Python has been running these things on Raspberry Pis for ages now, I suspect using a more standard implementation.
- em500 6y agoNo, the issues are different, please read the article. R also runs fine on ARM64 linux. But macOS is not linux, as mentioned in the article it has a different ABI and no free Fortran 90 compiler is available yet. The other issue is that R distinguishes between NA values and NaN values (NumPy doesn't), which are propagated differently on ARM64.
- gh02t 6y agoBased on the article I don't think those are the problem. I think the new Apple silicon is distinct enough that it needs a bit of porting effort to get a Fortran compiler running, along with the issue of quirks in handling NaN payloads and some other (seemingly rather minor) differences.
- ryukafalz 6y agoIf it works well, why change it?
- geofft 6y agoFortran is the standard implementation language for scientific computing in 2020. Try compiling NumPy and SciPy from source sometime.
- baron_harkonnen 6y agoFortran is still a foundation of many important libraries for numeric computing. A fairly large number of implementations of BLAS are written in Fortran (including the reference implementation), LAPACK is written in Fortran. One of the first domains solved by programming was efficient implementations of common linear algebra computations. Fortran was the original language of choice for many of those projects. When you care about absolutely optimal performance for these computations you're not going to mess with fined tuned code that has been slowly tweaked and improved for over 50 years.
- pdpi 6y agoIf I recall correctly, C and C++ allow some types of pointer aliasing that fortran forbids. If you're reading from one buffer and writing to another, those buffers can overlap in C or C++, but can't in fortran, so a fortran compiler is allowed more leeway with the way instructions are ordered (and maybe elided? Not sure). In compute-intensive workloads, every little bit helps.
- rlkf 6y agoCorrect. Back in the days, it used to be that you could run Fortran 77 code through f2c and then compile with gcc using the assume-no-aliasing option, and you would get roughly the same performance as if you compiled with f77.
- johncolanduoni 6y agoIt's also worth noting that although modern C/C++ has the `restrict` keyword, compilers for those languages are generally worse at actually using that information. For example, there's been a long running series of LLVM bugs that has (several times) required Rust to stop emitting that metadata because it would actually miscompile fairly simple vector operations that used `restrict`[1]. I'm hopeful that Flang (the Fortran LLVM compiler) will shake most of those out, since there's a large body of Fortran code that relies on good aliasing optimizations. [1]: https://github.com/rust-lang/rust/issues/54878 https://github.com/rust-lang/rust/issues/54878
- saagarjha 6y agoJust C. The various “restrict equivalents” in C++ are non-standard.
- ForHackernews 6y agoFortran is crazy fast and the standard language in many scientific domains where performance matters.
- gspr 6y ago> The question is why in 2020 R still uses a Fortran compiler? You will find that almost any scientific codebase of any size includes or relies on at least some Fortran code.
- deleted 6y ago[deleted]
- BooneJS 6y agoFORTRAN can be compiled on ARM Macs, but only commercially for now. https://www.nag.com/news/first-fortran-compiler-apple-silicon-macs https://www.nag.com/news/first-fortran-compiler-apple-silico...
- sgt 6y agoCan the R project use that compiler to build binaries and then upload the release to the public?
- pdpi 6y agoGCC/gfortran should soon follow, no?
- my123 6y agoThe current effort is done with this out of tree repository: https://github.com/iains/gcc-darwin-arm64 https://github.com/iains/gcc-darwin-arm64 Which supports macOS on Apple Silicon quite well, hopefully will be merged soon to mainline GCC.
- mistrial9 6y agoIt is possible that Apple Inc will forcefully eliminate GCC from their platforms, replaced with Clang/LLVM and other non-GPL tools only.
- baggy_trough 6y agoNo, it really isn't.
- pdpi 6y agoThey're certainly well on their way to remove all GPL code from macOS itself if they haven't yet, but there's not much they can do to prevent you from installing GPL software yourself (nor much of a motivation to do so for that matter).
- fedorareis 6y agoMacOS already doesn’t include GCC. The GCC command on MacOS is an alias for Clang unless you install gcc separately (unless there is a hidden GCC install I’m unaware of).
- CoolGuySteve 6y agoR doesn't even work that well on Intel, at least in Ubuntu. Recompiling the package with AVX support often leads to a 30% performance increase on modern CPUs. IMO the R base package should dynlink different shared libraries for different processors since vector extensions are mostly tailored to the kind of floating point numerical work that R does.
- physicsguy 6y agoThat's deliberate though, when you distribute software you choose the lowest common denominator in general, and that's SSE2 for 64 bit machines.
- johncolanduoni 6y agoMany linear algebra libraries will handle this at runtime for you (OpenBLAS & MKL do this for example). You generally only need to use specialized builds of these if you don't want to have to ship extra code paths you won't use.
- jsight 6y agoSee, this is why Gentoo is the right way to manage an OS. /s
- gnufx 6y agoThe SIMD makes little difference to the bulk of the system, but you want dynamic dispatch where it does, at least on a typical HPC system, which isn't heterogeneous five years down the line, and in things like libc generally.
- bobbylarrybobby 6y agoWhen you install Python's numpy, I'm pretty sure it chooses a pre-built package based on your hardware, and if it doesn't have one I think it's pretty easy to get it to build the best one from scratch.
- 6y ago
- willglynn 6y ago> The ARM architecture floating point units (VFP, NEON) support RunFast mode, which includes flush-to-zero and default NaN. The latter means that payload of NaN operands is not propagated, all result NaNs have the default payload, so in R, even NA * 1 is NaN. Luckily, RunFast mode can be disabled, and when it is, the NaN payload propagation is friendlier to R NAs than with Intel SSE (NaN + NA is NA). We have therefore updated R to disable RunFast mode on ARM on startup, which resolved all the issues observed. Hmm. ELF object files for Arm can represent this with build attributes [1]: Tag_ABI_FP_denormal, (=20), uleb128 0 The user built this code knowing that denormal numbers might be flushed to (+) zero 1 The user permitted this code to depend on IEEE 754 denormal numbers 2 The user permitted this code to depend on the sign of a flushed-to-zero number being preserved in the sign of 0 Tag_ABI_FP_number_model, (=23), uleb128 0 The user intended that this code should not use floating point numbers 1 The user permitted this code to use IEEE 754 format normal numbers only 2 The user permitted numbers, infinities, and one quiet NaN (see [RTABI32_]) 3 The user permitted this code to use all the IEEE 754-defined FP encodings Seems like their code should be tagged Tag_ABI_FP_denormal = 1, Tag_ABI_FP_number_model = 3 if it were an ELF .o, .so, or executable, in which case <waves hands> some other part of the toolchain or system would automatically configure the floating point unit to provide the required behavior. Does Mach-O have a similar mechanism? [1] https://github.com/ARM-software/abi-aa/blob/master/addenda32/addenda32.rst#procedure-call-related-attributes https://github.com/ARM-software/abi-aa/blob/master/addenda32...
- johncolanduoni 6y agoI wonder what happens if you `dlopen` a shared object that wants stricter behavior than the current executable and loaded shared objects. Does it somehow coordinate changing the state for all existing threads?
- loeg 6y agoFrom GP's link: > Procedure call-related attributes describe compatibility with the ABI. They summarize the features and facilities that must be agreed in an interface contract between functions defined in this relocatable file and elsewhere. Seems like it might be reasonable to reject mismatched combinations.
- wodenokoto 6y agoBrushing up on the difference between NA and NaN, makes the article considerably easier to read NA - Not available NaN - Not a number See this short and concise article on the differences: https://jameshoward.us/2016/07/18/nan-versus-na-r/ https://jameshoward.us/2016/07/18/nan-versus-na-r/
- melling 6y agoHow popular is R in general? I started learning it because I want to make an attempt to do some projects on Kaggle. Most people use Pandas, Seaborn, etc, which I will also use. However, to me R appears like a little better Swiss Army Knife to do initial analysis. ggplot2, tidyverse, ... Any help leveling up would be appreciated.
- kickout 6y agoProbably the most popular non-software engineer language for working with data. Millions and millions of users that have no idea what this blog post is technically about (but is interesting nonetheless)
- NegativeLatency 6y agoSecond after Excel?
- deleted 6y ago[deleted]
- kickout 6y agoFair, although I would say Excel and R are serving two separate purposes. But yes, Excel is of course #1
- wtetzner 6y ago> Fair, although I would say Excel and R are serving two separate purposes. It's true that they target different use cases overall (obviously with some overlap), but Excel tends to be used for lots of things that would be better handled with a different tool, because it's what people know.
- ISL 6y agoMatlab has a substantial userbase, too.
- analog31 6y agoAsk HN: Does the FORTRAN issue also affect Numpy/Scipy for Python?
- disgruntledphd2 6y agoYes, this will impact Numpy/Scipy as much as R, as they depend on many of the same linear algebra libraries. Edit: actually, looks like I'm wrong emghost appears to know more about this than me.
- 3JPLW 6y agoSome of the NaN-handling issues are unique, but the compiler availability itself is still a problem. https://twitter.com/StefanKarpinski/status/1292983717212893191 https://twitter.com/StefanKarpinski/status/12929837172128931... Since then, GCC has stepped up with this out-of-tree build, but it's still not 100% to my knowledge. https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96168 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96168
- motorbreath 6y agoThere's always R Cloud, accessible from a browser.
- gok 6y agoIs there a reason R can't use the BLAS/LAPACK implementation that comes with macOS in the Accelerate framework?
- kxyvr 6y agoA similar issue has come up for codes that I write. Among other things, I write low level mathematical optimization codes that need fast linear algebra to run effectively. While there's a lot of emphasis on BLAS/LAPACK, those libraries work on dense linear algebra. In the sparse world, there are fewer good options. For things like sparse QR and Choleski, the two fastest codes that I know about are out of SuiteSparse and Intel MKL. I've not tried it, but the SuiteSparse routines will probably work fine on ARM chips, but they're dual licensed GPL/commercial and the commercial license is incredibly expensive. MKL has faster routines and is completely free, but it won't work on ARM. Note, it works fantastically well on AMD chips. Anyway, it's not that I can't make my codes work on the new Apple chips, but I'd have to explain to my commercial clients that there's another $50-100k upcharge due to the architecture change and licensing costs due to GPL restrictions. That's a lot to stomach.
- bachmeier 6y ago> I'd have to explain to my commercial clients that there's another $50-100k upcharge due to the architecture change and licensing costs due to GPL restrictions. Your complaint is kind of strange. You're blaming "GPL restrictions" but the cost is for a commercial license.
- coldtea 6y agoWell, if the FOSS license used was e.g. MIT he wouldn't have to buy a commercial license, that's the parent's point. With GPL, he does, because else his clients have to make their own code/project conformant...
- bachmeier 6y agoOh, so it's terrible to pay for software? How awful! Especially ironic because I'm sure the parent isn't working for free.
- kxyvr 6y agoWe all pay for software, but it's the amount that really shapes decisions. Most organizations have a dollar limit where we can just charge a purchase card and when they have to seek approval. In this particular case, the software costs are higher than what can likely go onto a p-card, so now it becomes a real pain to acquire. In fact, the software is so expensive that it's cost would like eclipse the cost of the computer itself. So basically, we're looking at a decision where the client can use a more performant library and save $100k as long as they stay off of Apple silicon. That's really the point I'm trying to make and not to criticize anyone for using a GPL license. Moving to these new chips, in many cases, will be a much larger cost to an organization than just the cost of the computer.
- stabbles 6y agoDoes anybody know what the status is of Accelerate [1]? Is it implemented for Apple Silicon? Is it optimized for it? To me it seems very few people use this framework. [1] https://developer.apple.com/documentation/accelerate https://developer.apple.com/documentation/accelerate
- ejdyksen 6y agoFrom WWDC 2020, it has been "highly tuned" for Apple Silicon: https://developer.apple.com/videos/play/wwdc2020/10686/?time=270 https://developer.apple.com/videos/play/wwdc2020/10686/?time... Makes sense, since Accelerate has been available on iOS for many years.
- FullyFunctional 6y agoFWIW, RISC-V explicitly doesn't support NaN payload propagation so R will have a problem there as well.
- stevefan1999 6y agoSimilarly, Matlab is also not initially available for Apple Silicon natively now, and they are preparing an update to let Matlab run in Rosetta 2 instead, until the development cycle of native version completes.
- kevin_b_er 6y agoWill Apple even allow compilers that aren't Apple Compilers? They're not allowed on any other Apple Silicon.
- andromeduck 6y agoAs long as they support LLVM it shouldn't it be mostly painless?
- why_only_15 6y agoAre you thinking of JITs? And yes those are allowed.
- deleted 6y ago[deleted]
- sbassi 6y agoDoes R work on Amazon ARM chip (graviton)?
- gnufx 6y agoYes, if it's available for the OS you run, like EL or Debian. It's aarch64.
- superbatfish 6y agoIt sounds like R's design decision to use a non-standard NaN value to represent NA is an obscenely bad one. Wasn't it obvious that this would become a problem someday?
- oddthink 6y agoIt's not a "non-standard NaN". It's just a particular one, out of many possible quiet NaN values. If the Apple silicon isn't propagating the payload of the input NaN value to output, that's a violation of IEEE 754. (IIUC, that is. It may be something like a "should" not a "must".)
- superbatfish 6y agoThe article contradicts your assertion. Did you read it?
- oddthink 6y agoWow, a little hostile here? My assertion, that R's NaN is not "non-standard", seems upheld by the article. It's a quiet NaN with a payload, which is well-defined by the IEEE 754 standard. As other posters pointed out, it's relying on a "should" behavior from the spec, which is risky but common. It sounds like disabling the "RunFast" mode cleared up their issues, which seems quite far from it being an "obscenely bad" design decision. It's not terribly unusual to require IEEE 754 compliance in numerical code, like the usual options for avoiding --ffast-math -style stuff.
- superbatfish 6y agoFair enough. My snide question was uncalled for. Sorry. Thanks for the additional info.
- brandmeyer 6y agoIt is a "should", not a "shall". Quoth the standard (emphasis mine): > For an operation with quiet NaN inputs, other than maximum and minimum operations, if a floating-point result is to be delivered the result shall be a quiet NaN which should be one of the input NaNs.
- racl101 6y agoInstalling R on a MacBook, even the older 2019 ones, was nothing short of a fucking nightmare. Ended up installing on a vagrant machine instead.
- ImaCake 6y agoI‘ve had few problems with R on my 2020 13’ Macbook but several of my coworkers have struggled with R on theirs. Some of them are very new to programming and likely get stumped by what I would consider “simple“ bugs.
- psychometry 6y agoInstalling even common packages like data.table require mucking around with R's makevars. There's no common set of variables that "just work", since different packages need different compilers to install.
- creddit 6y agoThis is only true if you insist on installing with OpenMP support. I can think of maybe 3-5 packages, most relatively low use, that have intricacies required to install.
- creddit 6y agoI think maybe once in the past 6yrs I’ve had an issue with `brew install R` and I’m a power R user (upgrade regularly). How we were you attempting to install? Build from source?
- kristjansson 6y agoInstalling R through conda is a PITA esp. for packages that aren’t in conda-forge yet, I’ll give you that. Installing through homebrew or using the R project builds is very smooth in my experience
- gnufx 6y agoApple is obviously a contrast with the usual state of affairs where one of the first signs of a new CPU is in the GNU toolchain.
- Will_Do 6y agoQuestion: Would using R inside Docker on one of these Macs work somewhat well? Previous benchmarks[0] show that the overhead on Intel Macbooks for the Docker Linux VM is quite low for scientific computing. Would the x86 emulation hurt performance substantially or is there some other issue with this approach? [0]: https://lemire.me/blog/2020/06/19/computational-overhead-due-to-docker-under-macos/ https://lemire.me/blog/2020/06/19/computational-overhead-due...
- ryukafalz 6y agoI would imagine that Docker for Mac will likely get native support for ARM macOS soon enough, in which case there'd be no x86 emulation involved and you could run the ARM Linux version of R in a container just fine. My understanding is Rosetta 2 does not support x86 virtualization.
- mr_toad 6y agoWhy would you run R on a Mac in Docker? Docker isn’t an emulator. You’re still going to need ARM code.
- dehrmann 6y agoNot that this wasn't a known caveat of Docker, but I think a lot of people are going to realize this in the next year or two.
- jp0d 6y agoI reckon they'd finally have to get R working natively on the new chip. I don't foresee Apple offering the fat binary support in the long term. It's probably only an intermediate solution for the transitional period. Also, does it mean the native version of R will finally work on the iPad? I know Apple doesn't allow compilers but there are a few examples like Pythonista and Apple's own Swift playground. It'd be cool to get R Studio on the iPad.
- Wowfunhappy 6y agoJust to be clear, PPC-Intel fat/universal binaries are still supported even on Big Sur, the PPC portion is just ignored. I don't expect Intel-Arm binaries to go away any time soon. I believe what you're really thinking of is Rosetta though. That, indeed, is sadly unlikely to be around forever. We have history as an indication of that.
- jp0d 6y agoYes, you're right. I meant Rosetta2. It's good to have native binaries nevertheless.
- RandallBrown 6y agoWhen Apple transitioned from PowerPC to Intel the fat binary support (Rosetta) lasted 3 OS updates or about 3 years. Definitely won't be a super long term thing, but there's plenty of time I guess.
- ineedasername 6y agoAn inability to use R Studio would be a deal breaker for me.
- olliej 6y agoIt should run fine under rosetta, if anyone encounters issues please submit bug reports.
- istvan60 6y agoHi everyone, did any of you try out R or SPSS under a new M1 Macbook, do either of these work fine under Rosetta 2, as I suppose none has a native ARM version yet. In addition, did anyone try CorelDraw as well? I am asking these question, because I think a lot of us working in data science have second thoughts about moving to ARM, at least for the next year or so....