Installing R on Osx Big Sur (Edit: and Apple M1) for Use with Rcpp and Openmp

Installing R on OSX Big Sur (EDIT: and Apple M1) for use with Rcpp and openMP

Eventually, I found a process that works on a M1 mac with Big Sur.

Head over to https://mac.r-project.org/, it contains most things you will need
Download and install R via R-4.1-branch.pkg. The CRAN version might also work, but I used the installer from mac.r-project.org, which required opening the osx security settings to allow the installation.
Install RStudio, start it, and let it install the developer tools. Alternatively, run sudo xcode-select --install in Terminal.
Head to https://mac.r-project.org/openmp/. Download openmp-11.0.1-darwin20-Release.tar.gz and install it (see Terminal commands below).

curl -O https://mac.r-project.org/openmp/openmp-11.0.1-darwin20-Release.tar.gz
sudo tar fvx openmp-11.0.1-darwin20-Release.tar.gz -C /

Now we need to add compiler flags so that clan uses openMP. In Terminal, create the Makevars file.

cd ~
mkdir .R
nano .R/Makevars

in nano, paste these additional compiler flags into the Makevars file:

CPPFLAGS += -Xclang -fopenmp
LDFLAGS += -lomp

Hit Control+O, Control+X to save and close

Head over to the gfortran page: https://github.com/fxcoudert/gfortran-for-macOS/releases
Use the installer gfortran-ARM-11.0-BigSur.pkg to install gfortran.
For some reason it appears to install in /usr/local/gfortran, but R expects it in /opt. The mac-R team likes to separate arm64 and intel related files. We could go and fix paths, or simply also install gfortran under /opt. Download the tar file gfortran-ARM-11.0-BigSur.tar.xz. You can use curl, or just download it and point tar in the command line to it.

cd /opt/R/arm64/
sudo mkdir gfortran
sudo tar -xzyf gfortran-ARM-11.0-BigSur.tar.xz -C /opt/R/arm64/

(replace gfortran-ARM-11.0-BigSur.tar.xz with /users/YOURUSERNAME/downloads/gfortran-ARM-11.0-BigSur.tar.xz)

Now it should work.

Not an expert in OSX, but doing this so others can figure out how to use my R package. I'd like to streamline the process some more, but wiping the mac, reinstalling osx and testing it takes so much time.

Configuring compilers on Mac M1 (Big Sur, Monterey) for Rcpp and other tools

Background

Currently (2022-04-24), CRAN builds R 4.2 binaries for Apple silicon using Apple clang from Command Line Tools for Xcode 13.1 and using an experimental fork of GNU Fortran 12.

If you obtain R from CRAN (i.e., here), then you need to replicate CRAN's compiler setup on your system before building R packages that contain C/C++/Fortran code from their sources (and before using Rcpp, etc.). This requirement ensures that your package builds are compatible with R itself.

A further complication is the fact that Apple clang doesn't support OpenMP, so you need to do even more work to compile programs that make use of multithreading. You could circumvent the issue by building R itself and all R packages from sources with LLVM clang, which does support OpenMP, but that approach is onerous and "for experts only".

There is another approach that has been tested by a few people, including Simon Urbanek, the maintainer of R for macOS. It is experimental and also "for experts only", but it works on my machine and is much simpler than learning to build R yourself.

Instructions for obtaining a working toolchain

Warning: These come with no warranty and could break at any time. Some level of familiarity with C/C++/Fortran program compilation, Makefile syntax, and Unix shells is assumed. Everyone is encouraged to consult official documentation, which is more likely to be maintained than answers on SO. As usual, sudo at your own risk.

I will try to address compilers and OpenMP support at the same time. I am going to assume that you are starting from nothing. Feel free to skip steps you've already taken, though you might find a fresh start helpful.

I've tested these instructions on a machine running Big Sur, and at least one person has tested them on a machine running Monterey. I would be glad to hear from others.

Download an R 4.2 binary from CRAN here and install. Be sure to select the binary built for Apple silicon.
Run
```
$ sudo xcode-select --install
```
in Terminal to install the latest release version of Apple's Command Line Tools for Xcode, which includes Apple clang. You can obtain earlier versions from your browser here. However, the version that you install should not be older than the one that CRAN used to build your R binary.

Download the gfortran binary recommended here and install by unpacking to root:

$ curl -LO https://mac.r-project.org/tools/gfortran-12.0.1-20220312-is-darwin20-arm64.tar.xz
$ sudo tar xvf gfortran-12.0.1-20220312-is-darwin20-arm64.tar.xz -C /
$ sudo ln -sfn $(xcrun --show-sdk-path) /opt/R/arm64/gfortran/SDK

The last command updates a symlink inside of the gfortran installation so that it points to the SDK inside of your Command Line Tools installation.

Download an OpenMP runtime suitable for your Apple clang version here and install by unpacking to root. You can query your Apple clang version with clang --version. For example, I have version 1300.0.29.3, so I did:
```
$ curl -LO https://mac.r-project.org/openmp/openmp-12.0.1-darwin20-Release.tar.gz
$ sudo tar xvf openmp-12.0.1-darwin20-Release.tar.gz -C /
```
After unpacking, you should find these files on your system:
```
/usr/local/lib/libomp.dylib
/usr/local/include/ompt.h
/usr/local/include/omp.h
/usr/local/include/omp-tools.h
```

Add the following lines to $(HOME)/.R/Makevars, creating the file if necessary.

CPPFLAGS+=-I/usr/local/include -Xclang -fopenmp
LDFLAGS+=-L/usr/local/lib -lomp

FC=/opt/R/arm64/gfortran/bin/gfortran -mtune=native
FLIBS=-L/opt/R/arm64/gfortran/lib/gcc/aarch64-apple-darwin20.6.0/12.0.1 -L/opt/R/arm64/gfortran/lib -lgfortran -lemutls_w -lm

Run R and test that you can compile a program with OpenMP support. For example:

if (!requireNamespace("RcppArmadillo", quietly = TRUE)) {
    install.packages("RcppArmadillo")
}
Rcpp::sourceCpp(code = '
#include <RcppArmadillo.h>
#ifdef _OPENMP
# include <omp.h>
#endif

// [[Rcpp::depends(RcppArmadillo)]]
// [[Rcpp::export]]
void omp_test()
{
#ifdef _OPENMP
    Rprintf("OpenMP threads available: %d\\n", omp_get_max_threads());
#else
    Rprintf("OpenMP not supported\\n");
#endif
}
')
omp_test()

OpenMP threads available: 8

If the C++ code fails to compile, or if it compiles without error but you get linker warnings or you find that OpenMP is not supported, then one of us has probably made a mistake. Please report any issues.

References

Everything is a bit scattered:

R Installation and Administration manual [link]
R for macOS Developers page [link]

Installing gfortran on MacBook with Apple M1 chip for use in R

I deleted everything gfortran related and started over.

I downloaded the .tar.xz file here using the browser, which dumped the file into my Downloads folder. I double-clicked it to unpack it.

I moved that directory to where R wanted it:

markwhite@marks-air ~ % sudo mv Downloads/gfortran /opt/R/arm64/

Then I restarted R and tried again. It threw me some Fortran compilation messages—which I will not pretend to comprehend here—but it successfully installed:

> install.packages("mvtnorm")

  There is a binary version available but the source version is later:
        binary source needs_compilation
mvtnorm  1.1-2  1.1-3              TRUE

Do you want to install from sources the package which needs compilation? (Yes/no/cancel) Yes
installing the source package ‘mvtnorm’

trying URL 'https://cran.rstudio.com/src/contrib/mvtnorm_1.1-3.tar.gz'
Content type 'application/x-gzip' length 166421 bytes (162 KB)
==================================================
downloaded 162 KB

* installing *source* package ‘mvtnorm’ ...
** package ‘mvtnorm’ successfully unpacked and MD5 sums checked
** using staged installation
** libs
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -c C_FORTRAN_interface.c -o C_FORTRAN_interface.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -c miwa.c -o miwa.o
/opt/R/arm64/bin/gfortran -mtune=native -fno-optimize-sibling-calls  -fPIC  -Wall -g -O2  -c mvt.f -o mvt.o
mvt.f:861:11:

  861 |       hs = sign( one, dh - r*dk )
      |           1
Warning: Possible change of value in conversion from REAL(8) to INTEGER(4) at (1) [-Wconversion]
mvt.f:862:11:

  862 |       ks = sign( one, dk - r*dh )
      |           1
Warning: Possible change of value in conversion from REAL(8) to INTEGER(4) at (1) [-Wconversion]
mvt.f:1155:16:

 1155 |             K = MOD( C(NP, MIN(NDIM-1,KLIM-1))*DBLE(K), DBLE(P(NP)) )
      |                1
Warning: Possible change of value in conversion from REAL(8) to INTEGER(4) at (1) [-Wconversion]
mvt.f:1215:17:

 1215 |             JP = 1 + J*R(J)
      |                 1
Warning: Possible change of value in conversion from REAL(8) to INTEGER(4) at (1) [-Wconversion]
mvt.f:94:39:

   94 |       DOUBLE PRECISION COV(NL*(NL+1)/2), A(NL), B(NL), DL(NL), Y(NL)
      |                                       1
Warning: Array ‘cov’ at (1) is larger than limit set by ‘-fmax-stack-var-size=’, moved from stack to static storage. This makes the procedure unsafe when called recursively, or concurrently from multiple threads. Consider using ‘-frecursive’, or increase the ‘-fmax-stack-var-size=’ limit, or change the code to use an ALLOCATABLE array. [-Wsurprising]
mvt.f:86:33:

   86 |       SUBROUTINE MVSUBR( N, W, NF, F )
      |                                 1
Warning: Unused dummy argument ‘nf’ at (1) [-Wunused-dummy-argument]
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -c mvtnorm-init.c -o mvtnorm-init.o
/opt/R/arm64/bin/gfortran -mtune=native -fno-optimize-sibling-calls  -fPIC  -Wall -g -O2  -c tvpack.f -o tvpack.o
tvpack.f:395:14:

  395 |          HS = SIGN( ONE, DH - R*DK )
      |              1
Warning: Possible change of value in conversion from REAL(8) to INTEGER(4) at (1) [-Wconversion]
tvpack.f:396:14:

  396 |          KS = SIGN( ONE, DK - R*DH )
      |              1
Warning: Possible change of value in conversion from REAL(8) to INTEGER(4) at (1) [-Wconversion]
tvpack.f:60:52:

   60 |       DOUBLE PRECISION ONE, ZRO, EPS, ZROS(3), HS(3), TVT, TVTL
      |                                                    1
Warning: Unused variable ‘hs’ declared at (1) [-Wunused-variable]
tvpack.f:60:45:

   60 |       DOUBLE PRECISION ONE, ZRO, EPS, ZROS(3), HS(3), TVT, TVTL
      |                                             1
Warning: Unused variable ‘zros’ declared at (1) [-Wunused-variable]
tvpack.f:220:72:

  220 |       END
      |                                                                        ^
Warning: ‘fin’ may be used uninitialized in this function [-Wmaybe-uninitialized]
tvpack.f:196:58:

  196 |       DOUBLE PRECISION EI(NL), AI(NL), BI(NL), FI(NL), FIN, ERR, KRNRDT
      |                                                          ^
note: ‘fin’ was declared here
clang -arch arm64 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/opt/R/arm64/lib -o mvtnorm.so C_FORTRAN_interface.o miwa.o mvt.o mvtnorm-init.o tvpack.o -L/opt/R/arm64/gfortran/lib/gcc/aarch64-apple-darwin20.2.0/11.0.0 -L/opt/R/arm64/gfortran/lib -lgfortran -lemutls_w -lm -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
ld: warning: could not create compact unwind for _mvkrsv_: register 75 saved somewhere other than in frame
ld: warning: could not create compact unwind for _mvkbrv_: stack size is too large for frameless function
ld: warning: could not create compact unwind for _mvstdt_: register 75 saved somewhere other than in frame
ld: warning: could not create compact unwind for _mvbvtl_: registers 78 and 79 not saved contiguously in frame
ld: warning: could not create compact unwind for _pntgnd_: registers 72 and 73 not saved contiguously in frame
ld: warning: could not create compact unwind for _mvbvu_: registers 72 and 73 not saved contiguously in frame
ld: warning: could not create compact unwind for _tvtmfn_: register 73 saved somewhere other than in frame
ld: warning: could not create compact unwind for _krnrdt_: registers 74 and 75 not saved contiguously in frame
ld: warning: could not create compact unwind for _mvbvn_: registers 21 and 22 not saved contiguously in frame
ld: warning: could not create compact unwind for _bvnd_: registers 78 and 79 not saved contiguously in frame
ld: warning: could not create compact unwind for _bvtl_: registers 78 and 79 not saved contiguously in frame
ld: warning: could not create compact unwind for _tvtlrcall_: registers 72 and 73 not saved contiguously in frame
ld: warning: could not create compact unwind for _mvbvt_: registers 21 and 22 not saved contiguously in frame
ld: warning: could not create compact unwind for _mvspcl_: register 73 saved somewhere other than in frame
ld: warning: could not create compact unwind for _mvsort_: register 77 saved somewhere other than in frame
ld: warning: could not create compact unwind for _mvvlsb_: registers 27 and 28 not saved contiguously in frame
ld: warning: could not create compact unwind for _mvtdst_: registers 23 and 24 not saved contiguously in frame
ld: warning: could not create compact unwind for _mvbvtc_: register 73 saved somewhere other than in frame
installing to /Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/library/00LOCK-mvtnorm/00new/mvtnorm/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (mvtnorm)

The downloaded source packages are in
    ‘/private/var/folders/fx/f46lfqf56_df59fvzgqhcbyr0000gn/T/RtmptXzvKK/downloaded_packages’

How to download Sf package on MacOS

The error message lines "Warning in install.packages : downloaded length 68510897 != reported length 93022013" and "Timeout of 60 seconds was reached" suggests the package didn't download properly - this can occur when it takes longer than 60 seconds to download the package to your computer for installation i.e. the "timeout" of 60 seconds is reached.

One solution (the solution that worked in this case) is to increase the amount of time R will wait to download the package by running options(timeout = 1200) before running install.packages("sf")

If you need to build packages from source (with or without openMP) on an M1 mac, I believe the instructions posted here work: https://stackoverflow.com/a/68275558/12957340

Installing R on Osx Big Sur (Edit: and Apple M1) for Use with Rcpp and Openmp