# Installing the Arrow Package on Linux

TL;DR

In most cases, install.packages("arrow") should just work. There are things you can do to make the installation faster (see below). If, for any reason, it doesn’t work, set the environment variable ARROW_R_DEV=true, retry, and share the logs with us.

The Apache Arrow project is implemented in multiple languages, and the R package depends on the Arrow C++ library (referred to from here on as libarrow). This means that when you install arrow, you need both the R and C++ versions. If you install arrow from CRAN on a machine running Windows or MacOS, when you call install.packages("arrow"), a precompiled binary containing both the R package and libarrow will be downloaded. However, CRAN does not host R package binaries for Linux, and so you must choose from one of the alternative approaches.

This vignette outlines the recommend approaches to installing arrow on Linux, starting from the simplest and least customisable to the most complex but with more flexbility to customise your installation.

The intended audience for this document is arrow R package users on Linux, and not Arrow developers. If you’re contributing to the Arrow project, see vignette("developing", package = "arrow") for resources to help you on set up your development environment. You can also find a more detailed discussion of the code run during the installation process in the developers’ installation docs

Having trouble installing arrow? See the “Troubleshooting” section below.

# System dependencies

The arrow package is designed to work with very minimal system requirements, but there are a few things to note.

## Compilers

As of version 10.0.0, arrow requires a C++17 compiler to build. For gcc, this generally means version 7 or newer. Most contemporary Linux distributions have a new enough compiler; however, CentOS 7 is a notable exception, as it ships with gcc 4.8.

If you are on CentOS 7, to build arrow you will need to install a newer devtoolset, and you’ll need to update R’s Makevars to define the CXX17 variables. This script installs devtoolset-8 and configures R to be able to use C++17:

#!/usr/bin/env bash

yum install -y centos-release-scl
yum install -y devtoolset-8
# Optional: also install cloud storage dependencies, as described below
yum install -y libcurl-devel openssl-devel

source /opt/rh/devtoolset-8/enable

if [ ! R CMD config CXX17 ]; then
mkdir -p ~/.R
echo "CC = $(which gcc) -fPIC" >> ~/.R/Makevars echo "CXX17 =$(which g++) -fPIC" >> ~/.R/Makevars
echo "CXX17STD = -std=c++17" >> ~/.R/Makevars

# Install the nightly build

Daily development builds, which are not official releases, can be installed from the Ursa Labs repository:

Sys.setenv(NOT_CRAN = TRUE)
install.packages("arrow", repos = c(arrow = "https://nightlies.apache.org/arrow/r", getOption("repos")))

or for conda users via:

conda install -c arrow-nightlies -c conda-forge --strict-channel-priority r-arrow

# Install from git repo

You can also install the R package from a git checkout:

git clone https://github.com/apache/arrow
cd arrow/r
R CMD INSTALL .

If you don’t already have libarrow on your system, when installing the R package from source, it will also download and build libarrow for you. See the section above on build environment variables for options for configuring the build source and enabled features.

# Installation using install_arrow()

The previous instructions are useful for a fresh arrow installation, but arrow provides the function install_arrow(), which you can use if you:

• already have arrow installed and want to upgrade to a different version
• want to install a development build
• want to try to reinstall and fix issues with Linux C++ binaries

install_arrow() provides some convenience wrappers around the various environment variables described below.

Although this function is part of the arrow package, it is also available as a standalone script, so you can access it for convenience without first installing the package:

source("https://raw.githubusercontent.com/apache/arrow/master/r/R/install-arrow.R")

## Install the latest release

install_arrow()

## Install the nightly build

install_arrow(nightly = TRUE)

## Install with more verbose output for debugging errors

install_arrow(verbose = TRUE)

install_arrow() does not require environment variables to be set in order to satisfy C++ dependencies.

Note that, unlike packages like tensorflow, blogdown, and others that require external dependencies, you do not need to run install_arrow() after a successful arrow installation.

# Offline installation

The install-arrow.R file also includes the create_package_with_all_dependencies() function. Normally, when installing on a computer with internet access, the build process will download third-party dependencies as needed. This function provides a way to download them in advance.

Doing so may be useful when installing Arrow on a computer without internet access. Note that Arrow can be installed on a computer without internet access without doing this, but many useful features will be disabled, as they depend on third-party components. More precisely, arrow::arrow_info()\$capabilities() will be FALSE for every capability. One approach to add more capabilities in an offline install is to prepare a package with pre-downloaded dependencies. The create_package_with_all_dependencies() function does this preparation.

If you’re using binary packages you shouldn’t need to follow these steps. You should download the appropriate binary from your package repository, transfer that to the offline computer, and install that. Any OS can create the source bundle, but it cannot be installed on Windows. (Instead, use a standard Windows binary package.)

Note if you’re using RStudio Package Manager on Linux: If you still want to make a source bundle with this function, make sure to set the first repo in options("repos") to be a mirror that contains source packages (that is: something other than the RSPM binary mirror URLs).

### Step 1 - Using a computer with internet access, pre-download the dependencies:

• Install the arrow package or run source("https://raw.githubusercontent.com/apache/arrow/master/r/R/install-arrow.R")
• Run create_package_with_all_dependencies("my_arrow_pkg.tar.gz")
• Copy the newly created my_arrow_pkg.tar.gz to the computer without internet access

### Step 2 - On the computer without internet access, install the prepared package:

• Install the arrow package from the copied file
• install.packages("my_arrow_pkg.tar.gz", dependencies = c("Depends", "Imports", "LinkingTo"))
• This installation will build from source, so cmake must be available
• Run arrow_info() to check installed capabilities

#### Alternative, hands-on approach

• Download the dependency files (cpp/thirdparty/download_dependencies.sh may be helpful)
• Copy the directory of dependencies to the offline computer
• Create the environment variable ARROW_THIRDPARTY_DEPENDENCY_DIR on the offline computer, pointing to the copied directory.
• Install the arrow package as usual.

# Troubleshooting

The intent is that install.packages("arrow") will just work and handle all C++ dependencies, but depending on your system, you may have better results if you tune one of several parameters. Here are some known complications and ways to address them.

## Package failed to build C++ dependencies

If you see a message like

------------------------- NOTE ---------------------------
There was an issue preparing the Arrow C++ libraries.
See https://arrow.apache.org/docs/r/articles/install.html
---------------------------------------------------------

in the output when the package fails to install, that means that installation failed to retrieve or build the libarrow version compatible with the current version of the R package.

Please check the “Known installation issues” below to see if any apply, and if none apply, set the environment variable ARROW_R_DEV=TRUE for more verbose output and try installing again. Then, please report an issue and include the full installation output.

## Using system libraries

If a system library or other installed Arrow is found but it doesn’t match the R package version (for example, you have libarrow 1.0.0 on your system and are installing R package 2.0.0), it is likely that the R bindings will fail to compile. Because the Apache Arrow project is under active development, it is essential that versions of libarrow and the R package matches. When install.packages("arrow") has to download libarrow, the install script ensures that you fetch the libarrow version that corresponds to your R package version. However, if you are using a version of libarrow already on your system, version match isn’t guaranteed.

To fix version mismatch, you can either update your libarrow system packages to match the R package version, or set the environment variable ARROW_USE_PKG_CONFIG=FALSE to tell the configure script not to look for system version of libarrow. (The latter is the default of install_arrow().) System libarrow versions are available corresponding to all CRAN releases but not for nightly or dev versions, so depending on the R package version you’re installing, system libarrow version may not be an option.

Note also that once you have a working R package installation based on system (shared) libraries, if you update your system libarrow installation, you’ll need to reinstall the R package to match its version. Similarly, if you’re using libarrow system libraries, running update.packages() after a new release of the arrow package will likely fail unless you first update the libarrow system packages.

## Using prebuilt binaries

If the R package finds and downloads a prebuilt binary of libarrow, but then the arrow package can’t be loaded, perhaps with “undefined symbols” errors, please report an issue. This is likely a compiler mismatch and may be resolvable by setting some environment variables to instruct R to compile the packages to match libarrow.

A workaround would be to set the environment variable LIBARROW_BINARY=FALSE and retry installation: this value instructs the package to build libarrow from source instead of downloading the prebuilt binary. That should guarantee that the compiler settings match.

If a prebuilt libarrow binary wasn’t found for your operating system but you think it should have been, please report an issue and share the console output. You may also set the environment variable ARROW_R_DEV=TRUE for additional debug messages.

## Building libarrow from source

If building libarrow from source fails, check the error message. (If you don’t see an error message, only the ----- NOTE -----, set the environment variable ARROW_R_DEV=TRUE to increase verbosity and retry installation.) The install script should work everywhere, so if libarrow fails to compile, please report an issue so that we can improve the script.

## Known installation issues

• On CentOS, building the package requires a more modern devtoolset than the default system compilers. See “System dependencies” above.

• If you have multiple versions of zstd installed on your system, installation by building libarrow from source may fail with an “undefined symbols” error. Workarounds include (1) setting LIBARROW_BINARY to use a C++ binary; (2) setting ARROW_WITH_ZSTD=OFF to build without zstd; or (3) uninstalling the conflicting zstd. See discussion here.

# Contributing

As mentioned above, please report an issue if you encounter ways to improve this. If you find that your Linux distribution or version is not supported, we welcome the contribution of Docker images (hosted on Docker Hub) that we can use in our continuous integration. These Docker images should be minimal, containing only R and the dependencies it requires. (For reference, see the images that R-hub uses.)

You can test the arrow R package installation using the docker-compose setup included in the apache/arrow git repository. For example,

R_ORG=rhub R_IMAGE=ubuntu-gcc-release R_TAG=latest docker-compose build r
R_ORG=rhub R_IMAGE=ubuntu-gcc-release R_TAG=latest docker-compose run r

installs the arrow R package, including libarrow, on the rhub/ubuntu-gcc-release image.