The Apache Arrow C++ library provides rich, powerful features for working with columnar data. The `arrow` R package provides both a low-level interface to the C++ library and some higher-level, R-flavored tools for working with it. This vignette provides an overview of how the pieces fit together, and it describes the conventions that the classes and methods follow in R.

# Features

## Multi-file datasets

The `arrow` package lets you work efficiently with large, multi-file datasets using `dplyr` methods. See `vignette("dataset", package = "arrow")` for an overview.

## Reading and writing files

`arrow` provides some simple functions for using the Arrow C++ library to read and write files. These functions are designed to drop into your normal R workflow without requiring any knowledge of the Arrow C++ library and use naming conventions and arguments that follow popular R packages, particularly `readr`. The readers return `data.frame`s (or if you use the `tibble` package, they will act like `tbl_df`s), and the writers take `data.frame`s.

Importantly, `arrow` provides basic read and write support for the Apache Parquet columnar data file format.

    library(arrow)
    df <- read_parquet("path/to/file.parquet")

Just as you can read, you can write Parquet files:

    write_parquet(df, "path/to/different_file.parquet")

The `arrow` package also includes a faster and more robust implementation of the Feather file format, providing `read_feather()` and `write_feather()`. This implementation depends on the same underlying C++ library as the Python version does, resulting in more reliable and consistent behavior across the two languages, as well as improved performance. `arrow` also by default writes the Feather V2 format, which supports a wider range of data types, as well as compression.

For CSV and line-delimited JSON, there are `read_csv_arrow()` and `read_json_arrow()`, respectively. While `read_csv_arrow()` currently has fewer parsing options for dealing with every CSV format variation in the wild, for the files it can read, it is often significantly faster than other R CSV readers, such as `base::read.csv`, `readr::read_csv`, and `data.table::fread`.

## Working with Arrow data in Python

Using `reticulate`, `arrow` lets you share data between R and Python (`pyarrow`) efficiently, enabling you to take advantage of the vibrant ecosystem of Python packages that build on top of Apache Arrow. See `vignette("python", package = "arrow")` for details.

## Access to Arrow messages, buffers, and streams

The `arrow` package also provides many lower-level bindings to the C++ library, which enable you to access and manipulate Arrow objects. You can use these to build connectors to other applications and services that use Arrow. One example is Spark: the `sparklyr` package has support for using Arrow to move data to and from Spark, yielding significant performance gains.

# Internals

## Mapping of R <–> Arrow types

Arrow has a rich data type system that includes direct parallels with R's data types and much more.

In the tables, entries with a `-` are not currently implemented.

### R to Arrow

R type | Arrow type
--- | ---
logical | boolean
integer | int32
double (“numeric”) | float64
character | utf81^
factor | dictionary
raw | uint8
Date | date32
POSIXct | timestamp
POSIXlt | struct
data.frame | struct
list2^ | list
bit64::integer64 | int64
difftime | time32
vctrs::vctrs_unspecified | null

1: If the character vector exceeds 2GB of strings, it will be converted to a `large_utf8` Arrow type

2: Only lists where all elements are the same type are able to be translated to Arrow list type (which is a “list of” some type).

### Arrow to R

Arrow type | R type
--- | ---
boolean | logical
int8 | integer
int16 | integer
int32 | integer
int64 | integer3^
uint8 | integer
uint16 | integer
uint32 | integer3^
uint64 | integer3^
float16 | -
float32 | double
float64 | double
utf8 | character
binary | arrow_binary 5^
fixed_size_binary | arrow_fixed_size_binary 5^
date32 | Date
date64 | POSIXct
time32 | hms::difftime
time64 | hms::difftime
timestamp | POSIXct
duration | -
decimal | double
dictionary | factor4^
list | arrow_list 6^
fixed_size_list | arrow_fixed_size_list 6^
struct | data.frame
null | vctrs::vctrs_unspecified
map | -
union | -
large_utf8 | character
large_binary | arrow_large_binary 5^
large_list | arrow_large_list 6^

3: These integer types may contain values that exceed the range of R's `integer` type (32-bit signed integer). When they do, `uint32` and `uint64` are converted to `double` (“numeric”) and `int64` is converted to `bit64::integer64`.

4: Due to the limitation of R `factor`s, Arrow `dictionary` values are coerced to string when translated to R if they are not already strings.

5: `arrow*_binary` classes are implemented as lists of raw vectors.

6: `arrow*_list` classes are implemented as subclasses of `vctrs_list_of` with a `ptype` attribute set to what an empty Array of the value type converts to.

### R object attributes

Arrow supports custom key-value metadata attached to Schemas. When we convert a `data.frame` to an Arrow Table or RecordBatch, the package stores any `attributes()` attached to the columns of the `data.frame` in the Arrow object's Schema. These attributes are stored under the “r” key; you can assign additional string metadata under any other key you wish, like `x$metadata$new_key <- "new value"`.

This metadata is preserved when writing the table to Feather or Parquet, and when reading those files into R, or when calling `as.data.frame()` on a Table/RecordBatch, the column attributes are restored to the columns of the resulting `data.frame`. This means that custom data types, including `haven::labelled`, `vctrs` annotations, and others, are preserved when doing a round-trip through Arrow.

Note that the `attributes()` stored in `$metadata$r` are only understood by R. If you write a `data.frame` with `haven` columns to a Feather file and read that in Pandas, the `haven` metadata won't be recognized there. (Similarly, Pandas writes its own custom metadata, which the R package does not consume.) You are free, however, to define custom metadata conventions for your application and assign any (string) values you want to other metadata keys.

## Class structure and package conventions

C++ is an object-oriented language, so the core logic of the Arrow library is encapsulated in classes and methods. In the R package, these classes are implemented as `R6` reference classes, most of which are exported from the namespace.

In order to match the C++ naming conventions, the `R6` classes are in TitleCase, e.g. `RecordBatch`. This makes it easy to look up the relevant C++ implementations in the code or documentation. To simplify things in R, the C++ library namespaces are generally dropped or flattened; that is, where the C++ library has `arrow::io::FileOutputStream`, it is just `FileOutputStream` in the R package. One exception is for the file readers, where the namespace is necessary to disambiguate. So `arrow::csv::TableReader` becomes `CsvTableReader`, and `arrow::json::TableReader` becomes `JsonTableReader`.

Some of these classes are not meant to be instantiated directly; they may be base classes or other kinds of helpers. For those that you should be able to create, use the `$create()` method to instantiate an object. For example, `rb <- RecordBatch$create(int = 1:10, dbl = as.numeric(1:10))` will create a `RecordBatch`. Many of these factory methods that an R user might most often encounter also have a `snake_case` alias, in order to be more familiar for contemporary R users. So `record_batch(int = 1:10, dbl = as.numeric(1:10))` would do the same as `RecordBatch$create()` above.

The typical user of the `arrow` R package may never deal directly with the `R6` objects. We provide more R-friendly wrapper functions as a higher-level interface to the C++ library. An R user can call `read_parquet()` without knowing or caring that they're instantiating a `ParquetFileReader` object and calling the `$ReadFile()` method on it. The classes are there and available to the advanced programmer who wants fine-grained control over how the C++ library is used.