---
title: "Data cleaning"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{data_handing}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

Data cleaning of high-frequency data is a necessary step in all finance and financial econometrics applications. The reason is that most data providers like tick data offer raw instead of preprocessed data. The upside is that you can follow the cleaning process yourself instead of relying on another party. In this vignette, we will address a common workflow that makes use of both trades and quotes data on a microsecond resolution.

## Cleaning tick data on trades

We start by loading the highfrequency package and printing the head of one example data set that contains trade data for one stock on two days.

```{r}
library(highfrequency)
head(sample_tdataraw_microseconds)
```

A quick look via summary into the data let's us see that there are trades listed in the data set out of the bounds of the regular trading hours that are 9:30 to 16:00 on regular work days.

```{r eval = TRUE}
summary(sample_tdataraw_microseconds[, c("DT", "SIZE", "PRICE")])
```

Next, we will use one of three main cleaning functions called tradesCleanup. It implements the steps laid out in the seminal paper of Barndorff-Nielsen et al. (2009). It processes the given data by

- deleting entries with a price reported as zero,
- selecting trades from a single exchange ("N" stands for New York Stock Exchange (NYSE))
- filtering for the sales condition to be either "E" or "F", corresponding to cases of regular trade records and
- summarizing trades that have the same time stamp. The default is to take the median price and summed-up volume of the different trades.

```{r eval = TRUE}
tdata_cleaned <- tradesCleanup(tdataraw = sample_tdataraw_microseconds, exchange = "N")
```

When using tradesCleanup, the return value is a list of two. It contains a report about the number of observations after each cleaning step and the cleaned data itself.

```{r eval = TRUE}
tdata_cleaned$report

summary(tdata_cleaned$tdata[, c("DT", "SIZE", "PRICE")])
```

As just one example, one can see that all trades that had time stamps outside the openinig hours of the NYSE. The report of the cleaning procedure shows us that most of the observations are removed either due to originating from another exchange or having irregular sales conditions.

As a last step for cleaning tick data on trades it is advised to check whether compatible quotes have been made before the recorded transaction. For doing so, we first need to implement a similar cleaning procedure for quotes data.

## Cleaning tick data on quotes

The highfrequency function also brings a similarly-named function for cleaning quotes data, quotesCleanup. Here, the cleaning steps are

- removing quotes with zero price,
- selecting a single exchange,
- removing entries that have extraordinarly large spreads
- merging quotes that happen on the same time and
- removing quotes that may be seen to be outliers.

```{r eval = TRUE}
qdata_cleaned <- quotesCleanup(qdataraw = sample_qdataraw_microseconds, exchange = "N")
```

quotesCleanup also provides a list with two elements, a report and the cleaned data.

```{r eval = TRUE}
qdata_cleaned$report

summary(qdata_cleaned$qdata[, c("DT", "OFR", "OFRSIZ", "BID", "BIDSIZ", "MIDQUOTE")])
```

## Cleaning transaction data using cleaned quotes

Given our freshly cleaned trades and quotes data, we now want to filter for trades that are proceeded by a compatible set of quotes. Currently, the tradesCleanupUsingQuotes function only supports submitting trades and quotes data for one day only, Hence, we filter our data set to one of the days.

```{r eval = TRUE}
tqdata_cleaned <- tradesCleanupUsingQuotes(tdata = tdata_cleaned$tdata[as.Date(DT) == "2018-01-02"], 
                                           qdata = qdata_cleaned$qdata[as.Date(DT) == "2018-01-02"])
tqdata_cleaned
```

Here, we see that once more half of the trades are filtered out due to non-plausability.

## References

Barndorff-Nielsen, O. E., P. R. Hansen, A. Lunde, and N. Shephard (2009). Realized kernels in practice: Trades and quotes. Econometrics Journal 12, C1-C32.

