scrubr is a general purpose toolbox for cleaning biological occurrence records. Think of it like dplyr but specifically for occurrence data. It includes functionality for cleaning based on various aspects of spatial coordinates, unlikely values due to political centroids, taxonomic names, and more.
Install from CRAN
install.packages("scrubr")Or install the development version from GitHub
remotes::install_github("ropensci/scrubr")Load scrubr
library("scrubr")We’ll use sample datasets included with the package, they are lazy loaded, and available via sample_data_1 and sample_data_2
All functions expect data.frame’s as input, and output data.frame’s
We think that using a piping workflow with %>% makes code easier to build up, and easier to understand. However, in some examples below we provide commented out examples without the pipe to demonstrate traditional usage - which you can use if you remove the comment # at beginning of the line.
dframe() is a utility function to create a compact data.frame representation. You don’t have to use it. If you do, you can work with scrubr functions with a compact data.frame, making it easier to see the data quickly. If you don’t use dframe() we just use your regular data.frame. Problem is with large data.frame’s you deal with lots of stuff printed to the screen, making it hard to quickly wrangle data.
Remove impossible coordinates (using sample data included in the pkg)
# coord_impossible(dframe(sample_data_1)) # w/o pipe
dframe(sample_data_1) %>% coord_impossible()
#> # A tibble: 1,500 x 5
#> name longitude latitude date key
#> * <chr> <dbl> <dbl> <dttm> <int>
#> 1 Ursus americanus -79.7 38.4 2015-01-14 16:36:45 1065590124
#> 2 Ursus americanus -82.4 35.7 2015-01-13 00:25:39 1065588899
#> 3 Ursus americanus -99.1 23.7 2015-02-20 23:00:00 1098894889
#> 4 Ursus americanus -72.8 43.9 2015-02-13 16:16:41 1065611122
#> 5 Ursus americanus -72.3 43.9 2015-03-01 20:20:45 1088908315
#> 6 Ursus americanus -109. 32.7 2015-03-29 17:06:54 1088932238
#> 7 Ursus americanus -109. 32.7 2015-03-29 17:12:50 1088932273
#> 8 Ursus americanus -124. 40.1 2015-03-28 23:00:00 1132403409
#> 9 Ursus americanus -78.3 36.9 2015-03-20 21:11:24 1088923534
#> 10 Ursus americanus -76.8 35.5 2015-04-05 23:00:00 1088954559
#> # … with 1,490 more rowsRemove incomplete coordinates
# coord_incomplete(dframe(sample_data_1)) # w/o pipe
dframe(sample_data_1) %>% coord_incomplete()
#> # A tibble: 1,306 x 5
#> name longitude latitude date key
#> * <chr> <dbl> <dbl> <dttm> <int>
#> 1 Ursus americanus -79.7 38.4 2015-01-14 16:36:45 1065590124
#> 2 Ursus americanus -82.4 35.7 2015-01-13 00:25:39 1065588899
#> 3 Ursus americanus -99.1 23.7 2015-02-20 23:00:00 1098894889
#> 4 Ursus americanus -72.8 43.9 2015-02-13 16:16:41 1065611122
#> 5 Ursus americanus -72.3 43.9 2015-03-01 20:20:45 1088908315
#> 6 Ursus americanus -109. 32.7 2015-03-29 17:06:54 1088932238
#> 7 Ursus americanus -109. 32.7 2015-03-29 17:12:50 1088932273
#> 8 Ursus americanus -124. 40.1 2015-03-28 23:00:00 1132403409
#> 9 Ursus americanus -78.3 36.9 2015-03-20 21:11:24 1088923534
#> 10 Ursus americanus -76.8 35.5 2015-04-05 23:00:00 1088954559
#> # … with 1,296 more rowsRemove unlikely coordinates (e.g., those at 0,0)
# coord_unlikely(dframe(sample_data_1)) # w/o pipe
dframe(sample_data_1) %>% coord_unlikely()
#> # A tibble: 1,488 x 5
#> name longitude latitude date key
#> * <chr> <dbl> <dbl> <dttm> <int>
#> 1 Ursus americanus -79.7 38.4 2015-01-14 16:36:45 1065590124
#> 2 Ursus americanus -82.4 35.7 2015-01-13 00:25:39 1065588899
#> 3 Ursus americanus -99.1 23.7 2015-02-20 23:00:00 1098894889
#> 4 Ursus americanus -72.8 43.9 2015-02-13 16:16:41 1065611122
#> 5 Ursus americanus -72.3 43.9 2015-03-01 20:20:45 1088908315
#> 6 Ursus americanus -109. 32.7 2015-03-29 17:06:54 1088932238
#> 7 Ursus americanus -109. 32.7 2015-03-29 17:12:50 1088932273
#> 8 Ursus americanus -124. 40.1 2015-03-28 23:00:00 1132403409
#> 9 Ursus americanus -78.3 36.9 2015-03-20 21:11:24 1088923534
#> 10 Ursus americanus -76.8 35.5 2015-04-05 23:00:00 1088954559
#> # … with 1,478 more rowsDo all three
dframe(sample_data_1) %>%
coord_impossible() %>%
coord_incomplete() %>%
coord_unlikely()
#> # A tibble: 1,294 x 5
#> name longitude latitude date key
#> * <chr> <dbl> <dbl> <dttm> <int>
#> 1 Ursus americanus -79.7 38.4 2015-01-14 16:36:45 1065590124
#> 2 Ursus americanus -82.4 35.7 2015-01-13 00:25:39 1065588899
#> 3 Ursus americanus -99.1 23.7 2015-02-20 23:00:00 1098894889
#> 4 Ursus americanus -72.8 43.9 2015-02-13 16:16:41 1065611122
#> 5 Ursus americanus -72.3 43.9 2015-03-01 20:20:45 1088908315
#> 6 Ursus americanus -109. 32.7 2015-03-29 17:06:54 1088932238
#> 7 Ursus americanus -109. 32.7 2015-03-29 17:12:50 1088932273
#> 8 Ursus americanus -124. 40.1 2015-03-28 23:00:00 1132403409
#> 9 Ursus americanus -78.3 36.9 2015-03-20 21:11:24 1088923534
#> 10 Ursus americanus -76.8 35.5 2015-04-05 23:00:00 1088954559
#> # … with 1,284 more rowsDon’t drop bad data
dframe(sample_data_1) %>% coord_incomplete(drop = TRUE) %>% NROW
#> [1] 1306
dframe(sample_data_1) %>% coord_incomplete(drop = FALSE) %>% NROW
#> [1] 1500smalldf <- sample_data_1[1:20, ]
# create a duplicate record
smalldf <- rbind(smalldf, smalldf[10,])
row.names(smalldf) <- NULL
# make it slightly different
smalldf[21, "key"] <- 1088954555
NROW(smalldf)
#> [1] 21
dp <- dframe(smalldf) %>% dedup()
NROW(dp)
#> [1] 20
attr(dp, "dups")
#> # A tibble: 1 x 5
#> name longitude latitude date key
#> <chr> <dbl> <dbl> <dttm> <dbl>
#> 1 Ursus americanus -76.8 35.5 2015-04-05 23:00:00 1088954555Standardize/convert dates
# date_standardize(dframe(df), "%d%b%Y") # w/o pipe
dframe(sample_data_1) %>% date_standardize("%d%b%Y")
#> # A tibble: 1,500 x 5
#> name longitude latitude date key
#> <chr> <dbl> <dbl> <chr> <int>
#> 1 Ursus americanus -79.7 38.4 14Jan2015 1065590124
#> 2 Ursus americanus -82.4 35.7 13Jan2015 1065588899
#> 3 Ursus americanus -99.1 23.7 20Feb2015 1098894889
#> 4 Ursus americanus -72.8 43.9 13Feb2015 1065611122
#> 5 Ursus americanus -72.3 43.9 01Mar2015 1088908315
#> 6 Ursus americanus -109. 32.7 29Mar2015 1088932238
#> 7 Ursus americanus -109. 32.7 29Mar2015 1088932273
#> 8 Ursus americanus -124. 40.1 28Mar2015 1132403409
#> 9 Ursus americanus -78.3 36.9 20Mar2015 1088923534
#> 10 Ursus americanus -76.8 35.5 05Apr2015 1088954559
#> # … with 1,490 more rowsDrop records without dates
NROW(sample_data_1)
#> [1] 1500
NROW(dframe(sample_data_1) %>% date_missing())
#> [1] 1498Create date field from other fields
dframe(sample_data_2) %>% date_create(year, month, day)
#> # A tibble: 1,500 x 8
#> name longitude latitude key year month day date
#> <chr> <dbl> <dbl> <int> <chr> <chr> <chr> <chr>
#> 1 Ursus americanus -79.7 38.4 1065590124 2015 01 14 2015-01-14
#> 2 Ursus americanus -82.4 35.7 1065588899 2015 01 13 2015-01-13
#> 3 Ursus americanus -99.1 23.7 1098894889 2015 02 20 2015-02-20
#> 4 Ursus americanus -72.8 43.9 1065611122 2015 02 13 2015-02-13
#> 5 Ursus americanus -72.3 43.9 1088908315 2015 03 01 2015-03-01
#> 6 Ursus americanus -109. 32.7 1088932238 2015 03 29 2015-03-29
#> 7 Ursus americanus -109. 32.7 1088932273 2015 03 29 2015-03-29
#> 8 Ursus americanus -124. 40.1 1132403409 2015 03 28 2015-03-28
#> 9 Ursus americanus -78.3 36.9 1088923534 2015 03 20 2015-03-20
#> 10 Ursus americanus -76.8 35.5 1088954559 2015 04 05 2015-04-05
#> # … with 1,490 more rowsOnly one function exists for taxonomy cleaning, it removes rows where taxonomic names are either missing an epithet, or are missing altogether (NA or NULL).
Get some data from GBIF, via rgbif
if (requireNamespace("rgbif", quietly = TRUE)) {
library("rgbif")
res <- occ_data(limit = 500)$data
} else {
res <- sample_data_3
}Clean names
NROW(res)
#> [1] 500
df <- dframe(res) %>% tax_no_epithet(name = "name")
NROW(df)
#> [1] 481
attr(df, "name_var")
#> [1] "name"
attr(df, "tax_no_epithet")
#> # A tibble: 19 x 107
#> key scientificName decimalLatitude decimalLongitude issues datasetKey
#> <chr> <chr> <dbl> <dbl> <chr> <chr>
#> 1 1637… Aves -34.5 136. "" 40c0f670-…
#> 2 1989… Psychidae 25.0 122. "txma… e0b8cb67-…
#> 3 2542… Agaricales 55.9 12.3 "cdro… 84d26682-…
#> 4 2542… Corticiaceae 56.5 9.84 "" 84d26682-…
#> 5 2542… Corticiaceae 55.1 10.5 "cdro… 84d26682-…
#> 6 2542… Fungi 56.5 9.84 "cdro… 84d26682-…
#> 7 2542… Agaricales 55.9 12.5 "cdro… 84d26682-…
#> 8 2542… Polyporales 55.9 12.3 "cdro… 84d26682-…
#> 9 2542… Trichiaceae 56.7 9.87 "cdro… 84d26682-…
#> 10 2542… Xylariales 56.2 10.6 "cdro… 84d26682-…
#> 11 2542… Corticiaceae 56.5 9.84 "cdro… 84d26682-…
#> 12 2542… Hymenochaetac… 56.5 9.84 "cdro… 84d26682-…
#> 13 2542… Polyporales 56.0 12.3 "cdro… 84d26682-…
#> 14 2542… Fungi 55.8 12.5 "cdro… 84d26682-…
#> 15 2542… Fungi 55.9 12.4 "cdro… 84d26682-…
#> 16 2542… Fungi 55.9 12.3 "cdro… 84d26682-…
#> 17 2542… Fungi 55.9 12.3 "cdro… 84d26682-…
#> 18 2542… Hyaloscyphace… 54.9 11.5 "cdro… 84d26682-…
#> 19 2542… Physarales 54.9 11.5 "cdro… 84d26682-…
#> # … with 101 more variables: publishingOrgKey <chr>, installationKey <chr>,
#> # publishingCountry <chr>, protocol <chr>, lastCrawled <chr>,
#> # lastParsed <chr>, crawlId <int>, basisOfRecord <chr>,
#> # individualCount <int>, taxonKey <int>, kingdomKey <int>, phylumKey <int>,
#> # classKey <int>, orderKey <int>, familyKey <int>, genusKey <int>,
#> # speciesKey <int>, acceptedTaxonKey <int>, acceptedScientificName <chr>,
#> # kingdom <chr>, phylum <chr>, order <chr>, family <chr>, genus <chr>,
#> # species <chr>, genericName <chr>, specificEpithet <chr>, taxonRank <chr>,
#> # taxonomicStatus <chr>, year <int>, month <int>, eventDate <chr>,
#> # modified <chr>, lastInterpreted <chr>, references <chr>, license <chr>,
#> # class <chr>, countryCode <chr>, recordedByIDs <list>,
#> # identifiedByIDs <list>, rightsHolder <chr>, identifier <chr>,
#> # nomenclaturalCode <chr>, dynamicProperties <chr>, language <chr>,
#> # collectionCode <chr>, gbifID <chr>, occurrenceID <chr>, type <chr>,
#> # taxonRemarks <chr>, preparations <chr>, recordedBy <chr>,
#> # catalogNumber <chr>, vernacularName <chr>, institutionCode <chr>,
#> # previousIdentifications <chr>, ownerInstitutionCode <chr>,
#> # occurrenceRemarks <chr>, bibliographicCitation <chr>, accessRights <chr>,
#> # higherClassification <chr>, dateIdentified <chr>, elevation <dbl>,
#> # elevationAccuracy <dbl>, stateProvince <chr>, day <int>,
#> # geodeticDatum <chr>, country <chr>, recordNumber <chr>, municipality <chr>,
#> # locality <chr>, datasetName <chr>, identifiedBy <chr>, eventID <chr>,
#> # occurrenceStatus <chr>, locationRemarks <chr>, dataGeneralizations <chr>,
#> # taxonConceptID <chr>, coordinateUncertaintyInMeters <dbl>, lifeStage <chr>,
#> # infraspecificEpithet <chr>, associatedReferences <chr>, county <chr>,
#> # verbatimElevation <chr>, fieldNumber <chr>, continent <chr>,
#> # identificationVerificationStatus <chr>, taxonID <chr>, eventTime <chr>,
#> # behavior <chr>, informationWithheld <chr>, endDayOfYear <chr>,
#> # originalNameUsage <chr>, startDayOfYear <chr>, datasetID <chr>,
#> # habitat <chr>, associatedTaxa <chr>, locationAccordingTo <chr>,
#> # locationID <chr>, verbatimLocality <chr>, …