NOAA NCDC dataset attributes

Scott Chamberlain

2020-07-07

The attributes, or “flags”, for each row of the output for data may have a flag with it. Each datasetid has it’s own set of flags. The following are flag columns, and what they stand are. fl_ is the beginning of each flag column name, then one or more characters to describe the flag, keeping it short to maintain a compact data frame. Some of these fields are the same across datasetids, but they may have different possible values. See below details on each dataset.

Datasets

# Dataset: GHCND

flm_m (Measurement flag)

fl_q (Quality flag)

fl_s (Source flag)

fl_t (Time of observation flag)

Is the (2 digit hour, 2 digit minute) 24 hour clock time of the observation given as the local time at the station of record.

# Dataset: GSOM

More info: https://www1.ncdc.noaa.gov/pub/data/cdo/documentation/gsom-gsoy.pdf

Observations are synonymous with elements or values, and defined in Table A below. 9999 indicates missing data or data that has not been received.

flags: Missing flag , Consecutive Missing flag

fl_miss (Missing flag)

Defined as total number of days observation/element is missing in that month. This can be taken as a measure of quality or completeness as the higher the number of days sampled in the month, the more representative the value is for the entire month.

fl_cmiss (Consecutive missing flag)

Defined as the maximum number of consecutive days in the month that an observation/element is missing.

# Dataset: GSOY

More info: https://www1.ncdc.noaa.gov/pub/data/cdo/documentation/gsom-gsoy.pdf

fl_m (Measurement flag)

fl_q (Quality flag)

fl_d (Number of days flag )

Number of days is given as 00 when all days in the month are considered in computing data value or otherwise the maximum number of consecutive days in the month considered in computing the data value.

fl_u (Units flag)

# Dataset: NORMAL_ANN

More info: http://www1.ncdc.noaa.gov/pub/data/cdo/documentation/NORMAL_ANN_documentation.pdf

Description

The 1981-2010 Normals comprise all climate normals using the thirty year period of temperature, degree days, precipitation, snowfall, snow depth, wind, etc. Data is organized into hourly, daily, monthly, seasonal and annual. This document describes the elements and layout of the Seasonal and Annual Normals which are derived from a composite of climate records from numerous sources that were merged and then subjected to a suite of quality assurance reviews.

fl_c (Completeness flag)

flags accompany every Normals value and indicate the completeness of the data record used to compute each value, accounting for methodological differences for different product classes. There are six flag options described generally below. Due to methodological differences, the flags are applied somewhat differently between the temperature-based normals and the precipitation-based normals. For the precipitation-based and hourly normals, the following flags were assigned independently for each normals value reported based on number of years available for that individual calculation. For temperature-based normals, strong precedence is given to the monthly normals of maximum and minimum temperature or derived from the flags for these two variables.

Note: flags Q and R aren’t applicable to average number of days with different precipitation, snowfall, and snow depth threshold exceedance; precipitation/snowfall/snow probabilities of occurrence. Further, Q flags are not applicable for standard deviations.

# Dataset: NORMAL_DLY

More info: http://www1.ncdc.noaa.gov/pub/data/cdo/documentation/NORMAL_DLY_documentation.pdf

Description

The 1981-2010 Normals comprise all climate normals using the thirty year period of temperature, degree days, precipitation, snowfall, snow depth, wind, etc. Data is organized into hourly, daily, monthly, seasonal and annual. This document describes the elements and layout of the Daily Normals which are derived from a composite of climate records from numerous sources that were merged and then subjected to a suite of quality assurance reviews.

fl_c (Completeness flag)

Same as NORMAL_ANN, see the description above at Completeness flag.

# Dataset: NORMAL_HLY

More info: http://www1.ncdc.noaa.gov/pub/data/cdo/documentation/NORMAL_HLY_documentation.pdf

Description

The 1981-2010 Normals comprise all climate normals using the thirty year period of temperature, degree days, precipitation, snowfall, snow depth, wind, etc. Data is organized into hourly, daily, monthly, seasonal and annual normals. This document describes the elements and layout of the Hourly Normals which are derived from a composite of climate records from numerous sources that were merged and then subjected to a suite of quality assurance reviews. The hourly normals provide a suite of descriptive statistics based on hourly observations at a few hundred stations from across the United States and its Pacific territories. Statistics are provided as 30- year averages, frequencies of occurrence, and percentiles for each hour and day of the year. These products are useful in examination of the diurnal change of a particular variable.

For temperature, dew point and mean sea level pressure an average hourly value as well as a 10th and 90th percentile of hourly values is given. For heating and cooling degree hours, an average hourly value is given using a 65 degree F base. Average hourly values are also given for heat index and wind chill. Cloud cover statistics include percent frequency of clear, few, scattered, broken and overcast conditions. Wind statistics include prevailing and secondary wind direction and percent frequency, average wind speed, percentage of calm winds and mean wind vector direction and magnitude.

The statistics are computed from the ISD-lite dataset. 262 stations were selected from the ISD-lite data, based on their completeness and membership in a list of what were known as “first order stations.” These are typically airport locations with the needed 24 hours/day observations to make hourly normals meaningful. All stations had at least 27 of the 30 years represented.

Each hourly normal is computed on the basis of 450 possible values. This is the aggregation of the value for a particular date and time, plus and minus 7 days, over each of 30 years. If fewer than 350 valid values are present, the output is given as the special value 9999. No normals are computed for February 29, but data for February 29 is included in the 15 day window for leap years. The original data has been shifted from Greenwich Mean Time to an end product in local standard time.

fl_c (Completeness flag)

Same as NORMAL_ANN, see the description above at Completeness flag.

# Dataset: NORMAL_MLY

More info: http://www1.ncdc.noaa.gov/pub/data/cdo/documentation/NORMAL_MLY_documentation.pdf

Description

The 1981-2010 Normals comprise all climate normals using the thirty year period of temperature, degree days, precipitation, snowfall, snow depth, wind, etc. Data are organized into hourly, daily, monthly, seasonal and annual. This document describes the elements and layout of the Monthly Normals which are derived from a composite of climate records from numerous sources that were merged and then subjected to a suite of quality assurance reviews.

fl_c (Completeness flag)

Same as NORMAL_ANN, see the description above at Completeness flag.

# Dataset: PRECIP_HLY

More info: http://www1.ncdc.noaa.gov/pub/data/cdo/documentation/PRECIP_HLY_documentation.pdf

Description

Hourly Precipitation Data (labeled Precipitation Hourly in Climate Data Online system) is a database that gives time-sequenced hourly precipitation amounts for a network of over 7000 reporting station located primarily in the United States. Data is collected from a variety of sources including National Weather Service reporting stations, volunteer cooperative observers, Federal Aviation Administration (FAA), utility companies, etc.

Concerning rain gages/data processing: Data from weighing rain gages, Fischer-Porter gages, Universal rain gages and in recent years, more modern measuring equipment in conjunction with automated recording sites, etc. have been used in this dataset over the period. Precipitation values have been checked and edited as necessary by both automated and manual methods. Because of some inconsistencies identified with the earlier data (prior to 1996), historical data were reprocessed in 1997. This rehabilitated data covered 53 million observations between 1900 and 1995. Similar quality control checks are in place that maintain consistency between the historical and operationally received data.

fl_m (Measurement flag)

Note: This field is left blank when no flag is needed.

fl_q (Data quality flag)

# Dataset: PRECIP_15

More info: http://www1.ncdc.noaa.gov/pub/data/cdo/documentation/PRECIP_15_documentation.pdf

Description

15 Minute Precipitation Data (labeled Precipitation 15 Minute in Climate Data Online system) is a database that gives time-sequenced quarter-hour precipitation amounts for a network of over 3600 reporting station located primarily in the United States. Data is collected from a variety of sources including National Weather Service reporting stations, volunteer cooperative observers, Federal Aviation Administration (FAA), utility companies, etc.

Concerning rain gages/data processing: Data from Fischer-Porter gages between May 1971 and December 1983 have been used in this dataset. Precipitation values have been checked and edited as necessary by both automated and manual methods. Data processing procedures were updated in January 1984 to produce the element structured data base files and further enhanced beginning with the January 1996 data month. Currently, interactive quality control procedures are in place that has added many checks and features and data are subjected to automated editing procedures that reduce the manual handling of the data.

fl_m (Data measurement flag)

QPCP

Note: This field is left blank when no flag is needed.

QGAG

fl_q (Data quality flag)

QPCP

QGAG

fl_u (Units flag)

HI indicates data values (QGAG or QPCP) are in hundredths of inches. HT indicates data values (QGAG or QPCP) are in tenths of inches.