Daily Time Series Data

Daily time series data summarizes biological activity across days for each radar station. It is created by aggregating the 5-minute time series data to a daily time step.

Data Organization

The daily time series data has one record per radar station and day for three different periods: day, night, and utc_calendar_day. There is one file per year with data for all stations, such as:

daily/2017-daily.csv

The data looks like this:

Source: 2017-daily.csv

station date period period_length fraction_missing fraction_filled reflectivity_hours reflectivity_hours_unfiltered traffic traffic_unfiltered u v direction speed fraction_rain
KABR 2017-01-01 day 8.79761 0.0 0.0 53.7827 944.598 4270.73 43789.2 -11.6753 -2.4901 239.175 22.0559 0.209149
KABX 2017-01-01 day 9.83388 0.0 0.0 55.1615 5690.09 1464.45 177922.0 3.85558 4.83858 37.4669 7.39079 0.315083
KAKQ 2017-01-01 day 9.68391 0.0 0.0 240.798 3286.11 3252.33 175224.0 2.51224 1.61722 101.404 3.7591 0.178603
KAMA 2017-01-01 day 9.8393 0.0 0.0 444.095 499.481 21449.0 24063.7 9.18912 5.42586 79.406 13.4293 0.134888
KAMX 2017-01-01 day 10.5793 0.0 0.0 1297.3 1640.3 22395.8 28466.9 -3.76757 2.29601 299.236 4.79677 0.164509
KAPX 2017-01-01 day 8.86334 0.0 0.0 23.0456 27.8396 2621.75 2999.3 26.4911 12.5218 60.8039 31.5473 0.0820028
... ... ... ... ... ... ... ... ... ... ... ... ... ... ...

Schema: Daily Time Series

Source: daily.json

A daily time series of biological measurements for an individual radar station. The measurements are obtained by aggregating the 5-minute time series to a daily time step for different periods (night, day, and calendar date). Missing values may occur if a variable is missing in the 5-minute time series for all records in the period.

Name Description Type Unit
station *

The 4-character station ID. See station metadata.

  • required: true
  • pattern: [A-Z]{4}
 string
date *

UTC date of the record.

  • required: true
 date
period *

The period of the day (night, day, or utc_calendar_day) for which records of the 5-minute time series were aggregated to produce this record.

  • required: true
 string
period_length *

Length of the period. See the period field.

  • required: true
  • minimum: 0.0
 number h
fraction_missing

Fraction of 5-minute records in this period that were missing when aggregating.

  • minimum: 0.0
  • maximum: 1.0
 number
fraction_filled

Fraction of 5-minute records in this period that were filled in the 5-minute time series before aggregating.

  • minimum: 0.0
  • maximum: 1.0
 number
reflectivity_hours

Temporally and vertically integrated reflectivity. Represents accumulated hours of radar cross section in cm2 from a vertical column above one square kilometer of the earth’s surface. Only includes scattering volumes identified as biology.

  • minimum: 0.0
 number cm2 km-2 h
reflectivity_hours_unfiltered

Same as reflectivity_hours, but integrates reflectivity_unfiltered over time instead of reflectivity, so it includes measurements from all scattering volumes, including those identified as rain.

  • minimum: 0.0
 number cm2 km-2 h
traffic

Reflectivity traffic, computed by integrating traffic_rate over time. Represents total radar cross section in cm2 crossing over a one kilometer transect during the time period. Only includes scattering volumes identified as biology.

  • minimum: 0.0
 number cm2 km-1
traffic_unfiltered

Same as traffic, but integrates traffic_rate_unfiltered over time instead of traffic_rate, so it includes measurements from all scattering volumes, including those identified as rain.

  • minimum: 0.0
 number cm2 km-1
u

Zonal (east-west) velocity component, computed as reflectivity-weighted average over the time period.

 number m s-1
v

Meridional (north-south) velocity component, computed as reflectivity-weighted average over the time period.

 number m s-1
direction

Mean direction of travel, computed as reflectivity-weighted average over the time period. The angle is given as a compass bearing in degrees clockwise from north.

  • minimum: 0.0
  • maximum: 360.0
 number degree
speed

Mean (ground) speed of travel, computed as reflectivity-weighted average over the time period.

  • minimum: 0.0
 number m s-1
fraction_rain

Fraction of scattering volumes classified as precipitation, computed as a simple average over the time period.

  • minimum: 0.0
  • maximum: 1.0
 number

Combining and Unstacking Data

Like other time-series data, the daily time-series data are in a stacked format with one row per timestamp and station, so it is easy to combine data from different files by concatenating the rows.

For daily time-series data, timestamps are shared across stations. Analysts may want to pivot the data to an unstacked format where columns correspond to the same variable across different stations. In Python this can be done as follows:

import pandas as pd
df = pd.read_csv('daily/2017-daily.csv')
df = df[df['period']=='night']
df = df.pivot(
    index=["date"],
    columns="station",
    values="reflectivity_hours"
)

This gives:

Source: 2017-daily-unstacked.csv

date KABR KABX KAKQ KAMA KAMX ...
2017-01-01 18.4784 17.586 182.679 188.135 82.3071 ...
2017-01-02 63.5675 15.3236 7.40937 196.845 73.7455 ...
... ... ... ... ... ... ...