The neonUtilities R package provides utilities for discovering, downloading, and working with NEON data files. NEON data files can be downloaded from the NEON Data Portal (http://data.neonscience.org) or API (http://data.neonscience.org/data-api). Provisional NEON data files from instrumented and observation systems are delivered by NEON within zip files organized by site and year-month. Provisional NEON data files from the airborne observation platform (AOP) are organized by site and year.
neonUtilities is available on CRAN and most users will want to install it from there. If you want to use the current development version, you can install from GitHub, but be warned that the version here may not be stable.
A cheat sheet for the neonUtilities package is available here.
See NEON Data Tutorials for more information, particularly the Get Started tutorial series.
This package is actively maintained - please post any issues here and tag @chrlaney and/or @cklunch.
To get citation details for citing the neonUtilities package in a publication, run citation("neonUtilities") in R.
Install the package into your local environment from CRAN using the following code:
install.packages('neonUtilities')
library(neonUtilities)
neonUtilities is available on conda-forge.
You can setup an environment that includes all dependencies (including R) with the following command line operations.
Pixi environments are fully reproducible by default.
From your project directory, initialize a Pixi workspace and then add neonUtilities
pixi init
pixi add r-neonutilities
You can optionally activate an interactive shell with the environment loaded with
pixi shell
Create a conda environment and install neonUtilities from the conda-forge channel
conda create --name neon-env
conda config --name neon-env --add channels conda-forge
conda config --name neon-env --remove channels defaults
conda install --name neon-env r-neonutilities
and then activate the conda environment
conda activate neon-env
zipsByProduct() Download NEON observational (OS) and instrumentation (IS) data by data product.byFileAOP() Download NEON remote sensing data by site, year, and data product.byTileAOP() Download NEON remote sensing data by data product, year, and UTM coordinates.stackByTable() NEON OS and IS data are provided in separate files for each site and month of collection (and location within site, for IS). This function merges these data files cleanly for each data type. Works with data downloaded either via the NEON Data Portal or zipsByProduct().readTableNEON() Read NEON tabular data into R, detecting variable types via the variables file. Currently works for OS and IS data, but not eddy covariance data.loadByProduct() Combines the actions of zipsByProduct(), stackByTable(), and readTableNEON() for OS and IS data. In one step, downloads, merges, and loads data into R.stackEddy() Extracts NEON eddy covariance data from the provided HDF files and merges the resulting tabular data, similar to the IS data handling in stackByTable().footRaster() Makes a raster of the flux footprint data provided in the expanded data package for eddy covaraince.stackByTable() unzips monthly packages, finds the CSV data files, and joins them by table (e.g., 2DWSD_2min, 2DWSD_30min for 2D Wind Speed and Direction). For data products from instrumented systems that have multiple sensors placed at various heights (or depths) and/or horizontal positions away from the supporting tower, this function will create 2 columns in addition to the existing columns, one for horizontalPosition and the other for verticalPosition. This function will only work for data products that organize data in CSV files. Other data file types, such as HDF5 files from the eddy covariance system and remote sensing airborne observing platform (AOP) are not supported.
stackByTable(filepath = "testdata/NEON_size-dust-particulate.zip") # modify filepath to your directory
To load data directly into the current R environment, instead of saving the stacked files to the filepath, use the option savepath='envt'. When using this option, assign the output of the function to a variable name. The output object will be a named list of data tables.
dust <- stackByTable(filepath = "testdata/NEON_size-dust-particulate.zip", savepath="envt")
zipsByProduct() pulls data from the NEON API in the correct format to be stacked by stackByTable(). Depending on the data product and data volume, pulling data from the API with zipsByProduct() can take a very long time.
# Herbaceous clip harvest data, from all sites and months for which it is currently available
zipsByProduct(dpID="DP1.10023.001", site="all", package="basic", check.size=T)
stackByTable(paste0(getwd(), "/filesToStack10023"))
loadByProduct() performs the actions of both zipsByProduct() and stackByTable() and loads the resulting data into the current R environment. The object output by loadByProduct() is a named list of tables.
bird <- loadByProduct(dpID="DP1.10003.001", site="all", package="expanded")
names(bird)
# To get each table in the list as an independent object, outside of the list:
list2env(bird, .GlobalEnv)
Both zipsByProduct() and loadByProduct() can also subset by sites and date range:
wq <- loadByProduct(dpID="DP1.20288.001", site=c("ARIK","POSE"), startdate="2018-04", enddate="2018-08")
byFileAOP() pulls data from the NEON API, specifically for remote sensing (AOP) data. This function preserves the file directory hierarchy that AOP files are typically stored in, making it easier to navigate a large number of downloaded files.
# Lidar slant rangeform data from Santa Rita Experimental Range, 2017 flight
byFileAOP(dpID = "DP3.30001.001", site = "SRER", year = "2017", check.size = T)
byTileAOP() pulls AOP data from the NEON API, for tiles matching the coordinates specified in the function call. A buffer can also be included in the function call, to download tiles within a certain distance of the coordinates (e.g., if the coordinates are the plot centroids of NEON TOS plots, use buffer=20). byTileAOP() will only work on the Level 3, mosaicked AOP products.
# Vegetation indices from San Joaquin Experimental Range, 2017
# easting and northing must be matched vectors of UTM coordinates
byTileAOP(dpID="DP3.30026.001", site="SJER", year="2017", easting=easting, northing=northing, buffer=20)
transformFileToGeoCSV() takes any single NEON csv data file plus its respective variables file, and generates a new CSV with GeoCSV headers. This makes the data similar in format to data provided by organizations such as UNAVCO, and is good for embedding in a repeating script.
For a tutorial explaining how to use the neonUtilities package in more detail, including additional input options, view the Use the neonUtilities Package to Access NEON Data tutorial.
options(timeout=300). The timeout value is in seconds, and defaults to 60.The National Ecological Observatory Network is a project solely funded by the National Science Foundation and managed under cooperative agreement by Battelle. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
Information and documents contained within this repository are available as-is. Codes or documents, or their use, may not be supported or maintained under any program or service and may not be compatible with data currently available from the NEON Data Portal.
Updates:
Bug fix:
Updates:
Updates:
Bug fixes:
Bug fixes:
Enhancements:
Bug fixes:
Enhancements:
Bug fixes:
Changes:
Enhancements:
Other changes:
Enhancements:
byFileAOP() and byTileAOP() removedgetTaxonTable() is deprecated; moved to neonOS package as getTaxonList()
Bug fixes:
footRaster() now allows a vector of .h5 files as input; handling updated to match stackEddy()
byFileAOP() and byTileAOP() now correctly handle the rare occurrence of multiple months of data per yearEnhancements:
getNeonDOI() function added; finds DOI for a given data product and releasezipsByURI() can now handle R object inputsBug fixes:
Enhancements:
stackByTable(), stackEddy(), byTileAOP(), and byFileAOP() include issue log in outputsgetIssueLog() is also available as an independent functionBug fixes:
Bug fixes:
stackByTable() was failing on new data products; fixed.Enhancements:
stackEddy() variable inputs; see function documentationBug fixes:
readr package dependency removed to resolve incompatibility with readr 2.0.0footRaster() orientation correction in 2.1.0 did not work; corrected againstackEddy() time stamp handling improved for profile sensorsEnhancements:
zipsByProduct() and loadByProduct()
stackByTable()
stackFromStore() updated to work with neonstore v0.4.3Bug fixes:
stackEddy() handles scenarios with only one instance of a given variablefootRaster() coordinate conversion for Alaska and Hawaii fixedfootRaster() orientation correctedloadByProduct()
stackByTable() when file paths are longer than 260 characters in WindowsBug fixes:
stackByTable() resolved in portal data downloadsstackByTable() resolvedstackEddy()
Major version update, corresponding to the first Release of static, DOI-citable NEON data. Older versions of neonUtilities may not work correctly with Released data. For more information, see Releases web page.
Major changes:
stackByTable() and stackEddy() updated to work with new zip folder structurezipsByProduct() updated to use packages API endpoint instead of pre-packaged zip filesEnhancements:
stackByTable() adds release tag to stacked data when possiblegetDatatable() is deprecated and functionality is moved to loadByProduct(). Note download by table is not recommended for new users; familiarity with the data is a prerequisite.Bug fixes:
stackFromStore() now includes the full range of input options for sensor and SAE datastackByTable() skips them instead of failing. Note this is rare, the result of error in NEON publication systems.byTileAOP() updated to use latest versions of spatial packagesBug fixes:
stackEddy()
stackFromStore()
Bug fixes:
stackEddy()
footRaster() update to work with latest rgdalEnhancements:
stackFromStore() function to stack files from a local archivestackEddy() accepts a vector of filepaths to .h5 filesBug fixes:
Enhancements:
avg to zipsByProduct() changed to timeIndex for clarityBug fixes:
Enhancements:
Bug fixes:
loadByProduct() automatic deletion of temporary files restoredzipsByProduct() pauses to resetBug fixes:
byFileAOP() and byTileAOP() escape from infinite loops if availability is in errorstackEddy() works on expanded package data againbyTileAOP() prevents conversion of coordinates to unreadable formatsbyTileAOP() correctly handles sites that cross UTM zonesbyFileAOP() and byTileAOP() handle download for sites that are included in other sites' flight boxesloadByProduct() handles missing sensor_positions and readme filesreadTableNEON()
Enhancements:
footRaster() scales raster of flux footprint to geographical coordinateszipsByProduct() regenerates URLs if they expireBug fixes:
avg input in zipsByProduct()
stackByTable() now appear in variables fileEnhancements:
Enhancements:
stackByTable() optionally uses parallel processingstackByTable() preserves readme and sensor_positions filesstackByTable() appends a new column containing the publication time stampstackByTable() no longer requires folder inputloadByProduct() detects data types (numeric, character, date) based on variables fileBug fixes:
rhdf5 from Imports to Suggests for smoother installationstackByTable() to UTF-8Enhancements:
stackEddy() to extract and merge data from flux data HDF5 files (DP4.00200.001)Bug fixes:
byFileAOP() and byTileAOP()
Enhancements:
zipsByURI() to download via URLs within dataBug fixes:
stackByTable() that deleted existing files when using savepath argumentgetDatatable() that created incorrect urlsEnhancements:
zipsByProduct()
Bug fixes:
stackByTable() that merged tables with overlapping nameszipsByProduct() avg= option that only worked for a subset of data productsstackByTable() that failed unzipping if savepath=filepathEnhancements:
getDatatable()
stackByTable() to load files into the R environmentzipsByProduct()
loadByProduct() to download and load files in one stepbyTileAOP() to download only AOP tiles corresponding to certain coordinatesstackByTable()
zipsByProduct() option to download only one averaging interval (e.g. only 30-minute files) for speedier download and stackingzipsByProduct() input option to specify file path to save todata.table fread() and rbind() functions, respectively. This speeds up the stacking process.byFileAOP() for downloading large amounts of AOP data using the API; no longer fails if download duration exceeds 24 hours.stackByTable() is used for remote sensing (AOP), eddy covariance, or digital hemipsheric photos.stackByTable()
This has two main bug fixes: