IDEA - data and software
Michael Sumner
raadtools, software to extract maps of ocean properties and values at points-in-time
raadtools > 10 years old R package
| R function | Purpose |
|---|---|
| readsst() | global sea surface temperature |
| readice() | polar sea ice concentrations |
| readghrsst() | high resolution sea surface temperature |
| read_adt/ugos/vgos_daily() | global altimetry, sea height, surface currents |
| read_chla_daily() | global ocean colour |
| readtopo() | global or local bathymetry |
| … |
other functions we don’t have, yet …
software we want
| features we want | raadtools 🤔 | <new tool> |
|---|---|---|
| users don’t download files | ✅ | ✅ |
| data is up to date | ✅ | ✅ |
| we can add new data | ✅ | ✅ |
| you can add new data | ❌ | ✅ |
| use outside AAD without Mike or Ben | ❌ | ✅ |
| use outside of R | ❌ | ✅ |
| scale up on super computing | ❌ | ✅ |
| robust to research/local outage | ❌ | ✅ |
| available offline on Nuyina | ❌ | ?? |
Python support
Address entire data cubes, with one line of code e.g. daily data 1993 to November 2024
import xarray; <some settings>
ds = xarray.open_dataset('s3://vzarr/SEALEVEL_GLO_PHY_L4.parquet', <more settings>)<xarray.Dataset> Size: 574GB
Dimensions: (time: 11538, latitude: 720, longitude: 1440)
Coordinates:
* latitude (latitude) float32 3kB -89.88 -89.62 -89.38 ... 89.38 89.62 89.88
* longitude (longitude) float32 6kB -179.9 -179.6 -179.4 ... 179.6 179.9
* time (time) datetime64[ns] 92kB 1993-01-01 1993-01-02 ... 2024-11-25
Data variables:
adt (time, latitude, longitude) float64 96GB dask.array<chunksize=(1, 50, 50), meta=np.ndarray>
sla (time, latitude, longitude) float64 96GB dask.array<chunksize=(1, 50, 50), meta=np.ndarray>
ugos (time, latitude, longitude) float64 96GB dask.array<chunksize=(1, 50, 50), meta=np.ndarray>
ugosa (time, latitude, longitude) float64 96GB dask.array<chunksize=(1, 50, 50), meta=np.ndarray>
vgos (time, latitude, longitude) float64 96GB dask.array<chunksize=(1, 50, 50), meta=np.ndarray>
vgosa (time, latitude, longitude) float64 96GB dask.array<chunksize=(1, 50, 50), meta=np.ndarray>
Attributes: (12/44) ...How are we doing this
- The old and new tools reflect user-demand, tell us your ideas!
- Modern tech: cloud-native and efficient public-available files
- Data curation and cataloguing tools: {bowerbird}, STAC, VirtualiZarr
- Exploring best-practice usage in Python xarray, odc, and in R terra, gdalraster, rsi
- Contributing to software libraries GDAL.org, and community with AADC, SCAR, rOpenSci, Pangeo, Radiant Earth, Opendatacube, Digital Earth Antarctica