Public functions and types
MetopDatasets.brightness_temperature
— Methodbrightness_temperature(I::T, wavenumber::Real, default=T(NaN)) where T <: Real
Converting the IASI L1C spectrum from radiances to brightness temperature. Note that the wavenumber must in meters^-1
Example
julia> file_path = "test/testData/IASI_xxx_1C_M01_20240819103856Z_20240819104152Z_N_C_20240819112911Z"
julia> ds = MetopDataset(file_path);
julia> spectrum = ds["gs1cspect"][:,1,1,1]
julia> wavenumber = ds["spectra_wavenumber"][:, 1]
julia> # convert from radiances to brightness temperature
julia> T_B = brightness_temperature.(spectrum, wavenumber)
MetopDatasets.data_record_type
— Methoddata_record_type(header::MainProductHeader)::Type
Get the type of data record based on the main product header
Example
julia> file_pointer = open("ASCA_SZO_1B_M03_20230329063300Z_20230329063556Z_N_C_20230329081417Z")
julia> main_header = MetopDatasets.native_read(file_pointer, MainProductHeader)
julia> data_record_type(main_header)
ASCA_SZO_1B_V13
MetopDatasets.get_test_data_artifact
— Methodget_test_data_artifact()
Returns path to folder storing reduced test data. Note that the test data is downloaded from https://github.com/eumetsat/test-data-MetopDatasets the first time the function it called.
MetopDatasets.read_first_record
— Methodread_first_record(source, record_type)::record_type
A simple alias for read_single_record(source, record_type, 1)
MetopDatasets.read_single_record
— Methodread_single_record(ds::MetopDataset, record_type::Type{<:Record})
read_single_record(file_pointer::IO, record_type::Type{<:Record})
read_single_record(file_path::AbstractString, record_type::Type{<:Record})
Read the n'th record of type record_type
from the dataset. This can be used to access records that are not directly exposed through the MetopDataset
interface.
MetopDatasets.record_struct_expression
— Methodrecord_struct_expression(file_name, record_type)
Function to autogenerate Struct
code based on a CSV file. Also autogenerates get_description
and get_scale_factor
method for Struct
. Use it together with eval
.
Example
julia> eval(record_struct_expression(joinpath(@__DIR__, "TEST_FORmaT.csv"), DataRecord))
julia> TEST_FORMAT <: DataRecord
true
MetopDatasets.scale_iasi_spectrum
— Methodscale_iasi_spectrum(spec_raw, giadr::GIADR_IASI_XXX_1C; high_precision = false)
scale_iasi_spectrum(spec_raw, giadr::GIADR_IASI_XXX_1C, channel_range::OrdinalRange; high_precision = false)
Scaling the IASI L1C spectrum using the giadr record information. The channel_range
is needed if only a subset of the raw spectrum is passed to the function. Setting high_precision=true
will convert to Float64
instead of Float32
. Note that the end part of the ds["gs1cspect"]
does not have any scale factors. Here the spectrum is just filled with 0.0
.
Example
julia> file_path = "test/testData/IASI_xxx_1C_M01_20240819103856Z_20240819104152Z_N_C_20240819112911Z"
julia> ds = MetopDataset(file_path, auto_convert = false);
julia> giadr = MetopDatasets.read_first_record(ds, MetopDatasets.GIADR_IASI_XXX_1C_V11)
julia> # Scale full spectrum.
julia> scaled_spectrum = MetopDatasets.scale_iasi_spectrum(ds["gs1cspect"], giadr)
julia> # Scale subset of spectrum.
julia> scaled_spectrum_subset = MetopDatasets.scale_iasi_spectrum(ds["gs1cspect"][10:20,:,:,:], giadr, 10:20)
MetopDatasets.AbstractMetopDiskArray
— TypeAbstractMetopDiskArray{T, N} <: DiskArrays.AbstractDiskArray{T, N}
In most cases MetopDiskArray
is used but AbstractMetopDiskArray
allows defining additional DiskArray types to handle special corner cases.
MetopDatasets.FixedRecordLayout
— TypeFixedRecordLayout
Used to store the record layout of fixed size data records in a Native metop files.
MetopDatasets.MetopDataset
— MethodMetopDataset(file_path::AbstractString; auto_convert::Bool = true, high_precision::Bool=false, maskingvalue = missing)
MetopDataset(file_pointer::IO; auto_convert::Bool = true, high_precision::Bool=false, maskingvalue = missing)
MetopDataset(f::Function, file_path::AbstractString; auto_convert::Bool = true, high_precision::Bool=false, maskingvalue = missing)
Load a MetopDataset from a Metop Native binary file or from a IO
to a Native binary file. Only the meta data is loaded upon creation and all variables are lazy loaded. The variables corresponds to the different fields of the data records in the file. The attributes have all the information from the main product header in the file.
auto_convert=true
will automatically convert MetopDatasets
specific types such as VInteger
to common netCDF complaint types such as Float64
. This will also automatically scale variable where the scaling can't be expressed through a simple scale factor e.g. the IASI spectrum where different bands of the spectrum have different scaling factors.
Selected fields are converted to Float32
to save memory. Normally Float32
is more than sufficient to represent the instrument accuracy. Setting high_precision=true
will in some case convert these variables to Float64
.
maskingvalue = NaN
will replace missing
values with NaN. This normally floats but can create issues for integers. See documentation page for more information.
Example
julia> file_path = "test/testData/ASCA_SZR_1B_M03_20230329063300Z_20230329063558Z_N_C_20230329081417Z"
julia> ds = MetopDataset(file_path);
julia>
julia> # display metadata of a variable
julia> ds["latitude"]
latitude (82 × 96)
Datatype: Union{Missing, Float64} (Int32)
Dimensions: xtrack × atrack
Attributes:
description = Latitude (-90 to 90 deg)
missing_value = Int32[-2147483648]
scale_factor = 1.0e-6
julia>
julia> # load a subset of a variable
julia> lat_subset = ds["latitude"][1:2,1:3] # load a small subset of latitudes.
2×3 Matrix{Union{Missing,Float64}}:
-33.7308 -33.8399 -33.949
-33.7139 -33.823 -33.9322
julia>
julia> # load entire variable
julia> lat = ds["latitude"][:,:]
julia>
julia> # close data set
julia> close(ds);
MetopDatasets.MetopDiskArray
— TypeMetopDiskArray{T, N} <: AbstractMetopDiskArray{T, N}
Struct to handle lazy loading of a variable in a Metop product. The raw types in the product is mapped without any scaling. Auto conversion can be enabled for RecordSubType
e.g. converting VInteger
to Float64
.
MetopDatasets.MetopDiskArray
— MethodMetopDiskArray(file_pointer::IOStream,
record_layouts::Vector{FixedRecordLayout},
field_name::Symbol; auto_convert = true) -> MetopDiskArray
Constructor for MetopDiskArray that compute additional fields. auto_convert = true
will automatically convert custom RecordSubType
to commonly used data types e.g. converting VInteger
to Float64
.
MetopDatasets.MetopVariable
— TypeMetopVariable{T, N, R <: DataRecord} <: CommonDataModel.AbstractVariable{T, N}
MetopVariable
wraps an AbstractArray
so it can be used with MetopDataset
. The data array is normally AbstractMetopDiskArray
.
Internal functions and types
MetopDatasets._get_data_record_layouts
— Method_get_data_record_layouts(internal_pointer_records::Vector{InternalPointerRecord},
total_file_size::Integer, record_type::Type{<:DataRecord})::Vector{FixedRecordLayout}
Compute the record_layouts
MetopDatasets._get_flexible_dims_file
— Method_get_flexible_dims_file(file_pointer::IO, T::Type{<:BinaryRecord})
Read the flexible types from a product. Note that the IO
position is not changed by calling the function
MetopDatasets.construct_disk_array
— Methodconstruct_disk_array(file_pointer::IOStream,
record_layouts::Vector{<:RecordLayout},
field_name::Symbol; auto_convert = true)
Construct a disk array. The type of disk array is automatically determined. The standard type of disk array is MetopDiskArray
but there are also other types. FlexibleMetopDiskArray
is returned for fields where the size varies inside the product, eg. IASI L2 "temperature_error".
MetopDatasets.fixed_size
— Methodfixed_size(T::Type{<:BinaryRecord})::Bool
fixed_size(T::Type{<:BinaryRecord}, fieldname::Symbol)::Bool
Get if the data record has a binary fixed size. This is often used as a trait via the "julia Holy Traits Pattern" fieldname
is used to check if a specific field has a fixed binary size.
MetopDatasets.fixed_size_in_file
— Methodfixed_size_in_file(T::Type{<:BinaryRecord}, fieldname::Symbol)::Bool
Check if the field have a constant size in a product. Return false if the field size can vary within a single file.
MetopDatasets.get_description
— Methodget_description(T::Type{<:BinaryRecord}, field::Symbol)::AbstractString
Get the description for a given field in the BinaryRecord
Example
julia> get_description(ASCA_SZR_1B_V13, :sigma0_trip)
"Sigma0 triplet, re-sampled to swath grid, for 3 beams (fore, mid, aft) "
MetopDatasets.get_dimensions
— Methodget_dimensions(T::Type{<:BinaryRecord})::Dict{String, <:Integer}
Get the the named dimensions in a BinaryRecord and their length.
Example
julia> get_dimensions(ASCA_SZR_1B_V13)
Dict{String, Int64} with 2 entries:
"num_band" => 3
"xtrack" => 82
MetopDatasets.get_field_dimensions
— Methodget_field_dimensions(T::Type{<:BinaryRecord}, field::Symbol)::Vector{<:AbstractString}
Get the named dimensions of a field in a BinaryRecord
Example
julia> get_field_dimensions(ASCA_SZR_1B_V13, :sigma0_trip)
["num_band", "xtrack"]
MetopDatasets.get_flexible_dim_fields
— Methodget_flexible_dim_fields(T::Type{<:BinaryRecord})::Dict{Symbol,Symbol}
Get a dictionary with field names as key and the corresponding flexible dim as value. Only fields representing a flexible dim is included. Must be implemented for Records containing flexible dim values.
Example
julia> get_flexible_dim_fields(IASI_SND_02)
Dict{Symbol, Symbol} with 4 entries:
:co_nbr => :CO_NBR
:o3_nbr => :O3_NBR
:nerr => :NERR
:hno3_nbr => :HNO3_NBR
MetopDatasets.get_missing_value
— Methodget_missing_value(T::Type{<:Record}, field::Symbol)
Get the value representing missing
for the field. Default values are implemented for Integers
but they can be overwritten for specific record types to account for different conventions.
MetopDatasets.get_raw_format_dim
— Methodget_raw_format_dim(T::Type{<:BinaryRecord}, field::Symbol)
Get the dimensions of the field as defined in the record format specification.
Example
julia> get_raw_format_dim(ASCA_SZR_1B_V13, :sigma0_trip)
(3, 82, 1, 1)
MetopDatasets.get_scale_factor
— Methodget_scale_factor(T::Type{<:BinaryRecord}, field::Symbol)::Union{Number,Nothing}
get the scale_factor
for a given field in the BinaryRecord
. The variable can late be scaled from integer to float by dividing with 10^scale_factor
. Returns nothing
if no scale factor is set.
Example
julia> get_scale_factor(ASCA_SZR_1B_V13, :sigma0_trip)
6
MetopDatasets.layout_info_for_disk_array
— Methodlayout_info_for_disk_array(record_layouts::Vector{<:RecordLayout}, field_name::Symbol)
Extract information need in MetopDiskArray
MetopDatasets.native_read
— Methodnative_read(io::IO, T)::T
Read a single object of type T from io to a file in the native Metop format. Endianness is automatically converted.
Example
julia> file_pointer = open("ASCA_SZO_1B_M03_20230329063300Z_20230329063556Z_N_C_20230329081417Z")
julia> main_header = MetopDatasets.native_read(file_pointer, MainProductHeader)
julia> main_header.sensing_start
2023-03-29T06:33:00
MetopDatasets.native_sizeof
— Methodnative_sizeof(x)::Integer
The byte size of the type x in a METOP native product.
Example
julia> native_sizeof(RecordHeader)
20
MetopDatasets.read_record_layouts
— Methodread_record_layouts(file_pointer::IO, main_product_header::MainProductHeader)
Read the appropriate record layout from IO.
MetopDatasets.DummyRecord
— TypeDummyRecord <:Record
The Dummy Measurement Data Record is a special case of the MDR. It is a generic record that is used to indicate the location of lost data within any product. One DMDR can replace a contiguous block of lost MDRs
MetopDatasets.FlexibleMetopDiskArray
— TypeFlexibleMetopDiskArray{T, N} <: AbstractMetopDiskArray{T, N}
Similar to MetopDiskArray
but able to handle flexible record layout and fields where the size varies within a single product file. E.g. IASI L2 fields like "temperature_error".
MetopDatasets.FlexibleRecordLayout
— TypeFlexibleRecordLayout
Used to store the record layout of flexible size data records in a Native metop files.
MetopDatasets.IasiSpectrumDiskArray
— TypeIasiSpectrumDiskArray{T} <: AbstractMetopDiskArray{T, 4}
The IasiSpectrumDiskArray
is a wrapper around a MetopDiskArray
that enables the automatic scaling of the the IASI L1C spectrum using the GIADR_IASI_XXX_1C
record information.
MetopDatasets.IasiWaveNumberDiskArray
— TypeIasiWaveNumberDiskArray <: AbstractMetopDiskArray{Float64, 2}
The IasiWaveNumberDiskArray
is a disk array that computes the wavenumber of the IASI spectrum. The wavenumber is computed using :idefnsfirst1b
and :idefspectdwn1b
from each data record.
MetopDatasets.InternalPointerRecord
— TypeInternalPointerRecord <: Record
The Internal Pointer Records (IPR) specifies the start of each block of records in the file sharing the same record type. This can be used to find the locations of data records or dummy records.
MetopDatasets.LazyByteField
— TypeLazyByteField <: AbstractVector{Vector{UInt8}}
Type to read fields from records as a byte vector Note that LazyByteField does not work with conversion to netCDF.
MetopDatasets.RecordHeader
— TypeRecordHeader
Also known as GRH.