brightness_temperature(I::T, wavenumber::Real, default=T(NaN)) where T <: Real

Converting the IASI L1C spectrum from radiances to brightness temperature. Note that the wavenumber must in m^-1 and I in W/m2/sr/m-1

Example

julia> file_path = "test/testData/IASI_xxx_1C_M01_20240819103856Z_20240819104152Z_N_C_20240819112911Z"
julia> ds = MetopDataset(file_path);
julia> spectrum = ds["gs1cspect"][:,1,1,1]
julia> wavenumber = ds["spectra_wavenumber"][:, 1]
julia> # convert from radiances to brightness temperature
julia> T_B = brightness_temperature.(spectrum, wavenumber)

data_record_type(header::MainProductHeader)::Type

Get the type of data record based on the main product header

Example

julia> file_pointer = open("ASCA_SZO_1B_M03_20230329063300Z_20230329063556Z_N_C_20230329081417Z")
julia> main_header = MetopDatasets.native_read(file_pointer, MainProductHeader)
julia> data_record_type(main_header)
ASCA_SZO_1B_V13

get_scaled(record::T, field::Union{AbstractString,Symbol}) where T <: BinaryRecord

Get the property from a data record type and apply scale factor if defined.

get_test_data_artifact()

Returns path to folder storing reduced test data. Note that the test data is downloaded from https://github.com/eumetsat/test-data-MetopDatasets the first time the function it called.

read_first_record(source, record_type)::record_type

A simple alias for read_single_record(source, record_type, 1)

read_single_record(ds::MetopDataset, record_type::Type{<:Record})
read_single_record(file_pointer::IO, record_type::Type{<:Record})
read_single_record(file_path::AbstractString, record_type::Type{<:Record})

Read the n'th record of type record_type from the dataset. This can be used to access records that are not directly exposed through the MetopDataset interface.

record_struct_expression(file_name, record_type)

Function to autogenerate Struct code based on a CSV file. Also autogenerates get_description and get_scale_factor method for Struct. Use it together with eval.

Example

julia> eval(record_struct_expression(joinpath(@__DIR__, "TEST_FORmaT.csv"), DataRecord))
julia> TEST_FORMAT <: DataRecord
true

scale_iasi_spectrum(spec_raw, giadr::GIADR_IASI_XXX_1C; high_precision = false)
scale_iasi_spectrum(spec_raw, giadr::GIADR_IASI_XXX_1C, channel_range::OrdinalRange; high_precision = false)

Scaling the IASI L1C spectrum using the giadr record information. The channel_range is needed if only a subset of the raw spectrum is passed to the function. Setting high_precision=true will convert to Float64 instead of Float32. Note that the end part of the ds["gs1cspect"] does not have any scale factors. Here the spectrum is just filled with 0.0.

Example

julia> file_path = "test/testData/IASI_xxx_1C_M01_20240819103856Z_20240819104152Z_N_C_20240819112911Z"
julia> ds = MetopDataset(file_path, auto_convert = false);
julia> giadr = MetopDatasets.read_first_record(ds, MetopDatasets.GIADR_IASI_XXX_1C_V11)
julia> # Scale full spectrum.
julia> scaled_spectrum = MetopDatasets.scale_iasi_spectrum(ds["gs1cspect"], giadr)
julia> # Scale subset of spectrum.
julia> scaled_spectrum_subset = MetopDatasets.scale_iasi_spectrum(ds["gs1cspect"][10:20,:,:,:], giadr, 10:20)

AbstractMetopDiskArray{T, N} <: DiskArrays.AbstractDiskArray{T, N}

In most cases MetopDiskArray is used but AbstractMetopDiskArray allows defining additional DiskArray types to handle special corner cases.

FixedRecordLayout

Used to store the record layout of fixed size data records in a Native metop files.

MetopDataset(file_path::AbstractString; auto_convert::Bool = true, high_precision::Bool=false, maskingvalue = missing)
MetopDataset(file_pointer::IO; auto_convert::Bool = true, high_precision::Bool=false, maskingvalue = missing)
MetopDataset(f::Function, file_path::AbstractString; auto_convert::Bool = true, high_precision::Bool=false, maskingvalue = missing)

Load a MetopDataset from a Metop Native binary file or from a IO to a Native binary file. Only the meta data is loaded upon creation and all variables are lazy loaded. The variables corresponds to the different fields of the data records in the file. The attributes have all the information from the main product header in the file.

auto_convert=true will automatically convert MetopDatasets specific types such as VInteger to common netCDF complaint types such as Float64. This will also automatically scale variable where the scaling can't be expressed through a simple scale factor e.g. the IASI spectrum where different bands of the spectrum have different scaling factors.

Selected fields are converted to Float32 to save memory. Normally Float32 is more than sufficient to represent the instrument accuracy. Setting high_precision=true will in some case convert these variables to Float64.

maskingvalue = NaN will replace missing values with NaN. This normally floats but can create issues for integers. See documentation page for more information.

Example

julia> file_path = "test/testData/ASCA_SZR_1B_M03_20230329063300Z_20230329063558Z_N_C_20230329081417Z"
julia> ds = MetopDataset(file_path);
julia>
julia> # display metadata of a variable
julia> ds["latitude"]
latitude (82 × 96)
  Datatype:    Union{Missing, Float64} (Int32)
  Dimensions:  xtrack × atrack
  Attributes:
   description          = Latitude (-90 to 90 deg)
   missing_value        = Int32[-2147483648]
   scale_factor         = 1.0e-6
julia>
julia> # load a subset of a variable  
julia> lat_subset = ds["latitude"][1:2,1:3] # load a small subset of latitudes.
2×3 Matrix{Union{Missing,Float64}}:
    -33.7308  -33.8399  -33.949
    -33.7139  -33.823   -33.9322
julia>
julia> # load entire variable  
julia> lat = ds["latitude"][:,:]
julia>
julia> # close data set
julia> close(ds);

MetopDiskArray{T, N} <: AbstractMetopDiskArray{T, N}

Struct to handle lazy loading of a variable in a Metop product. The raw types in the product is mapped without any scaling. Auto conversion can be enabled for RecordSubType e.g. converting VInteger to Float64.

MetopDiskArray(file_pointer::IOStream,
    record_layouts::Vector{FixedRecordLayout},
    field_name::Symbol; auto_convert = true) -> MetopDiskArray

Constructor for MetopDiskArray that compute additional fields. auto_convert = true will automatically convert custom RecordSubType to commonly used data types e.g. converting VInteger to Float64.

MetopVariable{T, N, R <: DataRecord, A <: AbstractArray{T, N}} <: CommonDataModel.AbstractVariable{T, N}

MetopVariable wraps an AbstractArray so it can be used with MetopDataset. The data array is normally AbstractMetopDiskArray.

_get_data_record_layouts(internal_pointer_records::Vector{InternalPointerRecord},
    total_file_size::Integer, record_type::Type{<:DataRecord})::Vector{FixedRecordLayout}

Compute the record_layouts

_get_flexible_dims_file(file_pointer::IO, T::Type{<:BinaryRecord})

Read the flexible types from a product. Note that the IO position is not changed by calling the function

construct_disk_array(file_pointer::IOStream,
    record_layouts::Vector{<:RecordLayout},
    field_name::Symbol; auto_convert = true)

Construct a disk array. The type of disk array is automatically determined. The standard type of disk array is MetopDiskArray but there are also other types. FlexibleMetopDiskArray is returned for fields where the size varies inside the product, eg. IASI L2 "temperature_error".

fixed_size(T::Type{<:BinaryRecord})::Bool
fixed_size(T::Type{<:BinaryRecord}, fieldname::Symbol)::Bool

Get if the data record has a binary fixed size. This is often used as a trait via the "julia Holy Traits Pattern" fieldname is used to check if a specific field has a fixed binary size.

fixed_size_in_file(T::Type{<:BinaryRecord}, fieldname::Symbol)::Bool

Check if the field have a constant size in a product. Return false if the field size can vary within a single file.

get_description(T::Type{<:BinaryRecord}, field::Symbol)::AbstractString

Get the description for a given field in the BinaryRecord

Example

julia> get_description(ASCA_SZR_1B_V13, :sigma0_trip)
"Sigma0 triplet, re-sampled to swath grid, for 3 beams (fore, mid, aft) "

get_dimensions(T::Type{<:BinaryRecord})::Dict{String, <:Integer}

Get the the named dimensions in a BinaryRecord and their length.

Example

julia> get_dimensions(ASCA_SZR_1B_V13)
Dict{String, Int64} with 2 entries:
  "num_band" => 3
  "xtrack"   => 82

get_field_dimensions(T::Type{<:BinaryRecord}, field::Symbol)::Vector{<:AbstractString}

Get the named dimensions of a field in a BinaryRecord

Example

julia> get_field_dimensions(ASCA_SZR_1B_V13, :sigma0_trip)
["num_band", "xtrack"]

get_flexible_dim_fields(T::Type{<:BinaryRecord})::Dict{Symbol,Symbol}

Get a dictionary with field names as key and the corresponding flexible dim as value. Only fields representing a flexible dim is included. Must be implemented for Records containing flexible dim values.

Example

julia> get_flexible_dim_fields(IASI_SND_02)
Dict{Symbol, Symbol} with 4 entries:
  :co_nbr   => :CO_NBR
  :o3_nbr   => :O3_NBR
  :nerr     => :NERR
  :hno3_nbr => :HNO3_NBR

get_missing_value(T::Type{<:Record}, field::Symbol)

Get the value representing missing for the field. Default values are implemented for Integers but they can be overwritten for specific record types to account for different conventions.

get_raw_format_dim(T::Type{<:BinaryRecord}, field::Symbol)

Get the dimensions of the field as defined in the record format specification.

Example

julia> get_raw_format_dim(ASCA_SZR_1B_V13, :sigma0_trip)
(3, 82, 1, 1)

get_scale_factor(T::Type{<:BinaryRecord}, field::Symbol)::Union{Number,Nothing}

get the scale_factor for a given field in the BinaryRecord. The variable can late be scaled from integer to float by dividing with 10^scale_factor. Returns nothing if no scale factor is set.

Example

julia> get_scale_factor(ASCA_SZR_1B_V13, :sigma0_trip) 
6

layout_info_for_disk_array(record_layouts::Vector{<:RecordLayout}, field_name::Symbol)

Extract information need in MetopDiskArray

native_read(io::IO, T)::T

Read a single object of type T from io to a file in the native Metop format. Endianness is automatically converted.

Example

julia> file_pointer = open("ASCA_SZO_1B_M03_20230329063300Z_20230329063556Z_N_C_20230329081417Z")
julia> main_header = MetopDatasets.native_read(file_pointer, MainProductHeader)
julia> main_header.sensing_start
2023-03-29T06:33:00

native_sizeof(x)::Integer

The byte size of the type x in a METOP native product.

Example

julia> native_sizeof(RecordHeader)
20

read_record_layouts(file_pointer::IO, main_product_header::MainProductHeader)

Read the appropriate record layout from IO.

DummyRecord <:Record

The Dummy Measurement Data Record is a special case of the MDR. It is a generic record that is used to indicate the location of lost data within any product. One DMDR can replace a contiguous block of lost MDRs

FlexibleMetopDiskArray{T, N} <: AbstractMetopDiskArray{T, N}

Similar to MetopDiskArray but able to handle flexible record layout and fields where the size varies within a single product file. E.g. IASI L2 fields like "temperature_error".

FlexibleRecordLayout

Used to store the record layout of flexible size data records in a Native metop files.

IasiSpectrumDiskArray{T} <: AbstractMetopDiskArray{T, 4}

The IasiSpectrumDiskArray is a wrapper around a MetopDiskArray that enables the automatic scaling of the the IASI L1C spectrum using the GIADR_IASI_XXX_1C record information.

IasiWaveNumberDiskArray <: AbstractMetopDiskArray{Float64, 2}

The IasiWaveNumberDiskArray is a disk array that computes the wavenumber of the IASI spectrum. The wavenumber is computed using :idefnsfirst1b and :idefspectdwn1b from each data record.

InternalPointerRecord <: Record

The Internal Pointer Records (IPR) specifies the start of each block of records in the file sharing the same record type. This can be used to find the locations of data records or dummy records.

LazyByteField <: AbstractVector{Vector{UInt8}}

Type to read fields from records as a byte vector Note that LazyByteField does not work with conversion to netCDF.

RecordHeader

Also known as GRH.