Background¶

OOPS Observation Processing Flow¶

The flow of H(x) computation and QC filters application in OOPS is shown in the figure below.

Fig. 14 Flow chart for computing H(x) and running QC filters¶

The Observer calls the preProcess method of ObsFilters before the loop over time steps. After the loop, it calls the priorFilter and postFilter methods just before and just after calling the simulateObs method of ObsOperator. The observation filters are very generic and can perform a number of tasks, but mostly they are used for quality control.

In variational data assimilation, the above flow happens inside of the observation term of the cost function (J_o) evaluation.

Observation Filters¶

Observation filters have access to:

Observation values and metadata
Model values at observations locations (GeoVaLs)
Simulated observation value (for post-filter)
Their own private data

Most filters are written once and used with many observation types; several such generic filters already exist and are decribed below. Filters applied to observations from a specific ObsSpace need to be listed in the observations.obs filters section of the input YAML configuration file, together with any options controlling their behavior. Example:

observations:
- obs space:
    name: AMSUA-NOAA19
    obsdatain:
      obsfile: Data/obs/testinput_tier_1/amsua_n19_obs_2018041500_m.nc4
    simulated variables: [brightness_temperature]
    channels: 1-15
  obs filters:
  - filter: Bounds Check
    filter variables:
    - name: brightness_temperature
      channels: 1-15
    minvalue: 100.0
    maxvalue: 500.0
  - filter: Background Check
    filter variables:
    - name: brightness_temperature
      channels: 1-15
    threshold: 3.0

Derived Variables¶

Some filters need to create new or modify existing ObsSpace variables. For example, the Variable Transforms filter may create new variables representing wind velocity components computed from measurements of the wind speed and direction. Other filters may want to correct measurements or metadata loaded from the input file.

It is important to note that filters should never modify variables from the ObsValue group or add new variables to that group: otherwise initial and postprocessed measurements could not be distinguished, which would harm traceability. Instead, filters may create or modify variables from the DerivedObsValue group. The ObsSpace treats all groups with the Derived prefix in a special way. Each member function used to access existing variables, such as get_db(group, name, ...), checks first if a variable name exists in the group "Derived" + group and if so, it retrieves that variable; if not, the function looks for the variable name in the group group. As a result, variables from groups with the Derived prefix effectively “overshadow” corresponding variables from groups without that prefix. In the rare cases where it is important to access the original variables rather than their derived counterparts, ObsSpace member functions should be called with the skipDerived argument set to true.

There is no strict rule forbidding modifications to variables from other groups such as MetaData. However, for clarity, it may be a good idea to avoid modifying metadata loaded from the input file and instead store any corrected versions in variables from the DerivedMetaData group.

Note that to any variables from the DerivedObsValue group that need to be assimilated must be included in the obs space.derived simulated variables list rather than obs space.simulated variables (see observations). If these variables do not exist by the time the last filter finishes execution, an exception is thrown. (In contrast, variables from the obs space.simulated variables must exist already when the first filter starts execution.)

Observation Errors¶

Use the following variable groups to refer to estimates of observation errors of simulated variables valid at specific stages in the execution of a JEDI application:

ObsError: initial values loaded from the input ioda file.
ObsErrorData: up-to-date values (set by one of the preceding filters).
EffectiveError: final values obtained after execution of all filters. This group does not exist while filters are running, but is present in the output ioda file.

All these values represent standard deviations (square roots of variances).