Variational Bias Correction in UFO

Note

Summarized from :

Linear combination formulation

\[\vec{y} = H(\vec{x}) + \sum_{i=0}^{N} \beta_i p_i(\vec{x}) + \tilde{\vec{e}_o}\]

with scalar coefficients \(\beta_i, i = 0, . . . ,N\) . The selection of predictors \(p_i(\vec{x}), i = 1, . . . ,N\) is flexible and depends on the instrument and channel.

Augmented control variable

Define the augmented control vector

\[\vec{z}^T = \lbrack \vec{x}^T \vec{\beta}^T \rbrack\]

Therefore, the cost function to be minimized

\[J(\vec{z}) = \frac{1}{2} (\vec{z}_b - \vec{z})^T \textbf{Z}^{-1} (\vec{z}_b - \vec{z}) + \frac{1}{2} (\vec{y} - \tilde{H}(\vec{z}))^T \textbf{R}^{-1} (\vec{y} - \tilde{H}(\vec{z}))\]

where

(7)\[\tilde{H}(\vec{z}) = H(\vec{x}) + \sum_{i=0}^{N} \beta_i p_i(\vec{x})\]

Adjoint of the bias model

In the incremental formulation of the variational analysis, nonlinear observation operators are linearized about the latest outer-loop estimate \(\overline{\vec{x}}\) of \(\vec{x}\) . Similarly, for the modified operator we use

\[\begin{split}H(\vec{x}, \beta) \approx H(\overline{\vec{x}}, \beta) & = H(\overline{\vec{x}}) + \sum_{i=0}^{N} \beta_i p_i(\overline{\vec{x}}) \\ & = H(\overline{\vec{x}}) + \mathcal{P}(\overline{\vec{x}}) \cdot \vec{\beta}\end{split}\]

where \(\mathcal{P}(\overline{\vec{x}})\) is a \(m × n\) predictor matrix consisting of \(n\) predictors evaluated on \(m\) observation locations.

The modification to \(H(\vec{x})\) is therefore additive and linear in the bias parameters, and its adjoint with respect to these additional control parameters is trivial to implement.

For the linear predictor model (7), the derivatives with respect to the parameters are simply the values of the predictors at the observation locations

\[\frac{\partial H }{\partial \vec{\beta}} \Bigg \vert_{\vec{\beta} = \vec{\beta}_b} = \mathcal{P}(\overline{\vec{x}})\]

Background error covariance

In general the parameter estimation errors will be correlated with the state estimation errors, because they depend on the same data. We know of no practical way to account for this statistical dependence, and therefore take

\[\begin{split}\textbf{Z} = \begin{bmatrix} \textbf{B}_x & 0 \\ 0 & \textbf{B}_{\beta} \end{bmatrix}\end{split}\]

where \(\textbf{B}_x\) denotes the usual (state) background error covariance, and \(\textbf{B}_\beta\) the parameter background error covariance.

We take \(\textbf{B}_\beta\) diagonal:

\[\begin{split}\textbf{B}_\beta & = diag(\sigma_{\beta_1}^2, ...., \sigma_{\beta_n}^2) \\ & = \begin{bmatrix} \sigma_{\beta_1}^2 & & \\ & \ddots & \\ & & \sigma_{\beta_n}^2 \end{bmatrix} \\ & = \begin{bmatrix} \frac{\sigma_{o_1}^2}{N_1} & & \\ & \ddots & \\ & & \frac{\sigma_{o_n}^2}{N_j} \end{bmatrix}\end{split}\]

Here \(\beta_j\) denotes the \(j^{th}\) bias parameter, \(\sigma_{o_j}\) is the error standard deviation of the observations associated with \(\beta_j\), and \(N_j\) is a positive integer represents the number of observations.

Note

  • For example, taking \(N_j = 10,000\) for all parameters, the system will adapt quickly to changes in the bias for a clean channel generating thousands of radiances per analysis cycle.

  • On the other hand, it will respond slowly to a cloudy channel that generates only a few hundreds of data per cycle.

Note

  • When the \(N_j\) are sufficiently large (say, \(N_j >> 100\) ), the effect of neglecting off-diagonal elements of the parameter background error covariance matrix should be insignificant. This is because \(\mathcal{O}(N_j)\) observations are used to estimate just a few bias parameters; the estimation errors will be small even when the estimation is suboptimal.

  • The situation is, of course, very different for the state estimation, which can be extremely sensitive to the specification of the background error covariances, especially in data-sparse areas.

VarBC example

To use the VarBC in a Observation Operator, add the obs bias section as the highlighted lines.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
observations:
- obs space:
    name: AMSUA-NOAA19
    obsdatain:
      obsfile: Data/obs/testinput_tier_1/amsua_n19_obs_2018041500_m.nc4
    simulated variables: [brightness_temperature]
    channels: 1-15
  obs operator:
    name: CRTM
    Absorbers: [H2O,O3]
    obs options:
      Sensor_ID: &Sensor_ID amsua_n19
      EndianType: little_endian
      CoefficientPath: Data/crtm/
  obs error:
    covariance model: diagonal
  obs filters:
  - filter: Bounds Check
    filter variables:
    - name: brightness_temperature
      channels: &channels 1-15
    minvalue: 100.0
    maxvalue: 500.0
  - filter: Background Check
    filter variables:
    - name: brightness_temperature
      channels: *channels
    threshold: 3.0
  obs bias:
    prior:
      biasfile: Data/obs/testinput_tier_1/satbias_crtm_in_amsua_n19.nc4
      group: ObsBias
    analysis:
      biasfile: Data/hofx/satbias_crtm_out_amsua_n19.nc4
      group: ObsBias
    sensor: *Sensor_ID
    jobs: *channels
    covariance:
      minimal required obs number: 20
      variance range: [1.0e-6, 10.]
      variance:
        datain: Data/obs/testinput_tier_1/satbias_crtm_pc_in_amsua_n19.nc4
        dataout: Data/hofx/satbias_crtm_pc_out_amsua_n19.nc4
        group: ObsBiasCovariance
        inflation:
          ratio: 1.1
          ratio for small dataset: 2.0
    predictors:
    - predictor:
        name: constant
    - predictor:
        name: emissivity
    - predictor:
        name: scan_angle
        options:
          order: 4
    - predictor:
        name: scan_angle
        options:
          order: 3
    - predictor:
        name: scan_angle
        options:
          order: 2
    - predictor:
        name: scan_angle

Here is the detailed explanation:

  1. Defines the predictors (required)

Here, we defined 6 predictors, which are constant, emissivity, and 1st, 2nd, 3rd, 4th order scan_angle, respectively. To find what predictor functions are available, please refer to directory ufo/src/ufo/predictors/.

predictors:
- predictor:
    name: constant
- predictor:
    name: emissivity
- predictor:
    name: scan_angle
    options:
      order: 4
- predictor:
    name: scan_angle
    options:
      order: 3
- predictor:
    name: scan_angle
    options:
      order: 2
- predictor:
    name: scan_angle

  1. Defines the input file for the bias coefficients prior (optional)

    Usually, the prior is coming from the previous data assimilation cycle. if it is not available, all coefficients will start with zero.

prior:
  biasfile: Data/obs/testinput_tier_1/satbias_crtm_in_amsua_n19.nc4
  group: ObsBias       #  group name in NetCDF file

  1. Defines the bias coefficients analysis output file (optional)

    Usually, the analysis is the prior of the next data assimilation cycle.

analysis:
  biasfile: Data/hofx/satbias_crtm_out_amsua_n19.nc4
  group: ObsBias

  1. Defines the input (from the previous DA cycle) and the output (for the next DA cycle) for variance (optional)

Also defines the parameters which control the relationship between the number of effective observations and magnititude of the variances. please refer to [ZDC+14] for more details.

covariance:
  minimal required obs number: 20
  variance range: [1.0e-6, 10.]
  variance:
    datain: Data/obs/testinput_tier_1/satbias_crtm_pc_in_amsua_n19.nc4
    dataout: Data/hofx/satbias_crtm_pc_out_amsua_n19.nc4
    group: ObsBiasCovariance
    inflation:
      ratio: 1.1
      ratio for small dataset: 2.0

  1. Defines the Sensor_ID and Channels (jobs) for bias correction (required)

Usually, use the consistent values in ObsOperator

sensor: *Sensor_ID
jobs: *channels