Linear operators on GPs

Regularity of Stochastic Processes

Definition (mean-square continuity):

Let \(\{ X ( t ) : t \in \mathcal { T } \}\) be a mean-zero process. The kernel \(k\) is continuous in \((t,t)\) if and only if \(\mathbb { E } \left[ ( X ( t + h ) - X ( t ) ) ^ { 2 } \right] \rightarrow 0\) as \(h \rightarrow 0\). In particular, if \(k \in \mathrm { C } ( \mathcal { T } \times \mathcal { T } )\), then \(\{ X ( t ) : t \in \mathcal { T } \}\) is mean-square continuous.

Definition (mean-square derivative):

A process \(\{ X ( t ) : t \in \mathcal { T } \}\) is mean-square differentiable with mean-square derivative \(\frac { d X ( t ) } { d t }\) if, for all \(t \in \mathcal { T }\), we have as \(h \rightarrow 0\)

\(\| \frac { X ( t + h ) - X ( t ) } { h } - \frac { d X ( t ) } { d t } \| _ { L ^ { 2 } ( \Omega ) } = \mathbb { E } \left[ | \frac { X ( t + h ) - X ( t ) } { h } - \frac { d X ( t ) } { d t } | ^ { 2 } \right] ^ { 1 / 2 } \rightarrow 0\).

Theorem:

Let \(\{ X ( t ) : t \in \mathcal { T } \}\) be a stochastic process with mean zero. Suppose that the kernel \(k \in \mathrm { C } ^ { 2 } ( \mathcal { T } \times \mathcal { T } )\). Then \(X(t)\) is mean-square differentiable and the derivative \(\frac { d X ( t ) } { d t }\) has kernel \(\frac { \partial ^ { 2 } k ( s , t ) } { \partial s \partial t }\).

Proof:

For any \(s,t \in \mathcal{T}\) and real constants \(h_s,h_t > 0\),

\[\begin{split}\operatorname { Cov } ( \frac { X ( s + h_s ) - X ( s ) } { h_s } , \frac { X ( t + h_t ) - X ( t ) } { h_t } ) &= \frac { 1 } { h_s h_t } \mathbb { E } [ ( X ( s + h_s ) - X ( s ) ) ( X ( t + h_t ) - X ( t ) ) ] \\ &= \frac { 1 } { h_s h_t } ( k ( s + h_s , t + h_t ) - k ( s + h_s , t ) - k ( s , t + h_t ) + k ( s , t ) )\end{split}\]

A simple calculation with the Taylor series shows that the right-hand side converges to \(\frac { \partial ^ { 2 } k ( s , t ) } { \partial s \partial t }\) as \(h_s,h_t \rightarrow 0\).

With a similar approach and setting as in the previous theorem, we can calculate the covariance between a Gaussian process and its mean-square derivative.

\[\operatorname { Cov } ( X ( s ), & \frac { X ( t + h ) - X ( t ) } { h } ) = \frac { 1 } { h } \mathbb { E } [ ( X ( s ) ) ( X ( t + h ) - X ( t ) ) ] = \frac { 1 } { h } ( k ( s, t + h ) - k ( s , t ) )\]

The right hand side converges to \(\frac{\partial}{\partial t}k(s,t)\) as \(h \rightarrow 0\).

Theorem (mean-square regularity):

Let \(u(x)\) be a mean-zero second-order random field. If the kernel \(k \in C(D \times D)\), then \(u(x)\) is mean-square continuous so that \(\| u ( \mathbf{x} + \mathbf{h} ) - u ( \mathbf{x} ) \| _ { L ^ { 2 } ( \Omega ) } \rightarrow 0\) as \(h \rightarrow 0 \; \forall x \in D\). If \(k \in C^2(D \times D)\), then \(u(x)\) is mean-square differentiable. That is, a random field \(\frac { \partial u ( x ) } { \partial x _ { i } }\) exists such that

\[\| \frac { u \left( \mathbf { x } + h e _ { i } \right) - u ( \mathbf { x } ) } { h } - \frac { \partial u ( \mathbf { x } ) } { \partial x _ { i } } \| _ { L ^ { 2 } ( \Omega ) } \rightarrow 0 \quad \text { as } h \rightarrow 0\]

and \(\frac { \partial u ( x ) } { \partial x _ { i } }\) has the kernel \(k _ { i } ( x , y ) = \frac { \partial ^ { 2 } C ( x , y ) } { \partial x _ { i } \partial y _ { i } }\).

Especially this theorem tells us, how zero-mean Gaussian Processes transform, when taking derivatives. Raissi describes in his paper, that the following even holds for general linear transformations [13]:

Let \(u \sim GP(0, k_{uu})\) and \(\mathcal{L}_x\) be a linear transformation. Then for \(f = \mathcal{L}_x u\) it holds:

  1. \(f \sim GP(0, k_{ff})\)
  2. The covariance function of \(f\) is given by \(k_{ff} = \mathcal{L}_{x}\mathcal{L}_{x'}k_{uu}\).
  3. The covariance between \(u(x)\) and \(f(x')\) is given by \(k_{uf} = \mathcal{L}_{x'}k_{uu}\), whereas the covariance between \(f(x)\) and \(u(x')\) is given by \(k_{fu} = \mathcal{L}_x k_{uu}\).