Welcome to TUM-DI-LAB’s documentation!

• Preface

About Gaussian Processes

• Introduction to Gaussian processes
• Simple example of a Gaussian process
• Linear operators on GPs

Parameter estimation with Gaussian Processes

• 1D Linear operator with one parameter
• 2D Linear operator with one parameter
• 1D Linear operator with two parameters
• 1D linear operator with a zero parameter

Linear PDEs

• Heat equation
• Wave equation

Non-linear PDEs

• The inviscid Burgers’ Equation
• The inviscid Burgers’ Equation - A different approach

Approach with pyGPs

• Parameter Estimation for a linear operator using pyGPs

Conclusion

It has been demonstrated that estimating parameters using a Gaussian process approach works well for a variety of PDEs arising from linear operators. Even though we have worked with time schemes in the example of Burgers’ Equation, this methodology in general circumvents the need for any discretization methods. When working with truly noisy data, the parameter s (which we have set to 1e-7 in most cases) can be varied or even optimized as well. This approach unfortunately can’t really be applied to non-linear equations, only by using workarounds like replacing the non-linear term with the mean as we did in Burgers’ Equation, essentially transforming the non-linear equation to a linear one. Using this remedy, we still get good results as we saw in Chapters 4.2.4 and 4.2.5. In our case, five to eight data samples were sufficient to learn the parameters within a margin of error of 0.058 and 0.0009 in the linear, and 0.5 and 0.16 in the non-linear cases. Using 20 data samples, we were able to estimate the parameters to an accuracy of 0.6 percent.

Problems with the RBF kernel

One of the problems we noticed with the RBF kernel is that for more than 30 data points, the final covariance matrix was frequently ill-conditioned. This happens more often, when the length-scale is large. With the increased number of points, the probability of points being close to each other increases and then the respective columns of the covariance matrix will be almost equal, especially when working with a large length-scale. Hence, this kernel is not good enough for practical purposes on large datasets. The immediate solution that comes to mind to tackle this problem is to do a singular value decomposition. [1] outlines the RBF-QR algorithm to avoid these issues. Stable computations can also be acheived using Hermite polynomials [2], [3].

Non-linearity

In Burgers’ Equation we approximated the non-linear term by a constant term. This yields good results for some special cases but is not a generic approach. The problem arises because the product of Gaussian processes does not result in a Gaussian process. Nevertheless, we could utilize the fact that the product of Gaussian distributions is also Gaussian in order to come up with a proper solution. Another approach is to assume priors over the kernel hyperparameters and infer parameters with MCMC sampling schemes [4].

Kernel computations

The current framework involves two computations for every operator over the kernel. It is easy to do this by hand for simple operators but even then there is scope for manual error. It would be nice to have a tool to compute the transformed kernels automatically. Some progress has been made at this front resulting in a package for symbolic kernel computations available on this project’s GitHub repository [5].

[1]	Gregory E. Fasshauer and Michael J. McCourt. Stable Evaluation of Gaussian Radial Basis Function Interpolants. SIAM Journal on Scientific Computing, 34(2):A737–A762, jan 2012. URL: http://epubs.siam.org/doi/10.1137/110824784, arXiv:arXiv:1302.5877, doi:10.1137/110824784.

[2]

Bengt Fornberg, Elisabeth Larsson, and Natasha Flyer. Stable Computations with Gaussian Radial Basis Functions. SIAM Journal on Scientific Computing, 33(2):869–892, jan 2011. URL: http://epubs.siam.org/doi/10.1137/090750688 http://epubs.siam.org/doi/10.1137/09076756X, arXiv:arXiv:1302.5877, doi:10.1137/09076756X.

[3]	Anna Yurova and Katharina Kormann. Stable evaluation of Gaussian radial basis functions using Hermite polynomials. ArXiv e-prints, sep 2017. URL: https://arxiv.org/pdf/1709.02164.pdf http://arxiv.org/abs/1709.02164, arXiv:1709.02164.

[4]	Ben Calderhead, Mark Girolami, and Neil D Lawrence. Accelerating Bayesian Inference over Nonlinear Differential Equations with Gaussian Processes. Advances in Neural Information Processing Systems 21, pages 217–224, 2009.

[5]	Ahmed Ratnani, Kumar Harsha, Arthur Grundner, and Kangkang Wang. Machine Learning for Hidden Physics and Partial Differential Equations. 2018. URL: https://github.com/ratnania/mlhiphy.

[6]	E Solak, R Murray-Smith, W.E. Leithead, D.J. Leith, and C.E. Rasmussen. Derivative observations in Gaussian process models of dynamic systems. Nips 15, pages 8, 2002. URL: https://papers.nips.cc/paper/2287-derivative-observations-in-gaussian-process-models-of-dynamic-systems.pdf.

[7]	Carl E. Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning. Volume 14. The MIT Press, 2004. ISBN 026218253X. URL: http://www.gaussianprocess.org/gpml/chapters/RW.pdf.

[8]	R B Platte and T A Driscoll. Polynomials and potential theory for Gaussian radial basis function interpolation. SIAM Journal on Numerical Analysis, 2005. doi:10.1137/040610143.

[9]	Gabriel J. Lord, Catherine E. Powell, and Tony Shardlow. An Introduction to Computational Stochastic PDEs. Cambridge University Press, Cambridge, 2014. ISBN 9781139017329. URL: http://ebooks.cambridge.org/ref/id/CBO9781139017329, doi:10.1017/CBO9781139017329.

[10]	Marion Neumann, Shan Huang, Daniel E Marthaler, and Kristian Kersting. pyGPs – A Python Library for Gaussian Process Regression and Classification. Journal of Machine Learning Research, 16:26112616, 2015.

[11]	Carl Jidling, Niklas Wahlström, Adrian Wills, and Thomas B. Schön. Linearly constrained Gaussian processes. ArXiv e-prints, mar 2017. URL: http://arxiv.org/abs/1703.00787, arXiv:1703.00787.

[12]	Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Inferring solutions of differential equations using noisy multi-fidelity data. Journal of Computational Physics, 335:736–746, 2017. URL: http://dx.doi.org/10.1016/j.jcp.2017.01.060, arXiv:1607.04805, doi:10.1016/j.jcp.2017.01.060.

[13]	Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Machine learning of linear differential equations using Gaussian processes. Journal of Computational Physics, 348:683–693, 2017. URL: http://dx.doi.org/10.1016/j.jcp.2017.07.050, arXiv:1701.02440, doi:10.1016/j.jcp.2017.07.050.

[14]	J. Rashidinia, M. Khasi, and G. E. Fasshauer. A stable Gaussian radial basis function method for solving nonlinear unsteady convection–diffusion–reaction equations. Computers and Mathematics with Applications, 2018. doi:10.1016/j.camwa.2017.12.007.

[15]	Pedro Henrique de Almeida Konzen, Esequia Sauter, Fabio Souto de Azevedo, and Paulo Ricardo de Ávila Zingano. Numerical simulations with the finite element method for the Burgers’ equation on the real line. ArXiv e-prints, may 2016. URL: http://arxiv.org/abs/1605.01109, arXiv:1605.01109.

[16]	Michael P. Lamoureux. The mathematics of PDEs and the wave equation. In Seismic Imaging Summer School. Pacific Institute for the Mathematical Sciences, 2006. URL: http://www.mathtube.org/sites/default/files/lecture-notes/Lamoureux{\_}Michael.pdf.

[17]	Marion Neumann, Shan Huang, Daniel E. Marthaler, and Kristian Kersting. Pygps – a python library for gaussian process regression and classification. Journal of Machine Learning Research, 16:2611–2616, 2015. URL: http://jmlr.org/papers/v16/neumann15a.html.

Appendix

• Demo of pyGPs
• Own Covariance function for the Heat Equation with pyGPs
• Exploring the GPy package