Welcome to TUM-DI-LAB’s documentation!
Parameter estimation with Gaussian Processes
Conclusion
It has been demonstrated that estimating parameters using a Gaussian process approach works well for a variety of PDEs arising from linear operators.
Even though we have worked with time schemes in the example of Burgers’ Equation, this methodology in general circumvents the need for
any discretization methods. When working with truly noisy data, the parameter s (which we have set to 1e-7 in most cases) can be varied or even optimized as well. This approach unfortunately can’t really be applied to non-linear equations, only by using workarounds like replacing the non-linear term with the mean as we did in Burgers’ Equation, essentially transforming the non-linear equation to a linear one. Using this remedy, we still get good results as we saw in Chapters 4.2.4 and 4.2.5.
In our case, five to eight data samples were sufficient to learn the parameters within a margin of error of 0.058 and 0.0009 in the linear, and 0.5 and 0.16 in the non-linear cases. Using 20 data samples, we were able to estimate the parameters to an accuracy of 0.6 percent.
Problems with the RBF kernel
One of the problems we noticed with the RBF kernel is that for more than 30 data points, the final covariance matrix was frequently ill-conditioned. This happens more often, when the length-scale is large. With the increased number of points, the probability of points being close to each other increases and then the respective columns of the covariance matrix will be almost equal, especially when working with a large length-scale. Hence, this kernel is not good enough for practical purposes on large datasets. The immediate solution that comes to mind to tackle this problem is to do a singular value decomposition. [1] outlines the RBF-QR algorithm to avoid these issues. Stable computations can also be acheived using Hermite polynomials [2], [3].
Non-linearity
In Burgers’ Equation we approximated the non-linear term by a constant term. This yields good results for some special cases but is not a generic approach. The problem arises because the product of Gaussian processes does not result in a Gaussian process. Nevertheless, we could utilize the fact that the product of Gaussian distributions is also Gaussian in order to come up with a proper solution. Another approach is to assume priors over the kernel hyperparameters and infer parameters with MCMC sampling schemes [4].
Kernel computations
The current framework involves two computations for every operator over the kernel. It is easy to do this by hand for simple operators but even then there is scope for manual error. It would be nice to have a tool to compute the transformed kernels automatically. Some progress has been made at this front resulting in a package for symbolic kernel computations available on this project’s GitHub repository [5].
[4] | Ben Calderhead, Mark Girolami, and Neil D Lawrence. Accelerating Bayesian Inference over Nonlinear Differential Equations with Gaussian Processes. Advances in Neural Information Processing Systems 21, pages 217–224, 2009. |
[8] | R B Platte and T A Driscoll. Polynomials and potential theory for Gaussian radial basis function interpolation. SIAM Journal on Numerical Analysis, 2005. doi:10.1137/040610143. |
[10] | Marion Neumann, Shan Huang, Daniel E Marthaler, and Kristian Kersting. pyGPs – A Python Library for Gaussian Process Regression and Classification. Journal of Machine Learning Research, 16:26112616, 2015. |
[14] | J. Rashidinia, M. Khasi, and G. E. Fasshauer. A stable Gaussian radial basis function method for solving nonlinear unsteady convection–diffusion–reaction equations. Computers and Mathematics with Applications, 2018. doi:10.1016/j.camwa.2017.12.007. |
[15] | Pedro Henrique de Almeida Konzen, Esequia Sauter, Fabio Souto de Azevedo, and Paulo Ricardo de Ávila Zingano. Numerical simulations with the finite element method for the Burgers’ equation on the real line. ArXiv e-prints, may 2016. URL: http://arxiv.org/abs/1605.01109, arXiv:1605.01109. |
[17] | Marion Neumann, Shan Huang, Daniel E. Marthaler, and Kristian Kersting. Pygps – a python library for gaussian process regression and classification. Journal of Machine Learning Research, 16:2611–2616, 2015. URL: http://jmlr.org/papers/v16/neumann15a.html. |