-
Variational Inference
July 6, 2022
This is a primer on variational inference in machine learning, based on sections of Jordan et al. (An Introduction to Variational Methods for Graphical Models; 1999). I go over the mathematical forms of variational inference, and I include a discussion on what it means for something to be “variational.” I hope this conveys a bit of the generating ideas that give rise to the various forms of variational inference. …
-
Variational Solomonoff Induction
February 18, 2021
$$ \newcommand{\mb}{\mathbb} \newcommand{\mc}{\mathcal} \newcommand{\E}{\mb{E}} \newcommand{\B}{\mb{B}} \newcommand{\R}{\mb{R}} \newcommand{\kl}[2]{D_{KL}\left(#1\ \| \ #2\right)} \newcommand{\argmin}[1]{\underset{#1}{\mathrm{argmin}}\ } \newcommand{\argmax}[1]{\underset{#1}{\mathrm{argmax}}\ } \newcommand{\abs}[1]{\left\lvert#1\right\rvert} \newcommand{\set}[1]{\left\{#1\right\}} \newcommand{\ve}{\varepsilon} \newcommand{\t}{\theta} \newcommand{\T}{\Theta} \newcommand{\o}{\omega} \newcommand{\O}{\Omega} \newcommand{\sm}{\mathrm{softmax}} $$ The free energy principle is a variational Bayesian method for approximating posteriors. Can free energy minimization combined with program synthesis methods from machine learning tractably approximate Solomonoff induction (i.e. universal inference)? In these notes, I explore what the combination of these ideas looks like. Machine learning I want to make an important clarification about “Bayesian machine learning”.…