-
Bayesian information theory
April 9, 2021
$$ \newcommand{\0}{\mathrm{false}} \newcommand{\1}{\mathrm{true}} \newcommand{\mb}{\mathbb} \newcommand{\mc}{\mathcal} \newcommand{\mf}{\mathfrak} \newcommand{\and}{\wedge} \newcommand{\or}{\vee} \newcommand{\a}{\alpha} \newcommand{\t}{\theta} \newcommand{\T}{\Theta} \newcommand{\D}{\Delta} \newcommand{\o}{\omega} \newcommand{\O}{\Omega} \newcommand{\x}{\xi} \newcommand{\z}{\zeta} \newcommand{\fa}{\forall} \newcommand{\ex}{\exists} \newcommand{\X}{\mc{X}} \newcommand{\Y}{\mc{Y}} \newcommand{\Z}{\mc{Z}} \newcommand{\P}{\Psi} \newcommand{\y}{\psi} \newcommand{\p}{\phi} \newcommand{\l}{\lambda} \newcommand{\B}{\mb{B}} \newcommand{\m}{\times} \newcommand{\E}{\mb{E}} \newcommand{\e}{\varepsilon} \newcommand{\set}[1]{\left\{#1\right\}} \newcommand{\par}[1]{\left(#1\right)} \newcommand{\abs}[1]{\left\lvert#1\right\rvert} \newcommand{\inv}[1]{{#1}^{-1}} \newcommand{\ceil}[1]{\left\lceil#1\right\rceil} \newcommand{\dom}[2]{#1_{\mid #2}} \newcommand{\df}{\overset{\mathrm{def}}{=}} \newcommand{\M}{\mc{M}} \newcommand{\up}[1]{^{(#1)}} \newcommand{\tr}{\rightarrowtail} $$ $\newcommand{\H}{\Omega}$ Shannon’s information theory defines quantity of information (e.g. self-information $-\lg p(x)$) in terms of probabilities. In the context of data compression, these probabilities are given a frequentist interpretation (Shannon makes this interpretation explicit in his 1948 paper).…
-
Deconstructing Bayesian Inference
March 31, 2021
$$ \newcommand{\0}{\mathrm{false}} \newcommand{\1}{\mathrm{true}} \newcommand{\mb}{\mathbb} \newcommand{\mc}{\mathcal} \newcommand{\and}{\wedge} \newcommand{\or}{\vee} \newcommand{\a}{\alpha} \newcommand{\t}{\theta} \newcommand{\T}{\Theta} \newcommand{\o}{\omega} \newcommand{\O}{\Omega} \newcommand{\x}{\xi} \newcommand{\z}{\zeta} \newcommand{\l}{\lambda} \newcommand{\fa}{\forall} \newcommand{\ex}{\exists} \newcommand{\X}{\mc{X}} \newcommand{\Y}{\mc{Y}} \newcommand{\Z}{\mc{Z}} \newcommand{\H}{\mc{H}} \newcommand{\P}{\mc{P}} \newcommand{\y}{\psi} \newcommand{\p}{\phi} \newcommand{\B}{\mb{B}} \newcommand{\m}{\times} \newcommand{\E}{\mb{E}} \newcommand{\e}{\varepsilon} \newcommand{\set}[1]{\left\{#1\right\}} \newcommand{\abs}[1]{\left\lvert#1\right\rvert} \newcommand{\inv}[1]{{#1}^{-1}} \newcommand{\ceil}[1]{\left\lceil#1\right\rceil} $$ Constructing Bayesian Inference Before deconstructing Bayesian inference, I will present the general definition. At any point, feel free to look at the #Use Cases section for examples of Bayesian inference to use as intuition pumps. An “agent” here refers to a physical entity that tries to predict the future.…
-
Classical vs Bayesian Reasoning
February 24, 2021
$$ \newcommand{\0}{\mathrm{false}} \newcommand{\1}{\mathrm{true}} \newcommand{\mb}{\mathbb} \newcommand{\mc}{\mathcal} \newcommand{\and}{\wedge} \newcommand{\or}{\vee} \newcommand{\a}{\alpha} \newcommand{\t}{\theta} \newcommand{\T}{\Theta} \newcommand{\o}{\omega} \newcommand{\O}{\Omega} \newcommand{\x}{\xi} \newcommand{\fa}{\forall} \newcommand{\ex}{\exists} \newcommand{\X}{\mc{X}} \newcommand{\Y}{\mc{Y}} \newcommand{\Z}{\mc{Z}} \newcommand{\P}{\Psi} \newcommand{\y}{\psi} \newcommand{\p}{\phi} \newcommand{\B}{\mb{B}} \newcommand{\m}{\times} \newcommand{\E}{\mc{E}} \newcommand{\e}{\varepsilon} \newcommand{\set}[1]{\left\{#1\right\}} \newcommand{\abs}[1]{\left\lvert#1\right\rvert} \newcommand{\inv}[1]{{#1}^{-1}} \newcommand{\Iff}{\Leftrightarrow} $$ My goal is to identify the core conceptual difference between someone who accepts “Bayesian reasoning” as a valid way to obtain knowledge about the world, vs someone who does not accept Bayesian reasoning, but does accept “classical reasoning”. By classical reasoning, I am referring to the various forms of boolean logic that have been developed, starting with Aristotelian logic, through propositional logic like that of Frege, and culminating in formal mathematics (e.…