Empirical Risk Minimization with f -Divergence Regularization in Statistical Learning

This report presents the solution to the empirical risk minimization with $f$-divergence regularization, under mild conditions on $f$. Under such conditions, the optimal measure is shown to be unique and to always exist. The solution is presented as a closed-form expression of the Radon-Nikodym derivative of the optimal probability measure with respect to the reference measure. Examples for particular choices of the function $f$ are presented. For some choices, existing results are obtained as special cases of the main result. These include the unique solutions to the empirical risk minimization with relative entropy regularization (Type-I and Type-II).