Skip to contents

Extending compboost with losses

ext_losses.Rmd

Before Starting

  • Read the use-case site to get to know how to define a Compboost object using the R6 interface

What is Needed

compboost was designed to provide a component-wise boosting framework with maximal flexibility. This vignette gives an overview how to define custom losses in R as well as in C++ without recompiling the whole package. These custom losses can be used for training the model and/or logging mechanisms.

The loss function for boosting must be differentiable. Hence, we need to define the loss function and the gradient. Further, boosting is initialized as loss optimal constant. To capture this, we have to define the loss optimal constant as function of a response vector. Having these three components, it is quite easy to define custom losses.

As showcase, we are rebuilding two different loss functions:

  • The quadratic loss as easy example for C++
  • The Poisson loss for counting data as more sophisticated loss example in R

Define a New Loss With R

For this example we are using the VonBort dataset provided by the package vcd:

“Data from von Bortkiewicz (1898), given by Andrews & Herzberg (1985), on number of deaths by horse or mule kicks in 14 corps of the Prussian army.”

data(VonBort, package = "vcd")

We like to model the deaths using a Poisson regression in boosting. That means we have to define a proper loss function, the gradient, and the constant initialization.

The scheme for the loss, the gradient, and the constant initialization is to specify a function of the following form:

  • loss: function (truth, response)
  • gradient: function (truth, response)
  • constant initializer: function (truth)

The loss function

\[L(y,f) = -\log\left( \exp(f)^y \exp(\exp(f)) \right) - \log(y!)\]

lossPoisson = function (truth, response) {
  return(-log(exp(response)^truth * exp(-exp(response))) - gamma(truth + 1))
}

The gradient of the loss function

\[\frac{\partial}{\partial f} L(y,f) = \exp(f) - y\]

gradPoisson = function (truth, response) {
  return(exp(response) - truth)
}

The constant initialization

\[\mathsf{arg min}_{c\in\mathbb{R}} \sum_{i = 1}^n L\left(y^{(i)}, c\right) = \log(\bar{y})\]

constInitPoisson = function (truth) {
  return(log(mean(truth)))
}

Define the loss

Finally, having these three components allows to define a LossCustom object:

# Define custom loss:
my_poisson_loss = LossCustom$new(lossPoisson, gradPoisson, constInitPoisson)

Train a model

This loss object can be used for any task that requires a loss object:

cboost = Compboost$new(VonBort, "deaths", loss = my_poisson_loss)
cboost$addBaselearner("year", "spline", BaselearnerPSpline)
cboost$train(100, trace = 0)

Define a New Loss With C++

For this example, to keep it simple, we are using the iris dataset with Sepal.Length as target. The aim is to replicate the quadratic loss. This is achieved by exposing external pointers to R which hold the function definition and is passed to the constructor of the LossCustomCpp class.

A general advise is to write a .cpp file that contains the whole definition. This file needs to be sourced by Rcpp::sourceCpp().

We have to declare the head first to be able to expose functions:

// [[Rcpp::depends(RcppArmadillo)]]
#include <RcppArmadillo.h>

typedef arma::mat (*lossFunPtr) (const arma::mat& true_value, const arma::mat& prediction);
typedef arma::mat (*gradFunPtr) (const arma::mat& true_value, const arma::mat& prediction);
typedef double (*constInitFunPtr) (const arma::mat& true_value);

As the type definition already indicates, the C++ functions require the following signature:

  • loss: arma::mat lossFun (const arma::mat& truth, const arma::mat& response)
  • gradient: arma::mat gradFun (const arma::mat& truth, const arma::mat& response)
  • constant initializer: constInitFun (const arma::mat& true_value)

The loss function

\[L(y,f) = -0.5 \left(y - f\right)^2\]

arma::mat lossFun (const arma::mat& true_value, const arma::mat& prediction)
{
  return arma::pow(true_value - prediction, 2) / 2;
}

The gradient of the loss function

\[\frac{\partial}{\partial f} L(y,f) = f - y\]

arma::mat gradFun (const arma::mat& true_value, const arma::mat& prediction)
{
  return prediction - true_value;
}

The constant initialization

\[\mathsf{arg min}_{c\in\mathbb{R}} \sum_{i = 1}^n L\left(y^{(i)}, c\right) = \bar{y}\]

double constInitFun (const arma::mat& true_value)
{
  return arma::accu(true_value) / true_value.size();
}

Exposing external pointer

These functions are exposed to an XPtr. This stores the pointer to the function and can be used as parameter for the LossCustomCpp.

Note that it is not necessary to export the upper functions, exporting the pointer is the goal.

// [[Rcpp::export]]
Rcpp::XPtr<lossFunPtr> lossFunSetter ()
{
  return Rcpp::XPtr<lossFunPtr> (new lossFunPtr (&lossFun));
}

// [[Rcpp::export]]
Rcpp::XPtr<gradFunPtr> gradFunSetter ()
{
  return Rcpp::XPtr<gradFunPtr> (new gradFunPtr (&gradFun));
}

// [[Rcpp::export]]
Rcpp::XPtr<constInitFunPtr> constInitFunSetter ()
{
  return Rcpp::XPtr<constInitFunPtr> (new constInitFunPtr (&constInitFun));
}

Define the loss

We can get the pointer to the function after exposing them:

lossFunSetter()
gradFunSetter()
constInitFunSetter()

Now, we can pass the pointer to the LossCustomCpp class:

my_cpp_loss = LossCustomCpp$new(lossFunSetter(), gradFunSetter(), constInitFunSetter())

Train a model

Finally, we use the custom loss to train our model:

cboost = boostSplines(data = iris, target = "Sepal.Length",
  loss = my_cpp_loss, trace = 25)