Before Starting

  • Read the use-case site to get to know how to define a Compboost object using the R6 interface

What is Needed

compboost was designed to provide a component-wise boosting framework with maximal flexibility. This document gives an overview about how to define custom losses in R as well as in C++ without recompiling the whole package. These custom losses then can be used for training the model or logging mechanisms.

In boosting the loss function has one huge restriction, it must be differentiable. Therefore, we need not just the definition of the loss function, but also the gradient of it. Even worse, in boosting the first iteration is initialized as loss optimal constant dependent on the response. To capture this, we have also to define a loss optimal constant as function of the response. Having this three components, it is quite easy to define custom losses.

For this topic we are rebuilding two different loss functions:

  • The quadratic loss as easy example for C++
  • The Poisson loss for counting data as more sophisticated loss example for R

Define a New Loss With R

For this example we are using the VonBort dataset provided by the package vcd:

“Data from von Bortkiewicz (1898), given by Andrews & Herzberg (1985), on number of deaths by horse or mule kicks in 14 corps of the Prussian army.”

data(VonBort, package = "vcd")

We like to model the deaths using the poisson regression context in boosting. That means we have to define the proper loss function, the gradient, and the constant initializer.

The scheme for the loss and the gradient is to specify a function of the following form:

  • loss: function (truth, response)
  • gradient: function (truth, response)
  • constant initializer: function (truth)

The loss function

\[L(y,f) = -\log\left( \exp(f)^y \exp(\exp(f)) \right) - \log(y!)\]

lossPoisson = function (truth, response) {
  return(-log(exp(response)^truth * exp(-exp(response))) - gamma(truth + 1))

The gradient of the loss function

\[\frac{\partial}{\partial f} L(y,f) = \exp(f) - y\]

gradPoisson = function (truth, response) {
  return(exp(response) - truth)

The constant initialization

\[\mathsf{arg min}_{c\in\mathbb{R}} \sum_{i = 1}^n L\left(y^{(i)}, c\right) = \log(\bar{y})\]

constInitPoisson = function (truth) {

Define the loss

Finally, having the three components defined, we can pass them to the LossCustom object which sets them for the specific task:

# Define custom loss:
my.poisson.loss = LossCustom$new(lossPoisson, gradPoisson, constInitPoisson)

Train a model

This loss object can now be used for any task that requires a loss object, such as defining a compboost model:

cboost = Compboost$new(VonBort, "deaths", loss = my.poisson.loss)
cboost$addBaselearner("year", "spline", BaselearnerPSpline)
cboost$train(100, trace = 0)
#> Train 100 iterations in 0 Seconds.
#> Final risk based on the train set: -0.58

Define a New Loss With C++

For this example we are using the iris dataset with Sepal.Length as target to keep it simple. We now want to replicate the quadratic loss. This is done by exposing external pointer to R that holds the function definition which then passed to the LossCustomCpp class.

A general advise is to write a .cpp file that contains the whole definition. This file then just needs to be sourced by Rcpp::sourceCpp().

To be able to expose that functions we have to declare the head first:

As the type definition already indicates, the C++ function must have the following signatures:

  • loss: arma::mat lossFun (const arma::mat& truth, const arma::mat& response)
  • gradient: arma::mat gradFun (const arma::mat& truth, const arma::mat& response)
  • constant initializer: constInitFun (const arma::mat& true_value)

The gradient of the loss function

\[\frac{\partial}{\partial f} L(y,f) = f - y\]

The constant initialization

\[\mathsf{arg min}_{c\in\mathbb{R}} \sum_{i = 1}^n L\left(y^{(i)}, c\right) = \bar{y}\]

Exposing external pointer

Now wrap the function to an XPtr. This one stores the pointer to the function and can be used as parameter for the LossCustomCpp.

Note that it isn’t necessary to export the upper functions since we are interested in exporting the pointer not the function.

Define the loss

After exposing the function we can get the pointer to the function by calling it:

#> <pointer: 0x55a52d95be40>
#> <pointer: 0x55a52f3027a0>
#> <pointer: 0x55a52f3aa6b0>

This can now passed to the LossCustomCpp class:

my_cpp_loss = LossCustomCpp$new(lossFunSetter(), gradFunSetter(), constInitFunSetter())

Train a model

Finally, we use the custom loss to train our model:

cboost = boostSplines(data = iris, target = "Sepal.Length", 
  loss = my_cpp_loss, trace = 25)
#>   1/100   risk = 0.31  
#>  25/100   risk = 0.08  
#>  50/100   risk = 0.057  
#>  75/100   risk = 0.05  
#> 100/100   risk = 0.046  
#> Train 100 iterations in 0 Seconds.
#> Final risk based on the train set: 0.046