Description: An attempt at classifier hyperparameter optimization with a non-exhaustive search. This is based on the Weka CVParameterSelection class.
Classifiers have parameters that control their functionality. These are called 'hyperparameters'. The Wiki article referenced in [1] mentions the Auto-Weka [2] package. I tried the software for this a little while back and found it a bit difficult to use. The related paper [3] might be of interest.
Weka's CVParameterSelection is given high and low bounds and a number defining the number of steps to take through that range for one or more classifier hyperparameters. It then recursively considers each combination of parameters in these ranges to exhaustively find which gives a minimal cross-validation error.
After using this and the Weka GridSearch and doing some manual adjusting of parameters trying to find a best set for a given pair of data and algorithm I thought this often seemed sort of similar to my understanding of gradient descent. For example with RandomForest and the old default of only 10 trees you could normally optimize by increasing the number of trees 10 at a time. Sooner or later this would stop working. So if that gives the best improvement, like gradient gives the steepest descent, then you step in that direction until it stops giving improvements.
So I thought I could apply this idea to either CVParameterSelection or GridSearch and having decided on the former came up with CVStepper.