Training Neural Nets with the Reactive Tabu Search

In this paper the task of training sub-symbolic systems is considered as a combinatorial optimization problem and solved with the heuristic scheme of the Reactive Tabu Search (RTS) proposed by the authors and based on F. Glover's Tabu Search. An iterative optimization process based on a "modified greedy search" component is complemented with a meta-strategy to realize a discrete dynamical system that discourages limit cycles and the confinement of the search trajectory in a limited portion of the search space. The possible cycles are discouraged by prohibiting (i.e., making tabu) the execution of moves that reverse the ones applied in the most recent part of the search, for a prohibition period that is adapted in an automated way. The confinement is avoided and a proper exploration is obtained by activating a diversification strategy when too many configurations are repeated excessively often. The RTS method is applicable to non-differentiable functions, it is robust with respect to the random initialization and effective in continuing the search after local minima. The limited memory and processing required make RTS a competitive candidate for special-purpose VLSI implementations. We present and discuss four tests of the technique on feedforward and feedback systems