Carles Riera, Camilo Rey-Torres, Eloi Puertas, Oriol Pujol

2019

In this article, we study a proposal that enables to train extremely thin (4 or 8 neurons per layer) and relatively deep (more than 100 layers) feedforward networks without resorting to any architectural modification such as Residual or Dense connections, data normalization or model scaling. We accomplish that by alleviating two problems. One of them are neurons whose output is zero for all the dataset, which renders them useless. This problem is known to the academic community as dead neurons. The other is a less studied problem, dead points. Dead points refers to data points that are mapped to zero during the forward pass of the network. As such, the gradient generated by those points is not propagated back past the layer where they die, thus having no effect in the training process. In this work, we characterize both problems and propose a constraint formulation that added to the standard loss function solves them both. As an additional benefit, the proposed method allows to initialize the network weights with constant or even zero values and still allowing the network to converge to reasonable results. We show very promising results on a toy, MNIST, and CIFAR-10 datasets.

Carles Riera, Oriol Pujol

2017

This work proposes a novel solution to the problem of internal covariate shift and dying neurons using the concept of linked neurons. We define the neuron linkage in terms of two constraints: first, all neuron activations in the linkage must have the same operating point. That is to say, all of them share input weights. Secondly, a set of neurons is linked if and only if there is at least one member of the linkage that has a non-zero gradient in regard to the input of the activation function. This means that for any input in the activation function, there is at least one member of the linkage that operates in a non-flat and non-zero area. This simple change has profound implications in the network learning dynamics. In this article we explore the consequences of this proposal and show that by using this kind of units, internal covariate shift is implicitly solved. As a result of this, the use of linked neurons allows to train arbitrarily large networks without any architectural or algorithmic trick, effectively removing the need of using re-normalization schemes such as Batch Normalization, which leads to halving the required training time. It also solves the problem of the need for standarized input data. Results show that the units using the linkage not only do effectively solve the aforementioned problems, but are also a competitive alternative with respect to state-of-the-art with very promising results.

Carles Riera, Oriol Pujol

2016

We propose a novel approach to approximately solve on-line kernel Support Vector Machines (SVM) with the number of support vectors set beforehand. To this aim, we modify the original formulation introducing a new constraint that penalizes the deviation with respect to the desired number of support vectors. The resulting problem is solved using stochastic subgradient methods with block coordinate descent. Comparison with state-of-the-art online methods shows very promising results.

Carles Riera, Oriol Pujol

2016

This work proposes a new online classifier based on Kernel SVM that minimizes the number of support vectors thanks to an additional restriction imposed by means of continuation methods. We build an homotopy between a normal SVM formulation to a restricted one with threshold on the magnitude of the coefficients, tightening it iteratively. We show how the use of the continuation method helps in achieving solutions that would be unfeasible directly. Experimentation in UCI datasets show significant savings.