In my previous post (see MLP: Incremental learning), I demonstrated that Incremental learning does not work for MLP. So how can we make MLP learn several functions ? The solution is to mix the entries in the training set. If we create 4000 groups containing 1 entry for the AND operator, then 1 entry for…
Category: MLP
MLP: Incremental learning
This article is about the capability of the MLP to learn incrementally in the same way as the brain does. The main result of this article is that MLP does not learn incrementally because it forgets what has been learned using the preceding training sets. Incremental Learning When we (as human beings) learn something (for…
Gradient Descent Algorithm: Impact of topologies on the learning curve of MLP for XOR
In this article I compare the learning curves of 2 different topologies of MLP when modifying the hyperparameters η and α of the Gradient Descent algorithm used to learn the XOR function. This article follows another one (see Gradient Descent Algorithm: Impact of η and α on the learning curve of MLP for XOR )…
Gradient Descent Algorithm: Impact of eta and alpha on the learning curve of MLP for XOR
In their famous article (see Learning Representations by Back-propagating Errors) Rumelhart, Hinton and Williams popularized the backpropagation mechanism in order to modify the weights of the links between neurons of different layers. They also added the momentum (parameter α alpha) to the classical learning rate (η eta) as a way to improve the Gradient Descent…
Capability of the MLP to learn XOR
In a series of posts, I will study the properties of the Multilayer Perceptron (MLP), starting with the capability to learn some mathematical functions (XOR, y=X², ..). This subject has been studied long time ago by researchers and George Cybenko demonstrated that any function could be approximated by a MLP (see Cybenko, G. 1989. Approximation…