Improved learning algorithms for restricted Boltzmann machines

 |  Login

Show simple item record

dc.contributor Aalto-yliopisto fi
dc.contributor Aalto University en
dc.contributor.advisor Ilin, Alexander
dc.contributor.advisor Raiko, Tapani
dc.contributor.author Cho, KyungHyun
dc.date.accessioned 2012-07-02T08:08:54Z
dc.date.available 2012-07-02T08:08:54Z
dc.date.issued 2011
dc.identifier.uri https://aaltodoc.aalto.fi/handle/123456789/3666
dc.description.abstract A restricted Boltzmann machine (RBM) is often used as a building block for constructing deep neural networks and deep generative models which have gained popularity recently as one way to learn complex and large probabilistic models. In these deep models, it is generally known that the layer-wise pretraining of RBMs facilitates finding a more accurate model for the data. It is, hence, important to have an efficient learning method for RBM. The conventional learning is mostly performed using the stochastic gradients, often, with the approximate method such as contrastive divergence (CD) learning to overcome the computational difficulty. Unfortunately, training RBMs with this approach is known to be difficult, as learning easily diverges after initial convergence. This difficulty has been reported recently by many researchers. This thesis contributes important improvements that address the difficulty of training RBMs. Based on an advanced Markov-Chain Monte-Carlo sampling method called parallel tempering (PT), the thesis proposes a PT learning which can replace CD learning. In terms of both the learning performance and the computational overhead, PT learning is shown to be superior to CD learning through various experiments. The thesis also tackles the problem of choosing the right learning parameter by proposing a new algorithm, the adaptive learning rate, which is able to automatically choose the right learning rate during learning. A closer observation into the update rules suggested that learning by the traditional update rules is easily distracted depending on the representation of data sets. Based on this observation, the thesis proposes a new set of gradient update rules that are more robust to the representation of training data sets and the learning parameters. Extensive experiments on various data sets confirmed that the proposed rules indeed improve learning significantly. Additionally, a Gaussian-Bernoulli RBM (GBRBM) which is a variant of an RBM that can learn continuous real-valued data sets is reviewed, and the proposed improvements are tested upon it. The experiments showed that the improvements could also be made for GBRBMs. en
dc.format.extent xii + 84
dc.format.mimetype application/pdf
dc.language.iso en en
dc.publisher Aalto-yliopisto fi
dc.publisher Aalto University en
dc.title Improved learning algorithms for restricted Boltzmann machines en
dc.type G2 Pro gradu, diplomityö fi
dc.contributor.department Tietotekniikan laitos fi
dc.subject.keyword Boltzmann machine en
dc.subject.keyword restricted Boltzmann machine en
dc.subject.keyword annealed importance sampling en
dc.subject.keyword paraller tempering en
dc.subject.keyword enhanced gradient en
dc.subject.keyword adaptive learning rate en
dc.subject.keyword Gaussian-Bernoulli restricted Boltzmann machine en
dc.subject.keyword deep learning en
dc.identifier.urn URN:NBN:fi:aalto-201207022632
dc.type.dcmitype text en
dc.programme.major Informaatiotekniikka fi
dc.programme.mcode T-61
dc.type.ontasot Diplomityö fi
dc.type.ontasot Master's thesis en
dc.contributor.supervisor Karhunen, Juha


Files in this item

This item appears in the following Collection(s)

Show simple item record

Search archive


Advanced Search

article-iconSubmit a publication

Browse