Initialization of weights in deep belief neural network based on standard deviation of feature values in training data vectors

N. Rezazadeh¹

1. Dept.of Computer Science and Research Branch, Islamic Azad University, Tehran, Iran.

Correspondence should be addressed to: naderrezazadeh1984@gmail.com.

Section:Research Paper, Product Type: Isroset-Journal
Vol.5 , Issue.4 , pp.1-8, Aug-2017

Online published on Aug 30, 2017

Copyright © N. Rezazadeh . This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
 

View this paper at   Google Scholar | DPI Digital Library

XML View     PDF Download

How to Cite this Paper

478 Views    288 Downloads    167 Downloads

Abstract :
Nowadays, the feature engineering approach has become very popular in deep neural networks. The purpose of this approach is to extract higher-level and more efficient features compared to those of learning data and to improve the learning of machines. One of the common ways in feature engineering is the use of deep belief networks. In addition, one of the problems in deep neural networks` training is the training process. The problems of the training process will be further enhanced in the event of an increase in the dimensions of the features and the complexity of the relationship between the initial features and the higher-level features. In the present paper, we attempt to set the initial weights based on the standard deviation of the feature vector values. Hence, a part of the training process is initially conducted and a better starting point can be provided for the weight training process. However, the impact of this method, to a large extent, depends on the relationship between the training data itself and the degree of independence of the training data`s feature values. Experiments conducted in this field have achieved acceptable results.

Key-Words / Index Term :
Neural Network; Restricted Boltzman Machine; Deep Belief Network

References :
. Liu, S. Zhou, Q. Chen, “Discriminative deep belief networks for visual data classification”, Pattern Recognition, vol.44, Issue.10, pp. 2287–2296, 2011.
[2] N. Rezazadeh, “A modification of the initial weights in the restricted Boltzman machine to reduce training error of the deep belief neural network”,International Journal of Computer Science and Information Security, vol.15, Issue.7, pp.1-6, 2017.
[3] R. Salakhutdinov, G. Hinton, “Deep Boltzmann Machines”, International Conference on Artificial Intelligence and Statistics (AISTATS 2009), Canada, pp.448-455, 2009.
[4] H. Lee, C. Ekanadham, and A. Ng, “Sparse deep belief net model for visual area V2” Advances in neural information processing systems, vol.20, pp.873–880, 2008.
[5] G. E. Hinton, R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks”, Science, vol.313, Issue.578, pp.504–507, 2006.
[6] V. Nair and G. Hinton, “3D object recognition with deep belief nets”, Advances in Neural Information Processing Systems, vol.22, pp.1339–1347, 2009.
[7] R. Salakhutdinov, G. E. Hinton, “Deep boltzmann machines,” in Proceedings of the international conference on artificial intelligence and statistics, vol.5, pp.448–455, 2009.
[8] R. Salakhutdinov, A. Mnih, G. Hinton , “Restricted Boltzman Machine for Collaborative Filtering”, Proceedings of the 24th international conference on Machine learning(2007), pp.791-798, 2007.
[9] N. Le Roux, Y. Bengio , “Representation Power of Restricted Boltzman Machines and Deep Beliefe Networks”, Vol.20, Issue.6, pp.1631-1649, 2008.
[10] Y. Bengio, “Learning Deep Architectures for AI”,Foundations and Trends in Machine Learning” Vol. 2, Issue.1, pp.1–127, 2009
[11] A. Fischer ,C. Igel, “An Introduction to Restricted Boltzmann Machines”, Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applicationspp(CIARP 2012), pp.14-36, 2012.
[12] H. Larochelle, Y. Bengio, “Classification using discriminative restricted Boltzmann machines,” in Proceedings of the 25th international conference on Machine learning, New York, USA, pp. 536–543, 2008.
[13] G. Hinton, “A practical guide to training restricted boltzmann machines”, Machine Learning Group, University of Toronto, Technical report, 2010.
[14] X. Wang, Vincent Ly, Ruiguo, Chandra Kambhamettu, “2D-3D face recognition via Restricted Boltzmann Machines”, International Conference on Computer Vision Theory and Applications (VISAPP),Lisbon, 2015.
[15] S. Iyanaga, Y. Kawada, “Distribution of Typical Random Variables” , Encyclopedic Dictionary of Mathematics. Cambridge MA(MIT Press), pp.1483-1486, 1980.
[16] M. Abramowitz, Stegun, I. A. (Eds.). “Probability Functions” Ch. 26 in Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables, Vol.9, pp.925-964, 1972.
[17] [17]-Car Evaluation Dataset http://archive.ics.uci.edu/ml/datasets/Car+Evaluation [Online Available].
[18] Heart Disease Dataset http://archive.ics.uci.edu/ml/datasets/Heart+Disease [Online Available].
[19] G. Hinton, S. Osindero, and Y. W. Teh. “A fast learning algorithm for deep belief nets. Neural Computation”, Vol.18, Issue.7,1527–1554, 2006.