Random Search Enhancement of Incremental Regularized Multiple Hidden Layers ELM

Abstract

The extreme learning machine (ELM) represents one of the most successful approaches in the field of machine learning recently, especially in solving classification and regression problems. A key advantage of the multiple hidden layers' ELM (MELM) is that the computational time required to train the neural network is significantly lower because it uses random selection and analytical solution, respectively, to determine the weights of the hidden nodes and output nodes. However, due to the use of too many or too few hidden nodes during the training process, the phenomenon of over-fitting or under-fitting may occur in the prediction process. Aiming at the design of MELM neural network architecture, this paper applies the enhanced random search method to the MELM network model and proposes an incremental MELM training algorithm based on the Cholesky decomposition, namely, random search enhancement of incremental regularized MELM (EIR-MELM). The algorithm automatically determines the optimal MELM network structure by increasing the hidden nodes one by one and calculates the output weights by flexibly adopting the Cholesky decomposition method, which effectively reduces the computation burden caused by the incremental process of the hidden layer neurons. However, some hidden nodes added to the network may only have a weak influence on the final output of the network. Adding randomly generated nodes directly to the network only increases the complexity of the neural network structure. Therefore, in the process of adding hidden nodes, EIR-MELM adds a selection phase. According to the principle of structural risk minimization, the optimal node is selected from multiple randomly generated nodes to be added to the network, so that EIR-MELM has a more compact network structure. The experimental researches on the benchmark datasets for classification problems show that EIR-MELM can effectively determine the optimal MELM network structure automatically with high calculation efficiency.

Keywords

• Extreme learning machine
• multiple hidden layers
• incremental learning procedures
• Cholesky decomposition
• random search enhancement