Variable Selection of Lasso and Large Model

Abstract

In order to clarify the variable selection of Lasso, Lasso is compared with two other variable selection methods AIC and forward stagewise. First, the variable selection of Lasso was compared with that of AIC, and it was discovered that Lasso has a wider application range than AIC. The data simulation shows the variable selection of Lasso under orthonormal design is consistent with AIC, Lasso under orthonormal design can be solved by using the stepwise selection algorithm. The removed variables of Lasso appear again under nonorthonormal design, the variable selection of Lasso under nonorthonormal design isn’t consistent with AIC. We continue to compare the variable selection of Lasso and forward stagewise. Based on the analysis of these studies, it is pointed out that the variable selection of Lasso is complex. An infinite number of parameters enable the design matrix to achieve orthonormalization, so that the solution of Lasso can be found with the stepwise selection algorithm, which may be the reason for the success of the large model represented by ChatGPT.

Keywords

• Variable selection
• Lasso
• AIC
• forward stagewise
• complexity
• ChatGPT