Predicting Bank Users’ Time Deposits Based on LSTM-Stacked Modeling

In banking and finance, customer behavior prediction is a key research area. This predictive capability can help banks and financial institutions optimize marketing strategies, improve customer retention and increase product sales. For example, predicting whether a customer will choose a time deposit is a valuable task because time deposits are not only a stable source of funds for banks but also promote cross-selling of other financial products. Therefore, the development of efficient prediction models, especially models that can accurately predict whether customers will choose time deposits, is of great practical importance to the banking industry.

Traditional predictive models rely heavily on machine learning algorithms such as LR, Decision Trees (DTs), RFs, SVMs, etc. These methods perform well when dealing with simple structured data, but are usually ineffective when dealing with high-dimensional, complex data that contain nonlinear relationships. This is because these traditional models rely heavily on feature engineering, i.e., they require expert knowledge to select and extract features. In addition, these models often struggle to capture long-term dependencies in time-series data. Behavioral data of bank customers are usually time-dependent, and such dependencies reflect changes in customers' behaviors at different points in time. As a result, models based on traditional machine learning methods may struggle to fully utilize this temporal information, leading to limited predictive performance.

In recent years, deep learning methods, especially LSTM networks, have received much attention due to their advantages in processing time-series data and long dependencies. LSTM is a special kind of Recurrent Neural Network (RNN) that can efficiently capture long- and short-term dependencies in sequential data through its gating mechanism. Compared to traditional machine learning methods, LSTM models can automatically learn and extract complex features from data without explicit feature engineering. This makes LSTM perform well in many application areas, including natural language processing, speech recognition, and financial time series prediction.

However, a single LSTM model has some limitations in practical applications. Although LSTM performs well in feature extraction, its predictive performance may still be suboptimal in some cases. This is because the LSTM model may be overfitted to the training data, especially when the amount of data is relatively small or the complexity of data features is high. Therefore, in order to further improve the prediction performance, a strategy that combines LSTM feature extraction with integrated learning methods was proposed in this study. Integration learning is a technique that improves the model generalization ability by combining the prediction results from multiple base learners. Among them, stacking ensemble is a common integration learning method that learns how to best combine the outputs of base learners by training a metamodel.

In this study, a combined LSTM-stacked model-based approach was proposed for the prediction of bank users' time deposits. Features from two different bank marketing datasets were first extracted using LSTM models, and then these features were input into multiple base classifiers (RF, SVM, and KNN) for classification. Finally, the outputs of the base classifiers were input into a LR model for stacked integration.

The next sections detail the related work, datasets, model architecture, results and discussion, and final conclusions of this study. This research is expected to provide a more accurate and efficient solution for the banking industry in customer behavior prediction.

LSTM models have been extensively utilized in financial time series forecasting due to their superior ability to capture time-dependent features. For instance, Ala'raj et al. [1], [2] demonstrated that LSTMs can effectively manage complex time-dependent features and excel in modeling credit card customer behavior. Chen et al. [3] introduced a hybrid model that integrates K-means clustering with LSTMs for the forecasting of stock prices for commercial banks in China, achieving a significant improvement in forecasting accuracy. Additional studies have explored the synergy of LSTMs with other methodologies. For example, De et al. [4] merged nonlinear Lasso with LSTM to estimate the interconnectivity of tail risks, whereas Dong and Zhou [5] developed a financial forecasting model that combines CEEMDAN and CNN-LSTM, which excels in capturing complex patterns in financial time series.

In recent years, integrated learning methods have garnered significant attention in the financial sector. These methods, such as Stacking, substantially enhance the model's generalization ability by leveraging multiple base learners. Fan et al. [6] effectively combined LSTM with XGBoost for bank consumer credit assessment, showcasing the robustness of the method. Fang et al. [7] proposed an intelligent public affairs allocation model that integrates LSTM with a multilayer perceptron (MLP), significantly enhancing the model's predictive performance through integrated learning. Some researchers [8], [9] employed a combination of EEM and MLP in predicting bank stock, demonstrating the benefits of integrated learning in processing financial data.

Other integrated learning variants such as Bagging and Boosting have also been widely adopted in predicting bank customer behavior. For instance, Gao et al. [8] utilized a combination of XGBoost and LSTM to predict credit card default behavior, achieving superior performance compared to single-model approaches. Similar studies include a hybrid model for an intelligent stock trading system developed by Eskandari et al. [10], which effectively improves prediction accuracy and stability by integrating multiple base classifiers [11], [12], [13].

Beyond LSTM and integrated learning, the Transformer architecture and its variants have also shown potent capabilities in handling financial time series data. Notably, the attention mechanism of Transformers can effectively capture long-term dependencies, as demonstrated by Peng and Guo [14] and Li et al. [15] in enhancing long time-series predictions, particularly in stock market forecasting [16], [17], [18], [19].

Furthermore, Heryadi and Warnars [12] developed a model combining CNN and LSTM for detecting fraudulent behaviors in transactions, showcasing the potential of deep learning in complex behavior detection. Hou et al. [11] explored the application of deep learning in rural finance, combining multiple deep learning models to enhance financial development prediction in rural governance [20], [21].

To address the nonlinearity and complexity in financial forecasting, researchers have proposed numerous hybrid models. For example, Li et al. [15] developed a model that combines LSTM with Support Vector Machine (SVM) for predicting bank loan delinquency, demonstrating the hybrid models' advantage in handling complex financial data [22], [23]. Dong and Zhou [5] further optimized financial time-series data prediction by integrating CEEMDAN with CNN-LSTM, significantly enhancing the model's ability to capture nonlinear features [24], [25], [26].

Several studies have indicated substantial improvements in predicting bank customer behavior through the use of hybrid models that blend deep learning and integrated learning techniques. For instance, Wu et al. [27] proposed a stock price prediction algorithm using a graph convolutional network (GCN) combined with LSTM, proving the efficacy of integrating deep learning with integrated learning methods. Liu et al. [19] optimized the prediction of an Internet money portfolio by integrating LSTM with the La-VaR method [27], [28]. Additionally, Sun et al. [26] demonstrated the advantages of the SVD-LSTM model in short-term stock price forecasting.

In summary, hybrid models that combine deep learning (especially LSTM) with integrated learning methods (e.g., Stacking, Boosting) have achieved significant results in the field of bank customer behavior prediction in recent years. These studies confirm that the integration of deep learning and integrated learning methods can effectively enhance prediction accuracy, capture complex patterns and time-dependent features in the data, and offer a more effective solution for predicting bank customer behavior.

This study uses two bank marketing related datasets that record information about banks' interactions with customers in different marketing campaigns.

The first bank marketing dataset (https://www.kaggle.com/datasets/janiobachmann/bank-marketing-dataset) from the Kaggle website contains 17 feature fields and records 11,162 customer records, and the main information is shown in Table 1.

Figure 1 and Figure 2 show the data distribution by type and the numeric data distribution of the first bank marketing dataset.

The second bank marketing dataset from the Kaggle website (https://www.kaggle.com/datasets/henriqueyamahata/bank-marketing) contains 21 feature fields and records 41,188 customer records, with key information shown in Table 2.

Figure 3 and Figure 4 show the data distribution by subtypes and the numeric data distribution of the second bank marketing dataset.

In this study, the missing values of the data were handled as follows: columns with less than 5% of missing values were filled with the mean or median; columns with more than 30% of missing values were directly deleted to avoid affecting the model training; and samples with more than 20% of missing values were also deleted. For category features with sequential relationships (e.g., education level), label coding was used to convert them to integer values; for unordered category features (e.g., job type, marital status), solo thermal coding was used to convert each category into independent binary features to avoid incorrect assumptions about inter-category relationships. Standardization was used to adjust the mean of each feature to 0 and the standard deviation to 1 to improve the convergence speed and stability of the model. The standardization formula is as follows:

The dataset was divided into a training set and a test set, with 80% of the data used to train the LSTM model and base classifiers, and 20% of the data used to evaluate the generalization ability of the model.

The design steps for the LSTM-stacked model are as follows:

Step 1: Read the dataset.

Step 2: Divide the training and test sets.

Step 3: Input the training set into the LSTM method.

Step 4: Input the extracted features from the LSTM model into the stacked model.

Step 5: Predict the test set and output the probabilities.

Step 6: If the probability of 0 is greater than 0.5, output 0; otherwise, output 1.

LSTM is a special kind of RNN that excels in processing time-series data and capturing long- and short-term dependencies. LSTM effectively solves the gradient vanishing and exploding problems of traditional RNNs when trained on long sequences through its gating mechanisms (input gate, forgetting gate, and output gate), which enables LSTM to memorize long-term information.

LSTM extracts and filters information through three gate mechanisms. The $\sigma$ symbols labeled in yellow in Figure 5 indicate the neural network layers of these three gates, which usually adopt the sigmoid ($\sigma$) activation function, and their output ranges from 0 to 1. The horizontal line above the blue module in the figure indicates the information transfer of the cell state at the previous moment, $C_{t-1}$. The state is calculated by the output of the first gate (oblivion gate) with the current input information, which determines the amount of information to be retained or discarded in the cell state. The mathematical expression for the oblivion gate is as follows:

The second gate is the input gate, which consists of two parts: a neural network layer using a sigmoid function, and a layer using a tanh activation function to generate candidate cell state values $\widetilde{C}_t$. The input gate is a neural network layer that uses a sigmoid function. Eventually, the input gate is passed through the $i_t$ and $\widetilde{C_t}$ product of the cell state to update the information in the cell state. The formula for the input gate is as follows:

Finally, the output gate is a neural network layer with a sigmoid activation function that determines which information in the current cell state can be used as an output value by performing a dot-multiplication operation with the tanh ($C_t$) to decide which information in the current cell state can be used as the output value. The formula for the output gate is as follows:

Stacking is an integrated learning method that learns how to optimally combine the outputs of multiple base classifiers by training a metamodel. In this study, the stacked integration model was trained by using LR as a metamodel and the predictions from RF, SVM and KNN as input features. Each row of the final stacked feature matrix contains predictions from the three base classifiers. The LR model generates the final prediction results by learning the relationship between the base classifiers' outputs and the real labels. The specific training process is shown in Figure 6.

After features were extracted from the LSTM model, these features were fed into several traditional machine learning models for initial classification prediction, including RFs, SVMs, and KNNs.

RF is an integrated learning method for classification by constructing multiple DTs and voting or averaging them. Its main parameters include the number of DTs and the maximum depth of each tree. Based on the LSTM features, the RF model further improves the classification accuracy.

SVM is an algorithm commonly used for classification tasks and is particularly suitable for dealing with high dimensional feature data. SVM performs classification by constructing hyperplanes that maximize the spacing between categories. In this study, an SVM model with a Radial Basis Function (RBF) kernel was used to deal with nonlinearly differentiable data. An important parameter of SVM is $C$, which controls the size of the soft interval of the decision boundary. In this study, the $C$ values were tuned by cross-validation.

The KNN algorithm is an instance-based classification algorithm that performs classification by calculating the distance between the test and training samples. The main parameter of KNN is the $k$ value, i.e., the number of neighboring samples selected for voting.

This section analyzes in detail the experimental results of the bank users' time deposit prediction using the LSTM and the stacked integrated model, and validates the performance of different models by combining the metrics of accuracy, precision, recall and F1-score. The plain Bayes, KNN, DT and LSTM-stacked models were trained and tested on two datasets, respectively. Table 3 and Table 4 show the performance metrics of different models on the two datasets, including accuracy, precision, recall and F1-score.

From the overall results in Table 3 and Table 4, the LSTM-stacked model performs very well on both datasets, demonstrating its strong adaptability in different data environments. In dataset 1, the accuracy of the LSTM-stacked model reaches 81.77%, which is a clear advantage over other models; while in dataset 2, the accuracy of the model reaches 91.72%, which is also better than most of the comparison models.

The performance of the LSTM-stacked model is balanced and stable in terms of precision and recall. In dataset 1, the precision is 82.14% and the recall is 81.77%. In dataset 2, although the precision is slightly lower than that of the plain Bayes and SVM, the recall and F1-score remain high at 91.72% and 90.63%, respectively. These results show that the LSTM-stacked model can exhibit strong feature extraction and classification capabilities in datasets with different feature complexities.

The F1-score is a comprehensive measure of precision and recall. The LSTM-stacked model consistently maintains a high degree of equilibrium in both datasets, proving its strong robustness in dealing with unbalanced data and complex scenarios. Therefore, the LSTM-stacked model shows superior performance in dealing with the task of predicting bank users' time deposits, which is suitable for efficient prediction needs in practical applications.

In this study, a method predicting bank customers’ time deposits was proposed based on the LSTM neural network and the stacked integration model, which is an innovative solution to the limitations of traditional machine learning methods in dealing with time series data and high-dimensional nonlinear relationships. Experimental results show that the LSTM-stacked model significantly improves the accuracy of bank customers' time deposit prediction. On two different datasets, the LSTM-stacked model outperforms the traditional method with test accuracies of 81.77% and 91.72%, respectively. This approach not only achieves excellent results in the task of time deposit prediction but also provides banks with ideas that can be used in other prediction tasks (e.g., loan risk assessment, user behavior prediction, etc.). Although the model in this study performs well in experiments, there is still room for further improvement in future research. First, the introduction of more deep learning architectures, such as Transformer or attention mechanisms, could be considered to enhance the capability of feature extraction. Secondly, the selection and optimization of integrated learning methods could be further explored, and more different types of base learners could be combined or different meta-learners could be explored to further improve the prediction performance of the model.