RDQN: ensemble of deep neural network with reinforcement learning in classification based on rough set theory for digital transactional fraud detection

Abstract

Abstract All financial sectors are facing the most common frauds, which are digital transactional frauds. Fraudsters have always engaged in illegal activities such as stealing personal information and logging in with unauthorised credentials. Many machine learning algorithms predict whether the transaction is factual or nonfactual but fail to decrease the processing time. Hybrid models are used in this case to identify the fraud in a quick and efficient manner. This article demarcates to construct a novel model, RDQN, i.e., deep reinforcement learning, that combines with the rough set theory. This article has three steps, including data pre-processing to determine the quality of the data, which affects the learning ability of the model, determining the structural relationship and gaining useful features from the data set using rough set theory, and doing a hybridization of DNN (deep neural network) and Q learning, which is called DQN. It uses the MISH activation function and the ReLU activation function in different layers for training dynamics in the neural network. The proposed model classifies and predicts that the transaction belongs to the category implemented by the agents by activating the reward function. The reinforcement-learning agent’s performance improves based on reward assessment. This reward function gives a more precise value for each transaction, and no fraudster can escape from the agent’s sight. This novel approach improves accuracy and reduces processing time by considering the best feature selection during the process.

Keywords

• Digital transactions
• Rough set theory
• Deep neural network
• Deep reinforcement learning (DQN)
• Reward function
• Agent