Real-Time Bidding with Soft Actor-Critic Reinforcement Learning in Display Advertising

Abstract

The main task of advertising companies is to sell goods and services interesting to the user. Online auctions are the main mechanism for selecting ads to the user. Dynamic bidding allows advertiser to automatically calculate the bid that is profitable to set to maximize goals (for example, the number of clicks on an ad), depending on the user who sees the ad. In this case the advertiser must specify the budget of the ad and the optimization goal. During the advertising campaign the bid for each impression will be calculated by a special algorithm. In this paper, we propose a novel algorithm for calculating the dynamic bid for each impression of the ad in order to maximize the advertiser’s goals, which takes into account settings of the advertising campaign, budget, the ad lifetime and other parameters. This task is formulated as reinforcement learning problem, where states are the status of auction and parameters of the advertising campaign, the actions are bidding for each ad based on the input state. Every ad has an agent who observes the states all the time and calculates the bid for the impression. We evaluated the proposed model on real advertising campaigns in a large social network. Our method achieved average 26% improvement in comparison with the state-of-the-art approach.

Keywords

• display advertising
• reinforcement learning
• social networks