When online advertisements are displayed, a real-time bidding system is set up to choose the advertiser. On the advertiser's side, based on summary information (mostly categorical), the goal is to predict whether the user will click on the display or not (and thus choose whether to bid). Clicks being rare (1 per 1000 on average), there is a strong imbalance between the two classes (click and no-click). In this context, the usual prediction models are biased, since they do not take into account this imbalance. The usual evaluation measures, such as AUCROC , are also flawed when applied to these rare events. In this paper, we study these biases in the context of real-time auctions and propose a new evaluation measure. We focus our analysis on click prediction (pCT R), considering both linear (logistic regression) and non-linear (Deep Factorization Machine) prediction algorithms. We evaluate the performance of the models using a specific probabilistic evaluation function including the costs associated with the auction, and compare it to several classical evaluation measures. This measure highlights the limitations of classical metrics in highly unbalanced prediction problems. It thus allows for a better evaluation of specific click prediction models and also provides insights into the profitability of the display advertising campaign.