Do You Regret Your Predictions?

Steve Collins

I’ve talked in previous blog posts about our predictive suite, which enables you to take advantage of what you know about previous users to predict what new users might do in your app.

That’s an extremely powerful tool for helping to target your campaigns at just the right set of users (e.g. those who look like they’re just about to convert into a paying customer, or maybe those that are looking at risk of disengaging from your app).

But how do you put this into practise in your marketing strategy? In reality, with a large and diverse user base, you will have many different signals and contexts you’ll be using to try and decide what the next best action is for those users.

As a concrete example of this, let’s take Churn and Spend prediction.  With Churn prediction we’re identifying the users who are likely to stop using the app in the next few days or weeks.  Spend prediction, on the other hand, identifies those users who are likely to do something of value in the app, like purchase an in-app item or upgrade to a paid subscription.

Let’s look at a graph representation of this. Imagine we’ve run our analysis on the users in an app and have a propensity to churn score for each one.  In the diagram below we graph a typical set of users, with the x-axis representing a sorted list of the users (sorted by probability to churn) and the y-axis giving the propensity (a number from 0 to 1, where 0 means they are definitely not going to churn and 1 means they definitely are).

At this point you’re probably going to decide on a range of churn propensities, and target users above this range with a marketing action designed to encourage them to stay. In our example we’re going to offer a discount on a subscription to these users hoping that the discount will tip the balance in favour of them becoming a loyal subscriber.

In practise, we’ll group our users into 3 distinct categories:

• high propensity to churn: these users are very likely to churn regardless of what we do, so there’s little value in targeting them, particularly if there’s a cost associated with attempting to retain them as a user.
  
• low propensity to churn: these users are unlikely to churn, so there’s no need to target them with an anti-churn measure.
  
• medium propensity to churn: this is our target set of users.  There’s a good possibility of these users churning, but they are not lost causes and it’s worth making an effort to retain them.

Deciding what thresholds to use for each of these categories of user is tricky.  Let’s make life a little more interesting first though.

Simultaneously we’re using a model of user propensity to purchase, so now for each user we have the probability of them churning and the probability of them purchasing. We can graph both of these on the same chart as follows.

We can see that users likely to churn are unlikely to purchase and vice versa. In general there’s a negative correlation between the 2 groups – we’ve simplified it for this example, and you wouldn’t expect to see such neat curves, but the basic pattern is sound.

The group we’d like to target with a discount on the subscription are those who are unlikely to purchase the subscription anyway and who are also not lost causes in that they are going to churn anyway as shown below:

You can see from the picture above that we will end up offering a discount to potential churners for whom there’s a non zero chance they will purchase any way (the area under the green curve in the “offer discount” zone).  This is our “regret” i.e. the cost to us of targeting these users.  For marketing campaigns like this we want to minimize this regret while maximising the return on getting potential churning users to subscribe.

Finding the balance that maximizes the value of the campaign requires fine grain campaign analytics and an experimentation framework that allows you to optimize your campaign.

(89)