Adjusting your threshold of a classification probabilities can increase the score of your model.
In a classification problem, the trained model predict classes. It can be a binary classification as « yes » or « no » or a multiple labels classification as the MINST dataset where the goal is to classify image to a label from 0 to 9 [0..9].
With Kaggle competition it is extremely easy to evaluate a model. To do that, simply register to the website and participate to the competition by submitting your predictions. I choose the Titanic dataset and competition to play with the threshold of classification problem with a binary issue.
Titanic problem
Titanic problem and dataset is a fascinating problem. You get data from Titanic passengers. The dataset is splatted in 2 files. The training data with 891 passenger and the status, survived or not and the test dataset with 418 passengers for whom your model must predict the status (survived or not).
You have to train a model with training data and making prediction on test dataset to submit your predictions to Kaggle and immediately get the score.
Train a model
For this article, I did not too much invest in EDA (Exploratory Data Analysis) and just took the data, made some transformation and prepared the dataset for training process.
I created a simple Neural Network model to train on data and make predictions.
When training a classification model, the prediction can return the class (‘0’ or ‘1’) and the calculated probability. Let’s say the model return ‘1’ for an observation, the probability can be {1:0,65, 0:0.35}. This means the ‘1’ has been decided because the probability (native) is over 0.5. But we can adjust the threshold, by telling the prediction to be ‘1’ if the probability is over 0.7. In that case, the prediction would be ‘0’.
It is by playing with the threshold that I can change the score of the model on unseen data on Kaggle competition website.
The dataset
Cleaning data and creating feature is part of a machine learning project. I simply replaced non existing value with the mean() of the dataset and convert missing text value by creating a “Deck” feature.
train = pd.read_csv('data/train.csv')
test = pd.read_csv('data/test.csv')
#create a full dataset
data=pd.concat([train,test], axis=0)
data.Age.fillna(data.Age.mean(), inplace=True)
data.Cabin.fillna('U',inplace=True)
data.Fare.fillna(data.Fare.mean(),inplace=True)
data.Embarked.fillna('S',inplace=True)
data['Deck'] = data['Cabin'].str.slice(0,1)
data['Sex']=data['Sex'].replace({'male':0, 'female':1})
data.Sex.astype(int)
data.isnull().sum()
Split the data, create X & y and scale my data.
from sklearn.model_selection import train_test_split
X = train_set.drop('Survived', axis=1)
y = train_set['Survived']X_train, X_test, y_train, y_test = train_test_split(
X, y,
test_size=0.2, random_state=42)from sklearn.preprocessing import StandardScalerscaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
Neural Network model
Create a simple NN model to train with dataset.
model = tf.keras.Sequential([
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dense(256, activation='relu'),
tf.keras.layers.Dense(256, activation='relu'),
tf.keras.layers.Dense(1, activation='sigmoid')])model.compile(
loss=tf.keras.losses.binary_crossentropy,
optimizer=tf.keras.optimizers.Adam(lr=0.03),
metrics=[
tf.keras.metrics.BinaryAccuracy(name='accuracy'),
tf.keras.metrics.Precision(name='precision'),
tf.keras.metrics.Recall(name='recall')])history = model.fit(X_train_scaled, y_train, epochs=100)
Make predictions on unseen data, those to be submitted to Kaggle.
unseen = scaler.transform(test_set)
predicted_unseen = model.predict(unseen)Play with “threshold”
This is the key of this post, setting a threshold and computing predictions. In the next case, I defined the threshold to 0.685 (random choice) and preparing data for competition submission.
prediction_unseen_classes = [1 if prob > 0.685 else 0 for prob in np.ravel(predicted_unseen)]test['Survived']=prediction_unseen_classes
results_df=test[['PassengerId','Survived']].copy()results_df.to_csv("Predictions", index=False)
This result made me a score of 0.75%
Let’s adjust the threshold to a low value and check result.
prediction_unseen_classes = [1 if prob > 0.185 else 0 for prob in np.ravel(predicted_unseen)]In that case, the score is 10% less, with a 65% score.
Summary
- always think of your threshold in a classification problem by calculating prediction probabilty (predict_proba) and setting the value, this can change the accuracy of your prediction