💳Credit Card Fraud Detection : Dealing with Class Imbalance
The Problem
Credit card fraud detection is a binary classification problem with a defining challenge: fraud is rare. In this dataset, only 0.17% of transactions are fraudulent, meaning a model that predicts "legitimate" for every transaction would be 99.8% accurate while catching zero fraud. This makes standard accuracy meaningless here, and requires evaluating the model using metrics that specifically measure performance on the minority (fraud) class.
Key Definitions
Goal
Minimize FNs (missed fraud) as much as possible while maintaining strong Precision. Missed fraud causes direct financial loss, but excessive false alarms overwhelm a fraud review team and erode trust in the system, so both failure modes carry a real cost.
We want the best possible recall you can get, subject to precision staying high enough that the alerts remain usable.
Approach
This notebook evaluates multiple strategies for handling severe class imbalance, resampling techniques, cost-sensitive learning, and algorithm choice, before selecting a final model through systematic comparison on a held-out validation set, with performance confirmed on a separate test set.
Data Source : https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud
import kagglehub
# Download latest version
path = kagglehub.dataset_download("mlg-ulb/creditcardfraud")
print("Path to dataset files:", path)
Downloading to /root/.cache/kagglehub/datasets/mlg-ulb/creditcardfraud/3.archive...
100%|██████████| 66.0M/66.0M [00:00<00:00, 81.3MB/s]
Extracting files...
Path to dataset files: /root/.cache/kagglehub/datasets/mlg-ulb/creditcardfraud/versions/3
#import xgboost
#print(xgboost.__version__)
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, classification_report
from imblearn.over_sampling import SMOTE, RandomOverSampler, ADASYN
from imblearn.under_sampling import RandomUnderSampler, EditedNearestNeighbours
from sklearn.preprocessing import StandardScaler
from xgboost import XGBClassifier
# Ignore warnings
import warnings
warnings.filterwarnings('ignore')
data = pd.read_csv(path + "/creditcard.csv")
data.head()
| Time | V1 | V2 | V3 | V4 | V5 | V6 | V7 | V8 | V9 | ... | V21 | V22 | V23 | V24 | V25 | V26 | V27 | V28 | Amount | Class | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0.0 | -1.359807 | -0.072781 | 2.536347 | 1.378155 | -0.338321 | 0.462388 | 0.239599 | 0.098698 | 0.363787 | ... | -0.018307 | 0.277838 | -0.110474 | 0.066928 | 0.128539 | -0.189115 | 0.133558 | -0.021053 | 149.62 | 0 |
| 1 | 0.0 | 1.191857 | 0.266151 | 0.166480 | 0.448154 | 0.060018 | -0.082361 | -0.078803 | 0.085102 | -0.255425 | ... | -0.225775 | -0.638672 | 0.101288 | -0.339846 | 0.167170 | 0.125895 | -0.008983 | 0.014724 | 2.69 | 0 |
| 2 | 1.0 | -1.358354 | -1.340163 | 1.773209 | 0.379780 | -0.503198 | 1.800499 | 0.791461 | 0.247676 | -1.514654 | ... | 0.247998 | 0.771679 | 0.909412 | -0.689281 | -0.327642 | -0.139097 | -0.055353 | -0.059752 | 378.66 | 0 |
| 3 | 1.0 | -0.966272 | -0.185226 | 1.792993 | -0.863291 | -0.010309 | 1.247203 | 0.237609 | 0.377436 | -1.387024 | ... | -0.108300 | 0.005274 | -0.190321 | -1.175575 | 0.647376 | -0.221929 | 0.062723 | 0.061458 | 123.50 | 0 |
| 4 | 2.0 | -1.158233 | 0.877737 | 1.548718 | 0.403034 | -0.407193 | 0.095921 | 0.592941 | -0.270533 | 0.817739 | ... | -0.009431 | 0.798278 | -0.137458 | 0.141267 | -0.206010 | 0.502292 | 0.219422 | 0.215153 | 69.99 | 0 |
5 rows × 31 columns
len(data)
284807
# Check for duplicates
print(f'There are {data.duplicated().sum()} duplicates.')
# Remove duplicates
data.drop_duplicates(inplace=True)
print('All duplicates have been removed.')
There are 1081 duplicates. All duplicates have been removed.
# Check for missing values
print(f'Missing values present in dataset: {data.isna().any().any()}')
Missing values present in dataset: False
# Change all column names to lowercase
data.columns = data.columns.str.lower()
data.columns
Index(['time', 'v1', 'v2', 'v3', 'v4', 'v5', 'v6', 'v7', 'v8', 'v9', 'v10',
'v11', 'v12', 'v13', 'v14', 'v15', 'v16', 'v17', 'v18', 'v19', 'v20',
'v21', 'v22', 'v23', 'v24', 'v25', 'v26', 'v27', 'v28', 'amount',
'class'],
dtype='object')
# Drop unneccessary columns
data.drop(['time'], axis=1, inplace=True)
df = data.copy()
df['class'].value_counts().reset_index()
| class | count | |
|---|---|---|
| 0 | 0 | 283253 |
| 1 | 1 | 473 |
## Create a pie plot to show the distribution of the class label
plt.figure(figsize=(6,4))
plt.pie(df['class'].value_counts(), labels=['Legitimate', 'Fraud' ], autopct='%1.1f%%', colors=['green', 'red'])
plt.title('Distribution of Class Label')
plt.show()
Observations 🔍
Because of the substantial class imbalance, we will explore resampling methods to address it.
# Define X and y
X = df.drop('class', axis=1)
y = df['class']
# Train_test_split (80% train, 20% test)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
# Validation_split (80% train, 20% validation)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=42, stratify=y_train)
# Feature scaling
# Instantiate a scaler object
scaler = StandardScaler()
# Normalize the training and test feature sets to improve model performance
X_train_scaled = scaler.fit_transform(X_train)
X_val_scaled = scaler.transform(X_val)
X_test_scaled = scaler.transform(X_test)
# Evaluation function
from sklearn.metrics import precision_score, recall_score, f1_score
def build_and_evaluate_model(model, train_X, train_y, test_X, test_y, technique, threshold=None):
model.fit(train_X, train_y)
if threshold == None:
# Make predictions
y_pred = model.predict(test_X)
else:
y_pred_proba = model.predict_proba(test_X)[:, 1] # Get fraud class probability
y_pred = (y_pred_proba >= threshold).astype(int)
# Create classification report
print(f'Classification Report: {technique} \n')
print(classification_report(test_y, y_pred, target_names=['legit', 'fraud']))
# Create confusion matrix
cm = confusion_matrix(test_y, y_pred)
plt.figure(figsize=(3,3))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', xticklabels=['Legit', 'Fraud'], yticklabels=['Legit', 'Fraud'],cbar=False) # format=decimal
plt.xlabel('Predicted_Labels'),
plt.ylabel('Actual_Labels')
plt.xticks(fontsize=10)
plt.yticks(fontsize=10)
plt.title('Confusion Matrix')
plt.show()
# Return metrics for each model
return {
'Model': technique,
'Precision': round(precision_score(test_y, y_pred, pos_label=1), 3),
'Recall': round(recall_score(test_y, y_pred, pos_label=1), 3),
'F1 Score': round(f1_score(test_y, y_pred, pos_label=1),3)
}
results_list = []
logit= LogisticRegression(max_iter=1000, random_state=42)
results_list.append(build_and_evaluate_model(logit, X_train_scaled, y_train, X_val_scaled, y_val, 'Baseline'))
Classification Report: Baseline
precision recall f1-score support
legit 1.00 1.00 1.00 45320
fraud 0.91 0.68 0.78 76
accuracy 1.00 45396
macro avg 0.96 0.84 0.89 45396
weighted avg 1.00 1.00 1.00 45396
🔍 Baseline Observations
Precision 91%, Recall 68%, F1 0.78. The model is conservative: when it flags a transaction as fraud, it's usually right,we have very few false alarms.but it misses roughly a third of actual fraud cases. This is a reasonable starting point, but the missed-fraud rate is too high for real deployment.
Recall our goal is to minimize FNs (missed fraud) as much as possible without letting Precision collapse, since a high volume of false alarms would overwhelm a fraud review team and erode trust in the system, while missed fraud leads directly to financial loss.
ros = RandomOverSampler(random_state=42)
# Resample training data
X_resampled_ros, y_resampled_ros = ros.fit_resample(X_train_scaled, y_train)
## Create a pie plot to show the distribution of the class label
plt.figure(figsize=(6,4))
plt.pie(y_resampled_ros.value_counts(), labels=['Legitimate', 'Fraud' ], autopct='%1.1f%%', colors=['green', 'red'])
plt.title('Class Distribution After RandomOverSampler')
plt.show()
results_list.append(build_and_evaluate_model(logit, X_resampled_ros, y_resampled_ros, X_val_scaled, y_val, 'RandomOverSampler'))
Classification Report: RandomOverSampler
precision recall f1-score support
legit 1.00 0.97 0.99 45320
fraud 0.05 0.89 0.10 76
accuracy 0.97 45396
macro avg 0.53 0.93 0.54 45396
weighted avg 1.00 0.97 0.98 45396
🔍 RandomOverSampler Observations
Precision dropped sharply to 5%, while Recall rose to 89%. The model now catches nearly 9 in 10 fraud cases, but at the cost of flooding the output with false alarms; only 1 in 20 fraud flags is genuinely fraud. F1 fell to 0.10, showing this trade isn't worth it as-is.
smote = SMOTE(random_state=42)
# Resample training data
X_resampled_sm, y_resampled_sm = smote.fit_resample(X_train_scaled, y_train)
# Distribution of class labels
y_resampled_sm.value_counts().reset_index()
| class | count | |
|---|---|---|
| 0 | 0 | 181282 |
| 1 | 1 | 181282 |
results_list.append(build_and_evaluate_model(logit, X_resampled_sm, y_resampled_sm, X_val_scaled, y_val, 'SMOTE'))
Classification Report: SMOTE
precision recall f1-score support
legit 1.00 0.97 0.99 45320
fraud 0.05 0.88 0.09 76
accuracy 0.97 45396
macro avg 0.52 0.93 0.54 45396
weighted avg 1.00 0.97 0.98 45396
🔍 SMOTE Observations
Nearly identical to RandomOverSampler: Precision 5%, Recall 88%, F1 0.09. Generating synthetic fraud examples didn't meaningfully change the outcome compared to simple duplication in this case.
adasyn = ADASYN(random_state=42)
# Resample training data
X_resampled_adsn, y_resampled_adsn = adasyn.fit_resample(X_train_scaled, y_train)
y_resampled_adsn.value_counts().reset_index()
| class | count | |
|---|---|---|
| 0 | 0 | 181282 |
| 1 | 1 | 181280 |
results_list.append(build_and_evaluate_model(logit, X_resampled_adsn, y_resampled_adsn, X_val_scaled, y_val, 'ADASYN'))
Classification Report: ADASYN
precision recall f1-score support
legit 1.00 0.91 0.95 45320
fraud 0.02 0.91 0.03 76
accuracy 0.91 45396
macro avg 0.51 0.91 0.49 45396
weighted avg 1.00 0.91 0.95 45396
🔍 ADASYN Observations
Precision collapsed further to 2%, while Recall reached 91%, the highest of any resampling technique tested. Confusion matrix confirms the cost: 3976 false positives against only 7 false negatives. Catching almost all fraud isn't useful if it means investigating thousands of legitimate transactions for every real case.
undersample = RandomUnderSampler(random_state=42)
X_resampled_under, y_resampled_under = undersample.fit_resample(X_train_scaled, y_train)
# Class Distribution
y_resampled_under.value_counts().reset_index()
| class | count | |
|---|---|---|
| 0 | 0 | 302 |
| 1 | 1 | 302 |
We have a lot less data to train on.
results_list.append(build_and_evaluate_model(logit, X_resampled_under, y_resampled_under, X_val_scaled, y_val, 'RandomUnderSampler'))
Classification Report: RandomUnderSampler
precision recall f1-score support
legit 1.00 0.94 0.97 45320
fraud 0.03 0.91 0.05 76
accuracy 0.94 45396
macro avg 0.51 0.93 0.51 45396
weighted avg 1.00 0.94 0.97 45396
🔍RandomUnderSampler Observations
Precision 3%, Recall 91%, F1 0.05. Despite training on far less data (majority class cut down to match the minority), performance tracks closely with the oversampling methods: strong recall, unusable precision.
ENN removes samples that have a majority of their nearest neighbors belonging to a different class, effectively cleaning the decision boundary and reducing potential noise.
ENN = EditedNearestNeighbours()
X_resampled_ENN, y_resampled_ENN = ENN.fit_resample(X_train_scaled, y_train)
y_resampled_ENN.value_counts().reset_index()
| class | count | |
|---|---|---|
| 0 | 0 | 181173 |
| 1 | 1 | 302 |
Samples have been removed from the majority class, the minority class remain unchanged.
results_list.append(build_and_evaluate_model(logit, X_resampled_ENN, y_resampled_ENN, X_val_scaled, y_val, 'EditedNearestNeighbours'))
Classification Report: EditedNearestNeighbours
precision recall f1-score support
legit 1.00 1.00 1.00 45320
fraud 0.93 0.70 0.80 76
accuracy 1.00 45396
macro avg 0.96 0.85 0.90 45396
weighted avg 1.00 1.00 1.00 45396
🔍 ENN Observations
A clear step up: Precision 93%, Recall 70%, F1 0.80, the best F1 of any resampling method tried so far. Cleaning noisy majority-class samples near the decision boundary, rather than blindly duplicating or removing data, produced a far more balanced result.
(Algorithmic weighting instead of resampling)
Given that only 0.2% of samples are labeled as fraud, while 99.8% are legitimate transactions, using class weight adjustments is a good approach instead of oversampling or undersampling. This can help to prevent the model overfitting on artificial samples and also will stop us from having to remove valuable data. Additionally this method can help the model give more importance to our minority fraud class.
logit = LogisticRegression(class_weight='balanced', max_iter=1000, random_state=42)
results_list.append(build_and_evaluate_model(logit, X_train_scaled, y_train, X_val_scaled, y_val, 'Balanced Class Weights'))
Classification Report: Balanced Class Weights
precision recall f1-score support
legit 1.00 0.97 0.99 45320
fraud 0.05 0.89 0.10 76
accuracy 0.97 45396
macro avg 0.53 0.93 0.54 45396
weighted avg 1.00 0.97 0.98 45396
🔍 Balanced Class Weights Observations
Precision 5%, Recall 89%, F1 0.10, nearly identical to RandomOverSampler. Adjusting class weights instead of resampling didn't avoid the same precision collapse.
logit = LogisticRegression(class_weight={0:1, 1:1000}, max_iter=1000, random_state=42)
results_list.append(build_and_evaluate_model(logit, X_train_scaled, y_train, X_val_scaled, y_val, 'Custom Class Weights'))
Classification Report: Custom Class Weights
precision recall f1-score support
legit 1.00 0.96 0.98 45320
fraud 0.04 0.91 0.07 76
accuracy 0.96 45396
macro avg 0.52 0.93 0.52 45396
weighted avg 1.00 0.96 0.98 45396
🔍Custom Class Weights Observations
Precision 4%, Recall 91%, F1 0.07, slightly worse than the balanced-weight version. Manually pushing the fraud class weight higher increased recall marginally but degraded precision further, reinforcing that weight adjustment alone isn't sufficient here.
XGBoost is an optimized gradient boosting library that works exceptionally well for fraud detection. It provides scale_pos_weight, which adjusts the weight of the minority class.
fraud_weight = y_train.value_counts()[0] / y_train.value_counts()[1]
fraud_weight
np.float64(600.2715231788079)
xgb = XGBClassifier(n_estimators = 100, scale_pos_weight=fraud_weight, random_state=42, use_label_encoder=False,verbose=1, n_jobs=-1)
results_list.append(build_and_evaluate_model(xgb, X_train_scaled, y_train, X_val_scaled, y_val, 'XGBoost'))
Classification Report: XGBoost
precision recall f1-score support
legit 1.00 1.00 1.00 45320
fraud 0.94 0.86 0.90 76
accuracy 1.00 45396
macro avg 0.97 0.93 0.95 45396
weighted avg 1.00 1.00 1.00 45396
🔍XGBoost Observations
A significant jump: Precision 94%, Recall 86%, F1 0.90, the best result by a wide margin. Using scale_pos_weight instead of resampling let the model learn from the real data distribution while still accounting for the imbalance.
We can implement optuna to tune the hyperparameters of our XGBoost model.
# install optuna
!pip install optuna
Collecting optuna Downloading optuna-5.0.0-py3-none-any.whl.metadata (14 kB) Collecting alembic>=1.5.0 (from optuna) Downloading alembic-1.20.1-py3-none-any.whl.metadata (7.3 kB) Collecting colorlog (from optuna) Downloading colorlog-6.12.0-py3-none-any.whl.metadata (11 kB) Requirement already satisfied: numpy in /usr/local/lib/python3.13/dist-packages (from optuna) (2.1.3) Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.13/dist-packages (from optuna) (26.3) Requirement already satisfied: sqlalchemy>=1.4.2 in /usr/local/lib/python3.13/dist-packages (from optuna) (2.1.5) Requirement already satisfied: tqdm in /usr/local/lib/python3.13/dist-packages (from optuna) (4.67.3) Requirement already satisfied: PyYAML in /usr/local/lib/python3.13/dist-packages (from optuna) (6.0.3) Requirement already satisfied: Mako in /usr/lib/python3/dist-packages (from alembic>=1.5.0->optuna) (1.3.2.dev0) Requirement already satisfied: typing-extensions>=4.12 in /usr/local/lib/python3.13/dist-packages (from alembic>=1.5.0->optuna) (4.16.0) Downloading optuna-5.0.0-py3-none-any.whl (442 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 442.4/442.4 kB 7.2 MB/s eta 0:00:00 Downloading alembic-1.20.1-py3-none-any.whl (270 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 270.4/270.4 kB 27.7 MB/s eta 0:00:00 Downloading colorlog-6.12.0-py3-none-any.whl (12 kB) Installing collected packages: colorlog, alembic, optuna Successfully installed alembic-1.20.1 colorlog-6.12.0 optuna-5.0.0
import optuna
from sklearn.metrics import f1_score
# Define Optuna objective function
def objective(trial):
params = {
'n_estimators': trial.suggest_int('n_estimators', 100, 1000),
'max_depth': trial.suggest_int('max_depth', 3, 12),
'learning_rate': trial.suggest_loguniform('learning_rate', 0.01, 0.3),
'subsample': trial.suggest_uniform('subsample', 0.5, 1),
'colsample_bytree': trial.suggest_uniform('colsample_bytree', 0.5, 1),
'gamma': trial.suggest_uniform('gamma', 0, 5),
'min_child_weight': trial.suggest_int('min_child_weight', 1, 10),
}
# Train XGBoost model
model = XGBClassifier(**params, scale_pos_weight=fraud_weight, use_label_encoder=False, eval_metric='logloss', verbose=1, n_jobs=-1)
model.fit(X_train_scaled, y_train)
# Predict probabilities
y_pred_proba = model.predict_proba(X_val_scaled)[:, 1] # Get fraud class probability
# Convert probabilities to class labels using a low threshold (improves recall)
threshold = 0.5 # You can tune this further
y_pred = (y_pred_proba >= threshold).astype(int)
#---------------------------------------------------------------------
# Calculate recall
#recall = recall_score(y_test, y_pred)
#return recall
#---------------------------------------------------------------------
# Calculate F1-score
f1 = f1_score(y_val, y_pred)
return f1
# Run Optuna optimization
study = optuna.create_study(direction='maximize', sampler=optuna.samplers.TPESampler(seed=42)) # Maximise f1-score for fraud
study.optimize(objective, n_trials=50)
[I 2026-10-22 13:21:13,692] A new study created in memory with name: no-name-b4c9f44d-c272-413a-8ef7-c503847b8413
[I 2026-10-22 13:21:31,448] Trial 0 finished with value: 0.9090909090909091 and parameters: {'n_estimators': 437, 'max_depth': 12, 'learning_rate': 0.1205712628744377, 'subsample': 0.7993292420985183, 'colsample_bytree': 0.5780093202212182, 'gamma': 0.7799726016810132, 'min_child_weight': 1}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:21:45,124] Trial 1 finished with value: 0.9027777777777778 and parameters: {'n_estimators': 880, 'max_depth': 9, 'learning_rate': 0.11114989443094977, 'subsample': 0.5102922471479012, 'colsample_bytree': 0.9849549260809971, 'gamma': 4.162213204002109, 'min_child_weight': 3}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:21:49,305] Trial 2 finished with value: 0.5075757575757576 and parameters: {'n_estimators': 263, 'max_depth': 4, 'learning_rate': 0.028145092716060652, 'subsample': 0.762378215816119, 'colsample_bytree': 0.7159725093210578, 'gamma': 1.4561457009902097, 'min_child_weight': 7}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:21:54,623] Trial 3 finished with value: 0.7243243243243244 and parameters: {'n_estimators': 225, 'max_depth': 5, 'learning_rate': 0.03476649150592621, 'subsample': 0.728034992108518, 'colsample_bytree': 0.8925879806965068, 'gamma': 0.9983689107917987, 'min_child_weight': 6}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:22:03,900] Trial 4 finished with value: 0.8387096774193549 and parameters: {'n_estimators': 633, 'max_depth': 3, 'learning_rate': 0.07896186801026692, 'subsample': 0.5852620618436457, 'colsample_bytree': 0.5325257964926398, 'gamma': 4.7444276862666666, 'min_child_weight': 10}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:22:20,232] Trial 5 finished with value: 0.8666666666666667 and parameters: {'n_estimators': 828, 'max_depth': 6, 'learning_rate': 0.013940346079873234, 'subsample': 0.8421165132560784, 'colsample_bytree': 0.7200762468698007, 'gamma': 0.6101911742238941, 'min_child_weight': 5}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:22:23,316] Trial 6 finished with value: 0.8496732026143791 and parameters: {'n_estimators': 130, 'max_depth': 12, 'learning_rate': 0.024112898115291985, 'subsample': 0.831261142176991, 'colsample_bytree': 0.6558555380447055, 'gamma': 2.600340105889054, 'min_child_weight': 6}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:22:26,959] Trial 7 finished with value: 0.8859060402684564 and parameters: {'n_estimators': 266, 'max_depth': 12, 'learning_rate': 0.13962563737015762, 'subsample': 0.9697494707820946, 'colsample_bytree': 0.9474136752138245, 'gamma': 2.9894998940554256, 'min_child_weight': 10}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:22:30,475] Trial 8 finished with value: 0.2791666666666667 and parameters: {'n_estimators': 179, 'max_depth': 4, 'learning_rate': 0.011662890273931383, 'subsample': 0.6626651653816322, 'colsample_bytree': 0.6943386448447411, 'gamma': 1.3567451588694794, 'min_child_weight': 9}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:22:38,748] Trial 9 finished with value: 0.87248322147651 and parameters: {'n_estimators': 421, 'max_depth': 5, 'learning_rate': 0.06333268775321842, 'subsample': 0.5704621124873813, 'colsample_bytree': 0.9010984903770198, 'gamma': 0.3727532183988541, 'min_child_weight': 10}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:22:43,071] Trial 10 finished with value: 0.9090909090909091 and parameters: {'n_estimators': 237, 'max_depth': 12, 'learning_rate': 0.18376569837039086, 'subsample': 0.842154940467273, 'colsample_bytree': 0.5347932177377853, 'gamma': 0.47071072733724595, 'min_child_weight': 1}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:22:46,050] Trial 11 finished with value: 0.896551724137931 and parameters: {'n_estimators': 133, 'max_depth': 11, 'learning_rate': 0.10907846641372561, 'subsample': 0.9190887559024729, 'colsample_bytree': 0.5377323422933883, 'gamma': 1.2061543924544071, 'min_child_weight': 2}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:22:50,005] Trial 12 finished with value: 0.9090909090909091 and parameters: {'n_estimators': 324, 'max_depth': 10, 'learning_rate': 0.2841014760827459, 'subsample': 0.9360306828028742, 'colsample_bytree': 0.5277492667452817, 'gamma': 0.23478556282780194, 'min_child_weight': 2}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:22:58,283] Trial 13 finished with value: 0.9027777777777778 and parameters: {'n_estimators': 545, 'max_depth': 12, 'learning_rate': 0.07846102270426893, 'subsample': 0.8276505494128328, 'colsample_bytree': 0.6533880467034408, 'gamma': 1.2849780733791363, 'min_child_weight': 1}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:23:07,498] Trial 14 finished with value: 0.9027777777777778 and parameters: {'n_estimators': 747, 'max_depth': 12, 'learning_rate': 0.207486274449028, 'subsample': 0.8317354606131447, 'colsample_bytree': 0.524239953378738, 'gamma': 2.4661336117721646, 'min_child_weight': 1}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:23:11,300] Trial 15 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 170, 'max_depth': 12, 'learning_rate': 0.1919132531881732, 'subsample': 0.7368834002786683, 'colsample_bytree': 0.6878694783538342, 'gamma': 0.08472373470178662, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:23:17,725] Trial 16 finished with value: 0.9090909090909091 and parameters: {'n_estimators': 370, 'max_depth': 9, 'learning_rate': 0.2094465395371972, 'subsample': 0.703852499858488, 'colsample_bytree': 0.573934948755293, 'gamma': 0.41877976603145517, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:23:27,995] Trial 17 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 639, 'max_depth': 11, 'learning_rate': 0.1512865585167735, 'subsample': 0.7204069681864858, 'colsample_bytree': 0.6093777497627767, 'gamma': 0.1537150179721286, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:23:40,418] Trial 18 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 789, 'max_depth': 12, 'learning_rate': 0.15883918389714755, 'subsample': 0.7125725461652734, 'colsample_bytree': 0.6269689779432265, 'gamma': 0.1794367009079584, 'min_child_weight': 5}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:23:44,280] Trial 19 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 161, 'max_depth': 10, 'learning_rate': 0.16791284117317642, 'subsample': 0.7232916080084737, 'colsample_bytree': 0.6855119272216394, 'gamma': 0.06887343242206731, 'min_child_weight': 1}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:23:57,780] Trial 20 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 657, 'max_depth': 12, 'learning_rate': 0.06297497226627466, 'subsample': 0.7025219702720306, 'colsample_bytree': 0.7023448325018389, 'gamma': 0.04331246357921964, 'min_child_weight': 3}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:24:07,519] Trial 21 finished with value: 0.9090909090909091 and parameters: {'n_estimators': 683, 'max_depth': 12, 'learning_rate': 0.23555191202670364, 'subsample': 0.707652377876532, 'colsample_bytree': 0.5657957030355031, 'gamma': 0.5114877358062602, 'min_child_weight': 6}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:24:17,386] Trial 22 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 812, 'max_depth': 11, 'learning_rate': 0.1536049133088318, 'subsample': 0.8287689681289715, 'colsample_bytree': 0.6102056489536005, 'gamma': 0.6189748283288307, 'min_child_weight': 4}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:24:38,939] Trial 23 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 837, 'max_depth': 11, 'learning_rate': 0.14970448942879488, 'subsample': 0.661222707434837, 'colsample_bytree': 0.6852541118168689, 'gamma': 1.3525591004874213, 'min_child_weight': 1}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:24:42,302] Trial 24 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 162, 'max_depth': 12, 'learning_rate': 0.27313291479834073, 'subsample': 0.6655413512183916, 'colsample_bytree': 0.8101899166592852, 'gamma': 0.3183252914697175, 'min_child_weight': 1}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:24:57,283] Trial 25 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 971, 'max_depth': 10, 'learning_rate': 0.13148552987500797, 'subsample': 0.6611666930178595, 'colsample_bytree': 0.6774468108854831, 'gamma': 0.1447840306867564, 'min_child_weight': 3}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:25:09,628] Trial 26 finished with value: 0.9078014184397163 and parameters: {'n_estimators': 850, 'max_depth': 11, 'learning_rate': 0.24538403783575916, 'subsample': 0.7245087232276808, 'colsample_bytree': 0.5040857126834262, 'gamma': 0.09036520555639302, 'min_child_weight': 4}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:25:18,800] Trial 27 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 481, 'max_depth': 11, 'learning_rate': 0.13038536217777255, 'subsample': 0.7055802078158724, 'colsample_bytree': 0.6828466138883211, 'gamma': 0.26188792675120565, 'min_child_weight': 4}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:25:30,486] Trial 28 finished with value: 0.9090909090909091 and parameters: {'n_estimators': 790, 'max_depth': 11, 'learning_rate': 0.12510853075108996, 'subsample': 0.6528158803459945, 'colsample_bytree': 0.6565174647971405, 'gamma': 0.9667307951279538, 'min_child_weight': 5}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:25:33,611] Trial 29 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 163, 'max_depth': 12, 'learning_rate': 0.2695255770233388, 'subsample': 0.7028239849347261, 'colsample_bytree': 0.6620441852444867, 'gamma': 1.03256948590216, 'min_child_weight': 4}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:25:38,831] Trial 30 finished with value: 0.8904109589041096 and parameters: {'n_estimators': 309, 'max_depth': 12, 'learning_rate': 0.20624037788868915, 'subsample': 0.7884337733409553, 'colsample_bytree': 0.7312737890524048, 'gamma': 1.030345419702729, 'min_child_weight': 1}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:25:44,159] Trial 31 finished with value: 0.9027777777777778 and parameters: {'n_estimators': 282, 'max_depth': 10, 'learning_rate': 0.16266258137316084, 'subsample': 0.741574459799428, 'colsample_bytree': 0.7466313042533376, 'gamma': 0.14599131434433213, 'min_child_weight': 4}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:25:50,821] Trial 32 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 317, 'max_depth': 12, 'learning_rate': 0.12335058219134529, 'subsample': 0.7020945589953198, 'colsample_bytree': 0.6488230412230701, 'gamma': 0.4837946720741631, 'min_child_weight': 1}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:26:04,176] Trial 33 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 886, 'max_depth': 11, 'learning_rate': 0.09979756395033026, 'subsample': 0.6984041937766812, 'colsample_bytree': 0.5652117910973748, 'gamma': 0.16011678538450522, 'min_child_weight': 1}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:26:08,185] Trial 34 finished with value: 0.896551724137931 and parameters: {'n_estimators': 159, 'max_depth': 9, 'learning_rate': 0.10666727186810351, 'subsample': 0.6428821223708248, 'colsample_bytree': 0.681804554796026, 'gamma': 0.052617777673294806, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:26:13,015] Trial 35 finished with value: 0.9027777777777778 and parameters: {'n_estimators': 147, 'max_depth': 12, 'learning_rate': 0.12279659480840746, 'subsample': 0.7586333625158491, 'colsample_bytree': 0.5502605200144122, 'gamma': 0.07693199826341846, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:26:17,177] Trial 36 finished with value: 0.9090909090909091 and parameters: {'n_estimators': 178, 'max_depth': 10, 'learning_rate': 0.14749300085168304, 'subsample': 0.7123287204289958, 'colsample_bytree': 0.6446195528010665, 'gamma': 0.5048104229539396, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:26:25,445] Trial 37 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 435, 'max_depth': 12, 'learning_rate': 0.23350192630546987, 'subsample': 0.7009972037247556, 'colsample_bytree': 0.5639256188752297, 'gamma': 0.008945985984922855, 'min_child_weight': 3}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:26:29,991] Trial 38 finished with value: 0.9014084507042254 and parameters: {'n_estimators': 235, 'max_depth': 8, 'learning_rate': 0.2117346459539402, 'subsample': 0.6712495453319763, 'colsample_bytree': 0.6850102249228871, 'gamma': 0.14912719066767155, 'min_child_weight': 1}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:26:33,294] Trial 39 finished with value: 0.9027777777777778 and parameters: {'n_estimators': 127, 'max_depth': 10, 'learning_rate': 0.17521797506075237, 'subsample': 0.6426596201620651, 'colsample_bytree': 0.6363557444418436, 'gamma': 0.05436541041649444, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:26:37,616] Trial 40 finished with value: 0.9027777777777778 and parameters: {'n_estimators': 162, 'max_depth': 11, 'learning_rate': 0.23417073831780202, 'subsample': 0.7625285913984116, 'colsample_bytree': 0.6614455460788449, 'gamma': 0.4120079434712507, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:26:44,289] Trial 41 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 466, 'max_depth': 11, 'learning_rate': 0.17414106828639198, 'subsample': 0.7811471167982436, 'colsample_bytree': 0.6402983446404347, 'gamma': 0.16589097431531347, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:26:51,731] Trial 42 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 333, 'max_depth': 9, 'learning_rate': 0.10526604151218114, 'subsample': 0.7504287132720238, 'colsample_bytree': 0.6880602211158623, 'gamma': 0.20265383649187288, 'min_child_weight': 1}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:26:57,066] Trial 43 finished with value: 0.9090909090909091 and parameters: {'n_estimators': 326, 'max_depth': 11, 'learning_rate': 0.22273816464847132, 'subsample': 0.7958694832396015, 'colsample_bytree': 0.6075314187496843, 'gamma': 0.2952274975010696, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:27:01,607] Trial 44 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 131, 'max_depth': 11, 'learning_rate': 0.13441803917336914, 'subsample': 0.785731632439452, 'colsample_bytree': 0.7364515858793537, 'gamma': 0.36557279215418614, 'min_child_weight': 1}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:27:04,947] Trial 45 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 151, 'max_depth': 12, 'learning_rate': 0.17670826631482295, 'subsample': 0.8268894226499287, 'colsample_bytree': 0.6986099449557759, 'gamma': 0.2842018086489252, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:27:08,401] Trial 46 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 213, 'max_depth': 11, 'learning_rate': 0.17046702887814622, 'subsample': 0.8252361710938035, 'colsample_bytree': 0.6332754330172651, 'gamma': 0.6339151359153619, 'min_child_weight': 1}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:27:12,488] Trial 47 finished with value: 0.9090909090909091 and parameters: {'n_estimators': 144, 'max_depth': 12, 'learning_rate': 0.2223434812786068, 'subsample': 0.7654719864552866, 'colsample_bytree': 0.7348121377280795, 'gamma': 0.4886968163433367, 'min_child_weight': 1}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:27:19,959] Trial 48 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 522, 'max_depth': 12, 'learning_rate': 0.11724450352657322, 'subsample': 0.7138760470268312, 'colsample_bytree': 0.6395151745748655, 'gamma': 0.5295488062297992, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:27:26,966] Trial 49 finished with value: 0.9090909090909091 and parameters: {'n_estimators': 383, 'max_depth': 12, 'learning_rate': 0.2388657404951286, 'subsample': 0.7108520580738419, 'colsample_bytree': 0.664557151650255, 'gamma': 0.1638109245857607, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
# Best parameters
print("Best f1_score:", study.best_value)
print("Best parameters:", study.best_trial.params)
Best f1_score: 0.9154929577464789
Best parameters: {'n_estimators': 170, 'max_depth': 12, 'learning_rate': 0.1919132531881732, 'subsample': 0.7368834002786683, 'colsample_bytree': 0.6878694783538342, 'gamma': 0.08472373470178662, 'min_child_weight': 2}
xgb = XGBClassifier(**study.best_trial.params, scale_pos_weight=fraud_weight, random_state=42, use_label_encoder=False,verbose=1, n_jobs=-1)
results_list.append(build_and_evaluate_model(xgb, X_train_scaled, y_train, X_val_scaled, y_val, 'XGBoost + Optuna (threshold = 0.5)', threshold=0.5))
Classification Report: XGBoost + Optuna (threshold = 0.5)
precision recall f1-score support
legit 1.00 1.00 1.00 45320
fraud 0.98 0.86 0.92 76
accuracy 1.00 45396
macro avg 0.99 0.93 0.96 45396
weighted avg 1.00 1.00 1.00 45396
🔍 Optuna Observations
Precision improved further to 98%, Recall held at 86%, F1 0.92. Bayesian hyperparameter tuning improved precision without sacrificing recall, an improvement over the default XGBoost configuration.
Optuna searched 50 hyperparameter combinations for XGBoost, scoring each on the validation set, and returned study.best_trial.params which we use to build a new xgb model. We feed the models probability outputs into the precision-recall curve on the validation set, to find best_threshold.
Plotting Precision-Recall Tradeoff and Determining Best Threshold for Fraud Detection
import matplotlib.pyplot as plt
import numpy as np
from sklearn.metrics import precision_recall_curve
# Predict probabilities
y_pred_proba = xgb.predict_proba(X_val_scaled)[:, 1] # Get fraud class probability
# Compute precision-recall curve
precision, recall, thresholds = precision_recall_curve(y_val, y_pred_proba)
# Compute F1-score
f1_scores = (2 * precision[:-1] * recall[:-1]) / (precision[:-1] + recall[:-1] + 1e-10) # Avoid division by zero
best_idx = np.argmax(f1_scores)
best_threshold = thresholds[best_idx]
# Plot Precision and Recall against Threshold
plt.figure(figsize=(8, 6))
plt.plot(thresholds, precision[:-1], label="Precision", linestyle="--", marker="o", color="blue")
plt.plot(thresholds, recall[:-1], label="Recall", linestyle="--", marker="s", color="orange")
# Highlight best threshold
plt.axvline(best_threshold, color="red", linestyle="--", label=f"Best Threshold = {best_threshold}")
plt.scatter(best_threshold, precision[best_idx], color='red', s=100, label="Best Precision")
plt.scatter(best_threshold, recall[best_idx], color='blue', s=100, label="Best Recall")
# Labels and Legend
plt.xlabel("Threshold")
plt.ylabel("Score")
plt.title("Precision-Recall Tradeoff")
plt.legend()
plt.grid()
plt.show()
print(f"At threshold {best_threshold:.4f}: Precision = {precision[best_idx]:.4f}, Recall = {recall[best_idx]:.4f}")
At threshold 0.9721: Precision = 0.9848, Recall = 0.8553
The same tuned model, with the best threshold, we evaluate on the test set for a final result.
results_list.append(build_and_evaluate_model(xgb, X_train_scaled, y_train, X_test_scaled, y_test, 'XGBoost + Optuna', threshold=best_threshold))
Classification Report: XGBoost + Optuna
precision recall f1-score support
legit 1.00 1.00 1.00 56651
fraud 0.99 0.73 0.84 95
accuracy 1.00 56746
macro avg 0.99 0.86 0.92 56746
weighted avg 1.00 1.00 1.00 56746
🔍 Best Threshold Selection Observations
On the held-out test set, touched for the first time here, Precision 99%, Recall 73%, F1 0.84. Raising the decision threshold (found via the validation precision-recall curve) traded some recall for near-perfect precision, appropriate for a fraud system where false alarms carry a real operational cost.
results = pd.DataFrame(results_list)
results
| Model | Precision | Recall | F1 Score | |
|---|---|---|---|---|
| 0 | Baseline | 0.912 | 0.684 | 0.782 |
| 1 | RandomOverSampler | 0.053 | 0.895 | 0.100 |
| 2 | SMOTE | 0.049 | 0.882 | 0.093 |
| 3 | ADASYN | 0.017 | 0.908 | 0.033 |
| 4 | RandomUnderSampler | 0.027 | 0.908 | 0.052 |
| 5 | EditedNearestNeighbours | 0.930 | 0.697 | 0.797 |
| 6 | Balanced Class Weights | 0.054 | 0.895 | 0.101 |
| 7 | Custom Class Weights | 0.035 | 0.908 | 0.067 |
| 8 | XGBoost | 0.942 | 0.855 | 0.897 |
| 9 | XGBoost + Optuna (threshold = 0.5) | 0.985 | 0.855 | 0.915 |
| 10 | XGBoost + Optuna | 0.986 | 0.726 | 0.836 |
This project set out to detect fraudulent credit card transactions in a dataset where only 0.17% of transactions were fraudulent this level of imbalance makes naive accuracy meaningless and requires careful methodology to evaluate honestly.
What we found across techniques:
Standard resampling (RandomOverSampler, SMOTE, ADASYN, RandomUnderSampler) and cost-sensitive class weighting all followed the same pattern, they push Recall above 88% by making Precision collapse to single digits (as low as 1.7% for ADASYN). These approaches caught most fraud but flagged so many legitimate transactions in the process that they would be unusable for a real fraud review team.
ENNs (EditedNearestNeighbours) was the one resampling method that improved meaningfully over the baseline (F1 0.797 vs. 0.782), by cleaning noisy data near the decision boundary rather than blindly duplicating or removing records.
The real breakthrough came from changing the algorithm, not the data.
The final model using the best precision-recall threshold when evaluated once on a held-out test set that had no influence on any modeling decision, achieved a precision = 98.6%, recall=72.6% and F1= 0.836. This means that when the model flags a transaction as fraud, it is correct 98.6% of the time, virtually eliminating false alarms, while still catching roughly 3 out of every 4 actual fraud cases.
In fraud detection, false positives carry a real operational cost, they erode trust in the system and overwhelm the humans who must act on every alert. By prioritizing Precision, the final model produces alerts a fraud team can act on with confidence, rather than a flood of false alarms that would eventually get ignored. This reflects a deliberate business trade-off rather than a purely statistical one: catching more fraud was possible, but only at a cost to system usability that outweighed the benefit.