💳Credit Card Fraud Detection : Dealing with Class Imbalance

[1] Background & Problem Overview¶

The Problem

Credit card fraud detection is a binary classification problem with a defining challenge: fraud is rare. In this dataset, only 0.17% of transactions are fraudulent, meaning a model that predicts "legitimate" for every transaction would be 99.8% accurate while catching zero fraud. This makes standard accuracy meaningless here, and requires evaluating the model using metrics that specifically measure performance on the minority (fraud) class.

Key Definitions

  • Positive = Fraudulent transaction
  • Negative = Legitimate transaction
  • Precision — How trustworthy are the model's fraud predictions? Of everything flagged as fraud, how much actually is fraud?
    • High FP → Low Precision (more false alarms erode trust in the "fraud" label)
  • Recall — Of all actual fraud cases, how many did the model correctly catch?
    • High FN → Low Recall (more missed fraud means fewer real cases caught)

Goal

Minimize FNs (missed fraud) as much as possible while maintaining strong Precision. Missed fraud causes direct financial loss, but excessive false alarms overwhelm a fraud review team and erode trust in the system, so both failure modes carry a real cost.

We want the best possible recall you can get, subject to precision staying high enough that the alerts remain usable.

Approach

This notebook evaluates multiple strategies for handling severe class imbalance, resampling techniques, cost-sensitive learning, and algorithm choice, before selecting a final model through systematic comparison on a held-out validation set, with performance confirmed on a separate test set.

image.png

[2] Data Loading and Dependencies 📤¶

Data Source : https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud

In [1]:
import kagglehub

# Download latest version
path = kagglehub.dataset_download("mlg-ulb/creditcardfraud")

print("Path to dataset files:", path)
Downloading to /root/.cache/kagglehub/datasets/mlg-ulb/creditcardfraud/3.archive...
100%|██████████| 66.0M/66.0M [00:00<00:00, 81.3MB/s]
Extracting files...

Path to dataset files: /root/.cache/kagglehub/datasets/mlg-ulb/creditcardfraud/versions/3

Dependencies and Versions¶

  • scikit-learn: 1.6.1
  • xgboost: 2.1.3
In [2]:
#import xgboost
#print(xgboost.__version__)
In [3]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, classification_report
from imblearn.over_sampling import SMOTE, RandomOverSampler, ADASYN
from imblearn.under_sampling import RandomUnderSampler, EditedNearestNeighbours
from sklearn.preprocessing import StandardScaler
from xgboost import XGBClassifier
In [4]:
# Ignore warnings
import warnings
warnings.filterwarnings('ignore')
In [5]:
data = pd.read_csv(path + "/creditcard.csv")
data.head()
Out[5]:
Time V1 V2 V3 V4 V5 V6 V7 V8 V9 ... V21 V22 V23 V24 V25 V26 V27 V28 Amount Class
0 0.0 -1.359807 -0.072781 2.536347 1.378155 -0.338321 0.462388 0.239599 0.098698 0.363787 ... -0.018307 0.277838 -0.110474 0.066928 0.128539 -0.189115 0.133558 -0.021053 149.62 0
1 0.0 1.191857 0.266151 0.166480 0.448154 0.060018 -0.082361 -0.078803 0.085102 -0.255425 ... -0.225775 -0.638672 0.101288 -0.339846 0.167170 0.125895 -0.008983 0.014724 2.69 0
2 1.0 -1.358354 -1.340163 1.773209 0.379780 -0.503198 1.800499 0.791461 0.247676 -1.514654 ... 0.247998 0.771679 0.909412 -0.689281 -0.327642 -0.139097 -0.055353 -0.059752 378.66 0
3 1.0 -0.966272 -0.185226 1.792993 -0.863291 -0.010309 1.247203 0.237609 0.377436 -1.387024 ... -0.108300 0.005274 -0.190321 -1.175575 0.647376 -0.221929 0.062723 0.061458 123.50 0
4 2.0 -1.158233 0.877737 1.548718 0.403034 -0.407193 0.095921 0.592941 -0.270533 0.817739 ... -0.009431 0.798278 -0.137458 0.141267 -0.206010 0.502292 0.219422 0.215153 69.99 0

5 rows × 31 columns

In [6]:
len(data)
Out[6]:
284807

[3] Data Exploration and Cleaning 🧹¶

In [7]:
# Check for duplicates
print(f'There are {data.duplicated().sum()} duplicates.')

# Remove duplicates
data.drop_duplicates(inplace=True)
print('All duplicates have been removed.')
There are 1081 duplicates.
All duplicates have been removed.
In [8]:
# Check for missing values
print(f'Missing values present in dataset: {data.isna().any().any()}')
Missing values present in dataset: False
In [9]:
# Change all column names to lowercase
data.columns = data.columns.str.lower()
data.columns
Out[9]:
Index(['time', 'v1', 'v2', 'v3', 'v4', 'v5', 'v6', 'v7', 'v8', 'v9', 'v10',
       'v11', 'v12', 'v13', 'v14', 'v15', 'v16', 'v17', 'v18', 'v19', 'v20',
       'v21', 'v22', 'v23', 'v24', 'v25', 'v26', 'v27', 'v28', 'amount',
       'class'],
      dtype='object')
In [10]:
# Drop unneccessary columns
data.drop(['time'], axis=1, inplace=True)
In [11]:
df = data.copy()

[4] EDA 📊¶

Target Variable : Class distribution¶

In [12]:
df['class'].value_counts().reset_index()
Out[12]:
class count
0 0 283253
1 1 473
In [13]:
## Create a pie plot to show the distribution of the class label
plt.figure(figsize=(6,4))
plt.pie(df['class'].value_counts(), labels=['Legitimate', 'Fraud' ], autopct='%1.1f%%', colors=['green', 'red'])
plt.title('Distribution of Class Label')
plt.show()

Observations 🔍

Because of the substantial class imbalance, we will explore resampling methods to address it.

[5] Data Preparation¶

In [14]:
# Define X and y
X = df.drop('class', axis=1)
y = df['class']
In [15]:
# Train_test_split  (80% train, 20% test)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
In [16]:
# Validation_split (80% train, 20% validation)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=42, stratify=y_train)
In [17]:
# Feature scaling
# Instantiate a scaler object
scaler = StandardScaler()

# Normalize the training and test feature sets to improve model performance
X_train_scaled = scaler.fit_transform(X_train)
X_val_scaled = scaler.transform(X_val)
X_test_scaled = scaler.transform(X_test)

[6] Model Building and Evaluation¶

In [18]:
# Evaluation function
from sklearn.metrics import precision_score, recall_score, f1_score

def build_and_evaluate_model(model, train_X, train_y, test_X, test_y, technique, threshold=None):
  model.fit(train_X, train_y)

  if threshold == None:
      # Make predictions
    y_pred = model.predict(test_X)
  else:
    y_pred_proba = model.predict_proba(test_X)[:, 1]  # Get fraud class probability
    y_pred = (y_pred_proba >= threshold).astype(int)

  # Create classification report
  print(f'Classification Report: {technique} \n')
  print(classification_report(test_y, y_pred, target_names=['legit', 'fraud']))

  # Create confusion matrix
  cm = confusion_matrix(test_y, y_pred)
  plt.figure(figsize=(3,3))
  sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', xticklabels=['Legit', 'Fraud'], yticklabels=['Legit', 'Fraud'],cbar=False) # format=decimal
  plt.xlabel('Predicted_Labels'),
  plt.ylabel('Actual_Labels')
  plt.xticks(fontsize=10)
  plt.yticks(fontsize=10)
  plt.title('Confusion Matrix')
  plt.show()

  # Return metrics for each model
  return {
      'Model': technique,
      'Precision': round(precision_score(test_y, y_pred, pos_label=1), 3),
      'Recall': round(recall_score(test_y, y_pred, pos_label=1), 3),
      'F1 Score': round(f1_score(test_y, y_pred, pos_label=1),3)
  }
In [19]:
results_list = []

Baseline model¶

In [20]:
logit= LogisticRegression(max_iter=1000, random_state=42)
In [21]:
results_list.append(build_and_evaluate_model(logit, X_train_scaled, y_train, X_val_scaled, y_val, 'Baseline'))
Classification Report: Baseline 

              precision    recall  f1-score   support

       legit       1.00      1.00      1.00     45320
       fraud       0.91      0.68      0.78        76

    accuracy                           1.00     45396
   macro avg       0.96      0.84      0.89     45396
weighted avg       1.00      1.00      1.00     45396

🔍 Baseline Observations

Precision 91%, Recall 68%, F1 0.78. The model is conservative: when it flags a transaction as fraud, it's usually right,we have very few false alarms.but it misses roughly a third of actual fraud cases. This is a reasonable starting point, but the missed-fraud rate is too high for real deployment.

  • TN = 45315 transactions that are actually legit was also flagged by the model as legit.
  • FP = 5. The model incorrectly flagged these Legit transactions as fraud.
  • FN = 24. The model incorrectly flagged these fraudulent transactions as legit.
  • TP = 52 transactions that are actually fraud was also flagged by the model as fraud.

Recall our goal is to minimize FNs (missed fraud) as much as possible without letting Precision collapse, since a high volume of false alarms would overwhelm a fraud review team and erode trust in the system, while missed fraud leads directly to financial loss.

[7] Handling Class Imbalance 📈¶

7.1 Oversampling Techniques¶

7.1.1 RandomOverSampler¶

In [22]:
ros = RandomOverSampler(random_state=42)
In [23]:
# Resample training data
X_resampled_ros, y_resampled_ros = ros.fit_resample(X_train_scaled, y_train)
In [24]:
## Create a pie plot to show the distribution of the class label
plt.figure(figsize=(6,4))
plt.pie(y_resampled_ros.value_counts(), labels=['Legitimate', 'Fraud' ], autopct='%1.1f%%', colors=['green', 'red'])
plt.title('Class Distribution After RandomOverSampler')
plt.show()
In [25]:
results_list.append(build_and_evaluate_model(logit, X_resampled_ros, y_resampled_ros, X_val_scaled, y_val, 'RandomOverSampler'))
Classification Report: RandomOverSampler 

              precision    recall  f1-score   support

       legit       1.00      0.97      0.99     45320
       fraud       0.05      0.89      0.10        76

    accuracy                           0.97     45396
   macro avg       0.53      0.93      0.54     45396
weighted avg       1.00      0.97      0.98     45396

🔍 RandomOverSampler Observations

Precision dropped sharply to 5%, while Recall rose to 89%. The model now catches nearly 9 in 10 fraud cases, but at the cost of flooding the output with false alarms; only 1 in 20 fraud flags is genuinely fraud. F1 fell to 0.10, showing this trade isn't worth it as-is.

7.1.2 SMOTE¶

In [26]:
smote = SMOTE(random_state=42)
In [27]:
# Resample training data
X_resampled_sm, y_resampled_sm = smote.fit_resample(X_train_scaled, y_train)
In [28]:
# Distribution of class labels
y_resampled_sm.value_counts().reset_index()
Out[28]:
class count
0 0 181282
1 1 181282
In [29]:
results_list.append(build_and_evaluate_model(logit, X_resampled_sm, y_resampled_sm, X_val_scaled, y_val, 'SMOTE'))
Classification Report: SMOTE 

              precision    recall  f1-score   support

       legit       1.00      0.97      0.99     45320
       fraud       0.05      0.88      0.09        76

    accuracy                           0.97     45396
   macro avg       0.52      0.93      0.54     45396
weighted avg       1.00      0.97      0.98     45396

🔍 SMOTE Observations

Nearly identical to RandomOverSampler: Precision 5%, Recall 88%, F1 0.09. Generating synthetic fraud examples didn't meaningfully change the outcome compared to simple duplication in this case.

7.1.3 ADASYN¶

In [30]:
adasyn = ADASYN(random_state=42)
In [31]:
# Resample training data
X_resampled_adsn, y_resampled_adsn = adasyn.fit_resample(X_train_scaled, y_train)
In [32]:
y_resampled_adsn.value_counts().reset_index()
Out[32]:
class count
0 0 181282
1 1 181280
In [33]:
results_list.append(build_and_evaluate_model(logit, X_resampled_adsn, y_resampled_adsn, X_val_scaled, y_val, 'ADASYN'))
Classification Report: ADASYN 

              precision    recall  f1-score   support

       legit       1.00      0.91      0.95     45320
       fraud       0.02      0.91      0.03        76

    accuracy                           0.91     45396
   macro avg       0.51      0.91      0.49     45396
weighted avg       1.00      0.91      0.95     45396

🔍 ADASYN Observations

Precision collapsed further to 2%, while Recall reached 91%, the highest of any resampling technique tested. Confusion matrix confirms the cost: 3976 false positives against only 7 false negatives. Catching almost all fraud isn't useful if it means investigating thousands of legitimate transactions for every real case.

7.2 Undersampling Techniques¶

7.2.1 RandomUnderSampler¶

In [34]:
undersample = RandomUnderSampler(random_state=42)
In [35]:
X_resampled_under, y_resampled_under = undersample.fit_resample(X_train_scaled, y_train)
In [36]:
# Class Distribution
y_resampled_under.value_counts().reset_index()
Out[36]:
class count
0 0 302
1 1 302

We have a lot less data to train on.

In [37]:
results_list.append(build_and_evaluate_model(logit, X_resampled_under, y_resampled_under, X_val_scaled, y_val, 'RandomUnderSampler'))
Classification Report: RandomUnderSampler 

              precision    recall  f1-score   support

       legit       1.00      0.94      0.97     45320
       fraud       0.03      0.91      0.05        76

    accuracy                           0.94     45396
   macro avg       0.51      0.93      0.51     45396
weighted avg       1.00      0.94      0.97     45396

🔍RandomUnderSampler Observations

Precision 3%, Recall 91%, F1 0.05. Despite training on far less data (majority class cut down to match the minority), performance tracks closely with the oversampling methods: strong recall, unusable precision.

7.2.2 EditedNearestNeighbours¶

ENN removes samples that have a majority of their nearest neighbors belonging to a different class, effectively cleaning the decision boundary and reducing potential noise.

  • ENN removes samples from the majority class (and sometimes the minority class) to refine the decision boundary.
In [38]:
ENN = EditedNearestNeighbours()
In [39]:
X_resampled_ENN, y_resampled_ENN = ENN.fit_resample(X_train_scaled, y_train)
In [40]:
y_resampled_ENN.value_counts().reset_index()
Out[40]:
class count
0 0 181173
1 1 302

Samples have been removed from the majority class, the minority class remain unchanged.

In [41]:
results_list.append(build_and_evaluate_model(logit, X_resampled_ENN, y_resampled_ENN, X_val_scaled, y_val, 'EditedNearestNeighbours'))
Classification Report: EditedNearestNeighbours 

              precision    recall  f1-score   support

       legit       1.00      1.00      1.00     45320
       fraud       0.93      0.70      0.80        76

    accuracy                           1.00     45396
   macro avg       0.96      0.85      0.90     45396
weighted avg       1.00      1.00      1.00     45396

🔍 ENN Observations

A clear step up: Precision 93%, Recall 70%, F1 0.80, the best F1 of any resampling method tried so far. Cleaning noisy majority-class samples near the decision boundary, rather than blindly duplicating or removing data, produced a far more balanced result.

7.3 Cost‑Sensitive Learning¶

(Algorithmic weighting instead of resampling)

7.3.1 Balanced Class Weights¶

Given that only 0.2% of samples are labeled as fraud, while 99.8% are legitimate transactions, using class weight adjustments is a good approach instead of oversampling or undersampling. This can help to prevent the model overfitting on artificial samples and also will stop us from having to remove valuable data. Additionally this method can help the model give more importance to our minority fraud class.

In [42]:
logit = LogisticRegression(class_weight='balanced', max_iter=1000, random_state=42)

image.png

In [43]:
results_list.append(build_and_evaluate_model(logit, X_train_scaled, y_train, X_val_scaled, y_val, 'Balanced Class Weights'))
Classification Report: Balanced Class Weights 

              precision    recall  f1-score   support

       legit       1.00      0.97      0.99     45320
       fraud       0.05      0.89      0.10        76

    accuracy                           0.97     45396
   macro avg       0.53      0.93      0.54     45396
weighted avg       1.00      0.97      0.98     45396

🔍 Balanced Class Weights Observations

Precision 5%, Recall 89%, F1 0.10, nearly identical to RandomOverSampler. Adjusting class weights instead of resampling didn't avoid the same precision collapse.

7.3.2 Custom Class Weights¶

In [44]:
logit = LogisticRegression(class_weight={0:1, 1:1000}, max_iter=1000, random_state=42)
In [45]:
results_list.append(build_and_evaluate_model(logit, X_train_scaled, y_train, X_val_scaled, y_val, 'Custom Class Weights'))
Classification Report: Custom Class Weights 

              precision    recall  f1-score   support

       legit       1.00      0.96      0.98     45320
       fraud       0.04      0.91      0.07        76

    accuracy                           0.96     45396
   macro avg       0.52      0.93      0.52     45396
weighted avg       1.00      0.96      0.98     45396

🔍Custom Class Weights Observations

Precision 4%, Recall 91%, F1 0.07, slightly worse than the balanced-weight version. Manually pushing the fraud class weight higher increased recall marginally but degraded precision further, reinforcing that weight adjustment alone isn't sufficient here.

7.3.3 Cost‑Sensitive Algorithms¶

7.3.3.1 Support Vector Classification¶

  • Dataset is large, so it takes too long to build the model.

7.3.3.2 Random Forest¶

  • Dataset is large, model building is slow.

7.3.3.3 XGBoost (Extreme Gradient Boosting)¶

XGBoost is an optimized gradient boosting library that works exceptionally well for fraud detection. It provides scale_pos_weight, which adjusts the weight of the minority class.

In [46]:
fraud_weight = y_train.value_counts()[0] / y_train.value_counts()[1]
fraud_weight
Out[46]:
np.float64(600.2715231788079)
In [47]:
xgb = XGBClassifier(n_estimators = 100, scale_pos_weight=fraud_weight, random_state=42, use_label_encoder=False,verbose=1, n_jobs=-1)
In [48]:
results_list.append(build_and_evaluate_model(xgb, X_train_scaled, y_train, X_val_scaled, y_val, 'XGBoost'))
Classification Report: XGBoost 

              precision    recall  f1-score   support

       legit       1.00      1.00      1.00     45320
       fraud       0.94      0.86      0.90        76

    accuracy                           1.00     45396
   macro avg       0.97      0.93      0.95     45396
weighted avg       1.00      1.00      1.00     45396

🔍XGBoost Observations

A significant jump: Precision 94%, Recall 86%, F1 0.90, the best result by a wide margin. Using scale_pos_weight instead of resampling let the model learn from the real data distribution while still accounting for the imbalance.

[8] Model Optimization¶

8.1 Bayesian Optimization ( Optuna)¶

We can implement optuna to tune the hyperparameters of our XGBoost model.

In [49]:
# install optuna
!pip install optuna
Collecting optuna
  Downloading optuna-5.0.0-py3-none-any.whl.metadata (14 kB)
Collecting alembic>=1.5.0 (from optuna)
  Downloading alembic-1.20.1-py3-none-any.whl.metadata (7.3 kB)
Collecting colorlog (from optuna)
  Downloading colorlog-6.12.0-py3-none-any.whl.metadata (11 kB)
Requirement already satisfied: numpy in /usr/local/lib/python3.13/dist-packages (from optuna) (2.1.3)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.13/dist-packages (from optuna) (26.3)
Requirement already satisfied: sqlalchemy>=1.4.2 in /usr/local/lib/python3.13/dist-packages (from optuna) (2.1.5)
Requirement already satisfied: tqdm in /usr/local/lib/python3.13/dist-packages (from optuna) (4.67.3)
Requirement already satisfied: PyYAML in /usr/local/lib/python3.13/dist-packages (from optuna) (6.0.3)
Requirement already satisfied: Mako in /usr/lib/python3/dist-packages (from alembic>=1.5.0->optuna) (1.3.2.dev0)
Requirement already satisfied: typing-extensions>=4.12 in /usr/local/lib/python3.13/dist-packages (from alembic>=1.5.0->optuna) (4.16.0)
Downloading optuna-5.0.0-py3-none-any.whl (442 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 442.4/442.4 kB 7.2 MB/s eta 0:00:00
Downloading alembic-1.20.1-py3-none-any.whl (270 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 270.4/270.4 kB 27.7 MB/s eta 0:00:00
Downloading colorlog-6.12.0-py3-none-any.whl (12 kB)
Installing collected packages: colorlog, alembic, optuna
Successfully installed alembic-1.20.1 colorlog-6.12.0 optuna-5.0.0
In [50]:
import optuna
from sklearn.metrics import  f1_score


# Define Optuna objective function
def objective(trial):
    params = {
        'n_estimators': trial.suggest_int('n_estimators', 100, 1000),
        'max_depth': trial.suggest_int('max_depth', 3, 12),
        'learning_rate': trial.suggest_loguniform('learning_rate', 0.01, 0.3),
        'subsample': trial.suggest_uniform('subsample', 0.5, 1),
        'colsample_bytree': trial.suggest_uniform('colsample_bytree', 0.5, 1),
        'gamma': trial.suggest_uniform('gamma', 0, 5),
        'min_child_weight': trial.suggest_int('min_child_weight', 1, 10),
    }

    # Train XGBoost model
    model = XGBClassifier(**params, scale_pos_weight=fraud_weight, use_label_encoder=False, eval_metric='logloss', verbose=1, n_jobs=-1)
    model.fit(X_train_scaled, y_train)

    # Predict probabilities
    y_pred_proba = model.predict_proba(X_val_scaled)[:, 1]  # Get fraud class probability

    # Convert probabilities to class labels using a low threshold (improves recall)
    threshold = 0.5  # You can tune this further
    y_pred = (y_pred_proba >= threshold).astype(int)
    #---------------------------------------------------------------------
    # Calculate recall
    #recall = recall_score(y_test, y_pred)
    #return recall
    #---------------------------------------------------------------------
    # Calculate F1-score
    f1 = f1_score(y_val, y_pred)
    return f1

# Run Optuna optimization
study = optuna.create_study(direction='maximize', sampler=optuna.samplers.TPESampler(seed=42)) # Maximise f1-score for fraud
study.optimize(objective, n_trials=50)
[I 2026-10-22 13:21:13,692] A new study created in memory with name: no-name-b4c9f44d-c272-413a-8ef7-c503847b8413
[I 2026-10-22 13:21:31,448] Trial 0 finished with value: 0.9090909090909091 and parameters: {'n_estimators': 437, 'max_depth': 12, 'learning_rate': 0.1205712628744377, 'subsample': 0.7993292420985183, 'colsample_bytree': 0.5780093202212182, 'gamma': 0.7799726016810132, 'min_child_weight': 1}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:21:45,124] Trial 1 finished with value: 0.9027777777777778 and parameters: {'n_estimators': 880, 'max_depth': 9, 'learning_rate': 0.11114989443094977, 'subsample': 0.5102922471479012, 'colsample_bytree': 0.9849549260809971, 'gamma': 4.162213204002109, 'min_child_weight': 3}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:21:49,305] Trial 2 finished with value: 0.5075757575757576 and parameters: {'n_estimators': 263, 'max_depth': 4, 'learning_rate': 0.028145092716060652, 'subsample': 0.762378215816119, 'colsample_bytree': 0.7159725093210578, 'gamma': 1.4561457009902097, 'min_child_weight': 7}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:21:54,623] Trial 3 finished with value: 0.7243243243243244 and parameters: {'n_estimators': 225, 'max_depth': 5, 'learning_rate': 0.03476649150592621, 'subsample': 0.728034992108518, 'colsample_bytree': 0.8925879806965068, 'gamma': 0.9983689107917987, 'min_child_weight': 6}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:22:03,900] Trial 4 finished with value: 0.8387096774193549 and parameters: {'n_estimators': 633, 'max_depth': 3, 'learning_rate': 0.07896186801026692, 'subsample': 0.5852620618436457, 'colsample_bytree': 0.5325257964926398, 'gamma': 4.7444276862666666, 'min_child_weight': 10}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:22:20,232] Trial 5 finished with value: 0.8666666666666667 and parameters: {'n_estimators': 828, 'max_depth': 6, 'learning_rate': 0.013940346079873234, 'subsample': 0.8421165132560784, 'colsample_bytree': 0.7200762468698007, 'gamma': 0.6101911742238941, 'min_child_weight': 5}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:22:23,316] Trial 6 finished with value: 0.8496732026143791 and parameters: {'n_estimators': 130, 'max_depth': 12, 'learning_rate': 0.024112898115291985, 'subsample': 0.831261142176991, 'colsample_bytree': 0.6558555380447055, 'gamma': 2.600340105889054, 'min_child_weight': 6}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:22:26,959] Trial 7 finished with value: 0.8859060402684564 and parameters: {'n_estimators': 266, 'max_depth': 12, 'learning_rate': 0.13962563737015762, 'subsample': 0.9697494707820946, 'colsample_bytree': 0.9474136752138245, 'gamma': 2.9894998940554256, 'min_child_weight': 10}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:22:30,475] Trial 8 finished with value: 0.2791666666666667 and parameters: {'n_estimators': 179, 'max_depth': 4, 'learning_rate': 0.011662890273931383, 'subsample': 0.6626651653816322, 'colsample_bytree': 0.6943386448447411, 'gamma': 1.3567451588694794, 'min_child_weight': 9}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:22:38,748] Trial 9 finished with value: 0.87248322147651 and parameters: {'n_estimators': 421, 'max_depth': 5, 'learning_rate': 0.06333268775321842, 'subsample': 0.5704621124873813, 'colsample_bytree': 0.9010984903770198, 'gamma': 0.3727532183988541, 'min_child_weight': 10}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:22:43,071] Trial 10 finished with value: 0.9090909090909091 and parameters: {'n_estimators': 237, 'max_depth': 12, 'learning_rate': 0.18376569837039086, 'subsample': 0.842154940467273, 'colsample_bytree': 0.5347932177377853, 'gamma': 0.47071072733724595, 'min_child_weight': 1}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:22:46,050] Trial 11 finished with value: 0.896551724137931 and parameters: {'n_estimators': 133, 'max_depth': 11, 'learning_rate': 0.10907846641372561, 'subsample': 0.9190887559024729, 'colsample_bytree': 0.5377323422933883, 'gamma': 1.2061543924544071, 'min_child_weight': 2}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:22:50,005] Trial 12 finished with value: 0.9090909090909091 and parameters: {'n_estimators': 324, 'max_depth': 10, 'learning_rate': 0.2841014760827459, 'subsample': 0.9360306828028742, 'colsample_bytree': 0.5277492667452817, 'gamma': 0.23478556282780194, 'min_child_weight': 2}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:22:58,283] Trial 13 finished with value: 0.9027777777777778 and parameters: {'n_estimators': 545, 'max_depth': 12, 'learning_rate': 0.07846102270426893, 'subsample': 0.8276505494128328, 'colsample_bytree': 0.6533880467034408, 'gamma': 1.2849780733791363, 'min_child_weight': 1}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:23:07,498] Trial 14 finished with value: 0.9027777777777778 and parameters: {'n_estimators': 747, 'max_depth': 12, 'learning_rate': 0.207486274449028, 'subsample': 0.8317354606131447, 'colsample_bytree': 0.524239953378738, 'gamma': 2.4661336117721646, 'min_child_weight': 1}. Best is trial 0 with value: 0.9090909090909091.
[I 2026-10-22 13:23:11,300] Trial 15 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 170, 'max_depth': 12, 'learning_rate': 0.1919132531881732, 'subsample': 0.7368834002786683, 'colsample_bytree': 0.6878694783538342, 'gamma': 0.08472373470178662, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:23:17,725] Trial 16 finished with value: 0.9090909090909091 and parameters: {'n_estimators': 370, 'max_depth': 9, 'learning_rate': 0.2094465395371972, 'subsample': 0.703852499858488, 'colsample_bytree': 0.573934948755293, 'gamma': 0.41877976603145517, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:23:27,995] Trial 17 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 639, 'max_depth': 11, 'learning_rate': 0.1512865585167735, 'subsample': 0.7204069681864858, 'colsample_bytree': 0.6093777497627767, 'gamma': 0.1537150179721286, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:23:40,418] Trial 18 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 789, 'max_depth': 12, 'learning_rate': 0.15883918389714755, 'subsample': 0.7125725461652734, 'colsample_bytree': 0.6269689779432265, 'gamma': 0.1794367009079584, 'min_child_weight': 5}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:23:44,280] Trial 19 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 161, 'max_depth': 10, 'learning_rate': 0.16791284117317642, 'subsample': 0.7232916080084737, 'colsample_bytree': 0.6855119272216394, 'gamma': 0.06887343242206731, 'min_child_weight': 1}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:23:57,780] Trial 20 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 657, 'max_depth': 12, 'learning_rate': 0.06297497226627466, 'subsample': 0.7025219702720306, 'colsample_bytree': 0.7023448325018389, 'gamma': 0.04331246357921964, 'min_child_weight': 3}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:24:07,519] Trial 21 finished with value: 0.9090909090909091 and parameters: {'n_estimators': 683, 'max_depth': 12, 'learning_rate': 0.23555191202670364, 'subsample': 0.707652377876532, 'colsample_bytree': 0.5657957030355031, 'gamma': 0.5114877358062602, 'min_child_weight': 6}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:24:17,386] Trial 22 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 812, 'max_depth': 11, 'learning_rate': 0.1536049133088318, 'subsample': 0.8287689681289715, 'colsample_bytree': 0.6102056489536005, 'gamma': 0.6189748283288307, 'min_child_weight': 4}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:24:38,939] Trial 23 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 837, 'max_depth': 11, 'learning_rate': 0.14970448942879488, 'subsample': 0.661222707434837, 'colsample_bytree': 0.6852541118168689, 'gamma': 1.3525591004874213, 'min_child_weight': 1}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:24:42,302] Trial 24 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 162, 'max_depth': 12, 'learning_rate': 0.27313291479834073, 'subsample': 0.6655413512183916, 'colsample_bytree': 0.8101899166592852, 'gamma': 0.3183252914697175, 'min_child_weight': 1}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:24:57,283] Trial 25 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 971, 'max_depth': 10, 'learning_rate': 0.13148552987500797, 'subsample': 0.6611666930178595, 'colsample_bytree': 0.6774468108854831, 'gamma': 0.1447840306867564, 'min_child_weight': 3}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:25:09,628] Trial 26 finished with value: 0.9078014184397163 and parameters: {'n_estimators': 850, 'max_depth': 11, 'learning_rate': 0.24538403783575916, 'subsample': 0.7245087232276808, 'colsample_bytree': 0.5040857126834262, 'gamma': 0.09036520555639302, 'min_child_weight': 4}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:25:18,800] Trial 27 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 481, 'max_depth': 11, 'learning_rate': 0.13038536217777255, 'subsample': 0.7055802078158724, 'colsample_bytree': 0.6828466138883211, 'gamma': 0.26188792675120565, 'min_child_weight': 4}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:25:30,486] Trial 28 finished with value: 0.9090909090909091 and parameters: {'n_estimators': 790, 'max_depth': 11, 'learning_rate': 0.12510853075108996, 'subsample': 0.6528158803459945, 'colsample_bytree': 0.6565174647971405, 'gamma': 0.9667307951279538, 'min_child_weight': 5}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:25:33,611] Trial 29 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 163, 'max_depth': 12, 'learning_rate': 0.2695255770233388, 'subsample': 0.7028239849347261, 'colsample_bytree': 0.6620441852444867, 'gamma': 1.03256948590216, 'min_child_weight': 4}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:25:38,831] Trial 30 finished with value: 0.8904109589041096 and parameters: {'n_estimators': 309, 'max_depth': 12, 'learning_rate': 0.20624037788868915, 'subsample': 0.7884337733409553, 'colsample_bytree': 0.7312737890524048, 'gamma': 1.030345419702729, 'min_child_weight': 1}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:25:44,159] Trial 31 finished with value: 0.9027777777777778 and parameters: {'n_estimators': 282, 'max_depth': 10, 'learning_rate': 0.16266258137316084, 'subsample': 0.741574459799428, 'colsample_bytree': 0.7466313042533376, 'gamma': 0.14599131434433213, 'min_child_weight': 4}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:25:50,821] Trial 32 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 317, 'max_depth': 12, 'learning_rate': 0.12335058219134529, 'subsample': 0.7020945589953198, 'colsample_bytree': 0.6488230412230701, 'gamma': 0.4837946720741631, 'min_child_weight': 1}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:26:04,176] Trial 33 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 886, 'max_depth': 11, 'learning_rate': 0.09979756395033026, 'subsample': 0.6984041937766812, 'colsample_bytree': 0.5652117910973748, 'gamma': 0.16011678538450522, 'min_child_weight': 1}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:26:08,185] Trial 34 finished with value: 0.896551724137931 and parameters: {'n_estimators': 159, 'max_depth': 9, 'learning_rate': 0.10666727186810351, 'subsample': 0.6428821223708248, 'colsample_bytree': 0.681804554796026, 'gamma': 0.052617777673294806, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:26:13,015] Trial 35 finished with value: 0.9027777777777778 and parameters: {'n_estimators': 147, 'max_depth': 12, 'learning_rate': 0.12279659480840746, 'subsample': 0.7586333625158491, 'colsample_bytree': 0.5502605200144122, 'gamma': 0.07693199826341846, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:26:17,177] Trial 36 finished with value: 0.9090909090909091 and parameters: {'n_estimators': 178, 'max_depth': 10, 'learning_rate': 0.14749300085168304, 'subsample': 0.7123287204289958, 'colsample_bytree': 0.6446195528010665, 'gamma': 0.5048104229539396, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:26:25,445] Trial 37 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 435, 'max_depth': 12, 'learning_rate': 0.23350192630546987, 'subsample': 0.7009972037247556, 'colsample_bytree': 0.5639256188752297, 'gamma': 0.008945985984922855, 'min_child_weight': 3}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:26:29,991] Trial 38 finished with value: 0.9014084507042254 and parameters: {'n_estimators': 235, 'max_depth': 8, 'learning_rate': 0.2117346459539402, 'subsample': 0.6712495453319763, 'colsample_bytree': 0.6850102249228871, 'gamma': 0.14912719066767155, 'min_child_weight': 1}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:26:33,294] Trial 39 finished with value: 0.9027777777777778 and parameters: {'n_estimators': 127, 'max_depth': 10, 'learning_rate': 0.17521797506075237, 'subsample': 0.6426596201620651, 'colsample_bytree': 0.6363557444418436, 'gamma': 0.05436541041649444, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:26:37,616] Trial 40 finished with value: 0.9027777777777778 and parameters: {'n_estimators': 162, 'max_depth': 11, 'learning_rate': 0.23417073831780202, 'subsample': 0.7625285913984116, 'colsample_bytree': 0.6614455460788449, 'gamma': 0.4120079434712507, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:26:44,289] Trial 41 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 466, 'max_depth': 11, 'learning_rate': 0.17414106828639198, 'subsample': 0.7811471167982436, 'colsample_bytree': 0.6402983446404347, 'gamma': 0.16589097431531347, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:26:51,731] Trial 42 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 333, 'max_depth': 9, 'learning_rate': 0.10526604151218114, 'subsample': 0.7504287132720238, 'colsample_bytree': 0.6880602211158623, 'gamma': 0.20265383649187288, 'min_child_weight': 1}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:26:57,066] Trial 43 finished with value: 0.9090909090909091 and parameters: {'n_estimators': 326, 'max_depth': 11, 'learning_rate': 0.22273816464847132, 'subsample': 0.7958694832396015, 'colsample_bytree': 0.6075314187496843, 'gamma': 0.2952274975010696, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:27:01,607] Trial 44 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 131, 'max_depth': 11, 'learning_rate': 0.13441803917336914, 'subsample': 0.785731632439452, 'colsample_bytree': 0.7364515858793537, 'gamma': 0.36557279215418614, 'min_child_weight': 1}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:27:04,947] Trial 45 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 151, 'max_depth': 12, 'learning_rate': 0.17670826631482295, 'subsample': 0.8268894226499287, 'colsample_bytree': 0.6986099449557759, 'gamma': 0.2842018086489252, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:27:08,401] Trial 46 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 213, 'max_depth': 11, 'learning_rate': 0.17046702887814622, 'subsample': 0.8252361710938035, 'colsample_bytree': 0.6332754330172651, 'gamma': 0.6339151359153619, 'min_child_weight': 1}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:27:12,488] Trial 47 finished with value: 0.9090909090909091 and parameters: {'n_estimators': 144, 'max_depth': 12, 'learning_rate': 0.2223434812786068, 'subsample': 0.7654719864552866, 'colsample_bytree': 0.7348121377280795, 'gamma': 0.4886968163433367, 'min_child_weight': 1}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:27:19,959] Trial 48 finished with value: 0.9154929577464789 and parameters: {'n_estimators': 522, 'max_depth': 12, 'learning_rate': 0.11724450352657322, 'subsample': 0.7138760470268312, 'colsample_bytree': 0.6395151745748655, 'gamma': 0.5295488062297992, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
[I 2026-10-22 13:27:26,966] Trial 49 finished with value: 0.9090909090909091 and parameters: {'n_estimators': 383, 'max_depth': 12, 'learning_rate': 0.2388657404951286, 'subsample': 0.7108520580738419, 'colsample_bytree': 0.664557151650255, 'gamma': 0.1638109245857607, 'min_child_weight': 2}. Best is trial 15 with value: 0.9154929577464789.
In [51]:
# Best parameters
print("Best f1_score:", study.best_value)
print("Best parameters:", study.best_trial.params)
Best f1_score: 0.9154929577464789
Best parameters: {'n_estimators': 170, 'max_depth': 12, 'learning_rate': 0.1919132531881732, 'subsample': 0.7368834002786683, 'colsample_bytree': 0.6878694783538342, 'gamma': 0.08472373470178662, 'min_child_weight': 2}
In [52]:
xgb = XGBClassifier(**study.best_trial.params, scale_pos_weight=fraud_weight, random_state=42, use_label_encoder=False,verbose=1, n_jobs=-1)
results_list.append(build_and_evaluate_model(xgb, X_train_scaled, y_train, X_val_scaled, y_val, 'XGBoost + Optuna (threshold = 0.5)', threshold=0.5))
Classification Report: XGBoost + Optuna (threshold = 0.5) 

              precision    recall  f1-score   support

       legit       1.00      1.00      1.00     45320
       fraud       0.98      0.86      0.92        76

    accuracy                           1.00     45396
   macro avg       0.99      0.93      0.96     45396
weighted avg       1.00      1.00      1.00     45396

🔍 Optuna Observations

Precision improved further to 98%, Recall held at 86%, F1 0.92. Bayesian hyperparameter tuning improved precision without sacrificing recall, an improvement over the default XGBoost configuration.

8.2 Threshold Selection via Precision–Recall Analysis¶

Optuna searched 50 hyperparameter combinations for XGBoost, scoring each on the validation set, and returned study.best_trial.params which we use to build a new xgb model. We feed the models probability outputs into the precision-recall curve on the validation set, to find best_threshold.

Plotting Precision-Recall Tradeoff and Determining Best Threshold for Fraud Detection

In [53]:
import matplotlib.pyplot as plt
import numpy as np
from sklearn.metrics import precision_recall_curve

# Predict probabilities
y_pred_proba = xgb.predict_proba(X_val_scaled)[:, 1]  # Get fraud class probability

# Compute precision-recall curve
precision, recall, thresholds = precision_recall_curve(y_val, y_pred_proba)


# Compute F1-score
f1_scores = (2 * precision[:-1] * recall[:-1]) / (precision[:-1] + recall[:-1] + 1e-10)  # Avoid division by zero
best_idx = np.argmax(f1_scores)
best_threshold = thresholds[best_idx]


# Plot Precision and Recall against Threshold
plt.figure(figsize=(8, 6))
plt.plot(thresholds, precision[:-1], label="Precision", linestyle="--", marker="o", color="blue")
plt.plot(thresholds, recall[:-1], label="Recall", linestyle="--", marker="s", color="orange")

# Highlight best threshold
plt.axvline(best_threshold, color="red", linestyle="--", label=f"Best Threshold = {best_threshold}")
plt.scatter(best_threshold, precision[best_idx], color='red', s=100, label="Best Precision")
plt.scatter(best_threshold, recall[best_idx], color='blue', s=100, label="Best Recall")

# Labels and Legend
plt.xlabel("Threshold")
plt.ylabel("Score")
plt.title("Precision-Recall Tradeoff")
plt.legend()
plt.grid()
plt.show()

print(f"At threshold {best_threshold:.4f}: Precision = {precision[best_idx]:.4f}, Recall = {recall[best_idx]:.4f}")
At threshold 0.9721: Precision = 0.9848, Recall = 0.8553

The same tuned model, with the best threshold, we evaluate on the test set for a final result.

In [54]:
results_list.append(build_and_evaluate_model(xgb, X_train_scaled, y_train, X_test_scaled, y_test, 'XGBoost + Optuna', threshold=best_threshold))
Classification Report: XGBoost + Optuna 

              precision    recall  f1-score   support

       legit       1.00      1.00      1.00     56651
       fraud       0.99      0.73      0.84        95

    accuracy                           1.00     56746
   macro avg       0.99      0.86      0.92     56746
weighted avg       1.00      1.00      1.00     56746

🔍 Best Threshold Selection Observations

On the held-out test set, touched for the first time here, Precision 99%, Recall 73%, F1 0.84. Raising the decision threshold (found via the validation precision-recall curve) traded some recall for near-perfect precision, appropriate for a fraud system where false alarms carry a real operational cost.

[9] Conclusions¶

In [55]:
results = pd.DataFrame(results_list)
results
Out[55]:
Model Precision Recall F1 Score
0 Baseline 0.912 0.684 0.782
1 RandomOverSampler 0.053 0.895 0.100
2 SMOTE 0.049 0.882 0.093
3 ADASYN 0.017 0.908 0.033
4 RandomUnderSampler 0.027 0.908 0.052
5 EditedNearestNeighbours 0.930 0.697 0.797
6 Balanced Class Weights 0.054 0.895 0.101
7 Custom Class Weights 0.035 0.908 0.067
8 XGBoost 0.942 0.855 0.897
9 XGBoost + Optuna (threshold = 0.5) 0.985 0.855 0.915
10 XGBoost + Optuna 0.986 0.726 0.836

This project set out to detect fraudulent credit card transactions in a dataset where only 0.17% of transactions were fraudulent this level of imbalance makes naive accuracy meaningless and requires careful methodology to evaluate honestly.

What we found across techniques:

  • Standard resampling (RandomOverSampler, SMOTE, ADASYN, RandomUnderSampler) and cost-sensitive class weighting all followed the same pattern, they push Recall above 88% by making Precision collapse to single digits (as low as 1.7% for ADASYN). These approaches caught most fraud but flagged so many legitimate transactions in the process that they would be unusable for a real fraud review team.

  • ENNs (EditedNearestNeighbours) was the one resampling method that improved meaningfully over the baseline (F1 0.797 vs. 0.782), by cleaning noisy data near the decision boundary rather than blindly duplicating or removing records.

  • The real breakthrough came from changing the algorithm, not the data.

    • Switching to XGBoost with scale_pos_weight outperformed every resampling technique tried, reaching Precision 94.2% and Recall 85.5% simultaneously.
    • Bayesian hyperparameter tuning with Optuna improved this further to Precision 98.5%, and tuning the decision threshold via the validation set's precision-recall curve allowed us to prioritize Precision even further for deployment.

The final model using the best precision-recall threshold when evaluated once on a held-out test set that had no influence on any modeling decision, achieved a precision = 98.6%, recall=72.6% and F1= 0.836. This means that when the model flags a transaction as fraud, it is correct 98.6% of the time, virtually eliminating false alarms, while still catching roughly 3 out of every 4 actual fraud cases.

In fraud detection, false positives carry a real operational cost, they erode trust in the system and overwhelm the humans who must act on every alert. By prioritizing Precision, the final model produces alerts a fraud team can act on with confidence, rather than a flood of false alarms that would eventually get ignored. This reflects a deliberate business trade-off rather than a purely statistical one: catching more fraud was possible, but only at a cost to system usability that outweighed the benefit.