đź’ŽPredicting Diamond Prices: A Comprehensive Regression Analysis

Project Overview

This project aims to develop a machine learning model that predicts diamond prices with both high accuracy and clear interpretability. Using a rich dataset of 53,940 entries, the analysis cleans and explores the data, examining relationships between the target variable price and the diamond attribute predictors, and comparing linear and tree-based models.

We do not only focus on predictive performance but also aim to understand model behaviour. We observe how correlated size-related features affect linear model coeffcients, whether feature reduction helps. and how prediction errors behave on both log-transformed and original price scales.

The final model is an Optuna-tuned XGBoost regressor, selected using validation performance and evaluated once on a held-out test set.

Project Highlights

Dataset

Data is comprised of 53,940 observations and 10 features, with price as the target variable. Key features include: carat, cut, color, clarity, depth, table, x, y, z. x, y, z we renamed as length, width, height for improved readability.

Data Cleaning

Methodology

1. Exploratory Data Analysis

The EDA examines the distributions of numerical features, including skewness and kurtosis; price relationships with categorical and numerical features; outliers; and correlations among predictors. ANOVA and Cramér’s V are used to investigate categorical feature relationships.

The analysis shows that diamond price is strongly related to carat weight and other size-related measurements. Its also highlights wha average prieces across quality grades need careful interpretation. Diamonds in different grades can have different carat weight distributions.

2. Preprocessing and data split

The dataset is split into training(64%), validation(16%) and test(20%) subsets. The validation set is used to compare models and tune hyperparameters. The test set is held back for final evaluation.

Preprocessing includes:

Models are trained to predict log1p(price). For metrics in dollars, predictions are transformed back to the original price scale.

3. Model Comparision

The notebook compares several approaches:

The linear model experiments reveal strong correlations among size-related features can make individual(independent) coefficients difficult to interpret. Adding interaction terms improves on the baseline linear model, while tree-based models better capture non-linear relationships and feature interactions.

4. Feature importance and feature reduction experiments

The notebook examines XGBoost’s built-in feature importance and SHAP values. These are used to explore how the model relies on the predictors and how feature values contribute to predictions.

Several reduced feature experiments test whether a smaller model can retain performance:

These experiments support retaining all eight predictors in the final model. Feature importance and SHAP describe model behaviour; they do not establish that a feature has an independent causal effect.

5. XGBoost optimization and residual analysis

Optuna is used to tune XGBoost hyperparameters, with validation RMSE on the original dollar scale as the optimization objection objective. The tuned model improves validation RMSE from $552.97 for the original XGBoost model to $532.46, an improvement of approximately 3.7%.

Residuals are examined on both the log-transformed and original price scales. On the original scale, absolute errors tend to increase for more expensive diamonds, meaning predictions are less precise in dollar terms at the upper end of the price range.

Final Model Performance

The Optimized XGBoost Regressor using all eight predictors, was selected based on validation performance and evaluated on the held-out test set.

Metric Train Validation Test
RMSE $413.29 $532.46 $522.62
RMSE (% of mean price) 10.53% 13.35% 13.39%
MAE $212.60 $264.45 $262.04
MedAE $80.45 $94.45 $95.31
R² (log-transformed target) 0.995 0.993 0.992

Interpreting the test results

Test performance is close to validation performance, which is encouraging for this data split. It is not a guarantee that the same performance will hold on other samples or real-world diamond listings.

Key Findings

  1. Size-related features are strong predictors. Carat weight and physical dimensions rank highly in the EDA and XGBoost interpretation.
  2. Multicollinearity complicates linear model interpretation. Correlated size-related predictors make it difficult to isolate their individual coefficient effects.
  3. Non-linear models perform better in this project. The interaction model improves on baseline linear regression and tree-based models further reduce prediction error.
  4. Tuning improves XGBoost’s validation performance. Optuna reduces validation RMSE by approximately 3.7% compared to the original XGBoost model.
  5. Feature reduction has trade-offs. The tested smaller feature sets do not outperform the tuned model suing all eight features.
  6. Prediction errors are uneven across the price range. Residual analysis indicates larger absolute errors for more expensive diamonds.

Limiations and Next Steps

Repository Contents

Getting started

Clone the repository and open the notebook in a Python environment with Jupyter installed:

# Clone the repository
git clone https://github.com/mcadriaans/ml_diamond-price-prediction.git

# Navigate into the project directory
cd ml_diamond-price-prediction

# Launch the notebook interface
pip install requirements.txt
jupyter notebook