Medical Cost Estimator

Introduction

This project was introduced by Dataquest and was one of the first projects I practised on while I was learning the basics. The objective was to build a linear regression model to predict the medical or insurance cost of individuals.

Description

The dataset contained a few features, such as age, smoker, and gender. During the exploratory data analysis, I discovered an interesting pattern in a scatterplot of cost vs age. The data points were clearly clustered into 3 distinct bands.

I compared the clustering performance of DBSCAN and Gaussian Mixture Model. The latter one successfully labelled the instances into the three groups.

Result

With the newly created feature, the label of the clusters, the R-squared of the linear regression model reached .967.

Nonetheless, although the residuals were generally centred around zero, the three distinct clusters suggested that the model’s performance varied across different groups of observations. This pattern may indicate unmodelled non-linear relationships or underlying differences in factors such as insurance providers and coverage levels. To improve performance more consistently across all groups, separate models could be trained for observations belonging to each cluster.

Conclusion

As a beginner, I was thrilled to achieve the result of the model as I went through data cleaning to model evaluation without the provided basic guidance from Dataquest. There is never a single, model answer to any problem. However, there is always a better solution. The initiative I took to complete this project on my own fuelled my motivation to continue learning, exploring, and applying data science knowledge and techniques to tackle different real-world problems.