Assignment Sample: Evaluating Classification Models for Loan Default Prediction

Model Answer

Introduction

Consumer credit default prediction is one of the most consequential applications of applied machine learning in financial services: overly conservative models needlessly deny credit to sound applicants, while overly permissive models expose lenders to loss and can leave vulnerable borrowers over-indebted (Lessmann et al., 2015). This assignment evaluates two supervised classification models – logistic regression and a gradient-boosted decision tree ensemble – trained to predict twelve-month loan default on a simulated retail lending dataset of 8,000 historical applications, of which 720 (9%) resulted in default. The task requires not only fitting models but justifying, with worked calculations, which evaluation metrics are appropriate given the pronounced class imbalance, and making a defensible recommendation for deployment.

The assignment proceeds in four stages. First, it outlines the data preparation and modelling approach, including the treatment of class imbalance. Second, it presents the two candidate models and their headline performance. Third, it undertakes a detailed, worked comparison of the models using a confusion matrix and derived metrics, explaining why accuracy alone is a misleading criterion in this setting. Fourth, it evaluates the models against business and ethical considerations relevant to consumer lending, including the Financial Conduct Authority’s (FCA, 2022) expectations around fair treatment of customers, before concluding with a recommendation.

Credit scoring has moved through several methodological generations: from judgemental, rules-based lending decisions, through statistical scorecards built on logistic regression from the 1990s onward, to the more recent adoption of ensemble and gradient-boosting methods that can capture non-linear relationships and interactions between applicant characteristics without requiring the analyst to specify them in advance (Lessmann et al., 2015). Logistic regression remains the industry benchmark in UK consumer credit not because it is the most accurate available technique, but because its coefficients are directly interpretable and its outputs are straightforward to justify to regulators and to declined applicants. Any newer technique proposed to replace or supplement it, including the gradient-boosted model evaluated in this assignment, must therefore be assessed on interpretability and regulatory fit as well as on raw predictive performance, a theme returned to in the evaluation section below.

Approach and Method

The dataset was split into a 70% training set (n = 5,600) and a 30% held-out test set (n = 2,400), stratified by the default label to preserve the 9% base rate in both partitions, following standard practice for imbalanced classification tasks (He and Garcia, 2009). Ten predictor variables were retained after initial screening, including annual income, existing debt-to-income ratio, credit utilisation, months since most recent delinquency, employment length and loan purpose, consistent with variables shown to carry predictive signal in prior credit-scoring literature (Lessmann et al., 2015). Categorical variables were one-hot encoded and continuous variables standardised prior to fitting the logistic regression model; the gradient-boosted tree model, which is invariant to monotonic feature scaling, was fitted on the same encoded features without standardisation.

Given the 9% default rate, class imbalance was addressed in two complementary ways rather than relying on a single technique, consistent with Chawla et al.’s (2002) observation that resampling and cost-sensitive learning are often more effective in combination than alone. First, class weights inversely proportional to class frequency were applied during training for both models, so that misclassifying the minority default class was penalised roughly ten times more heavily than misclassifying the majority non-default class. Second, five-fold stratified cross-validation was used during hyperparameter tuning, ensuring that each fold preserved the 9% default rate and that hyperparameters were not selected based on performance on an unrepresentative split. The gradient-boosted model’s key hyperparameters – number of trees, maximum tree depth and learning rate – were tuned via grid search over the cross-validation folds, selecting the combination that maximised the area under the precision-recall curve, a metric more appropriate than the area under the ROC curve for datasets with substantial class imbalance (Saito and Rehmsmeier, 2015).

Two further preparation steps were applied before modelling. Missing values, present in 3.1% of applications for the debt-to-income and credit utilisation fields, were imputed using the median of the training fold to avoid leaking information from the test set, and a binary “was-missing” indicator was retained as an additional feature, since missingness itself can carry predictive signal in credit applications (Crook, Edelman and Thomas, 2007). Two engineered ratio features were also constructed – loan amount relative to annual income, and current monthly obligations relative to income – since domain knowledge suggests that relative, rather than absolute, indebtedness is more predictive of default risk. The logistic regression model was specified with L2 (ridge) regularisation, selected via the same cross-validation procedure used for the gradient-boosted model’s hyperparameters, to reduce the risk of overfitting given the number of one-hot encoded categorical levels relative to the size of the minority class. The gradient-boosted model’s final tuned configuration, selected via grid search, used 300 trees, a maximum depth of four, and a learning rate of 0.05; shallower trees and a conservative learning rate were deliberately favoured over a larger, more aggressive configuration, since preliminary tuning runs with deeper trees showed a widening gap between cross-validation and held-out performance, indicative of overfitting to the training folds.

Model Results and Worked Comparison

Table 1 reports the confusion matrix for the gradient-boosted model on the 2,400-row held-out test set, using a decision threshold of 0.30 (rather than the conventional 0.50) to reflect the asymmetric cost of missed defaults relative to false alarms, a threshold-adjustment approach recommended for imbalanced classification by He and Garcia (2009).

From this confusion matrix, the core evaluation metrics are calculated step by step. Accuracy, the proportion of all predictions that are correct, is calculated first to illustrate why it is a misleading headline metric here:

Accuracy = (TP + TN) / Total = (151 + 1,965) / 2,400 = 2,116 / 2,400 = 0.882 (88.2%)

An accuracy of 88.2% appears strong, but a trivial model that predicted “no default” for every applicant would achieve 2,184 / 2,400 = 91.0% accuracy while identifying zero defaulters, illustrating why accuracy is an unsuitable criterion for a 9% base-rate problem (Saito and Rehmsmeier, 2015). Precision and recall are therefore calculated instead:

Precision = TP / (TP + FP) = 151 / (151 + 219) = 151 / 370 = 0.408 (40.8%)

Recall (Sensitivity) = TP / (TP + FN) = 151 / (151 + 65) = 151 / 216 = 0.699 (69.9%)

The F1 score, the harmonic mean of precision and recall, was then calculated to summarise the trade-off in a single figure:

F1 = 2 × (Precision × Recall) / (Precision + Recall) = 2 × (0.408 × 0.699) / (0.408 + 0.699) = 2 × 0.2852 / 1.107 = 0.5704 / 1.107 = 0.515

The equivalent calculation for the logistic regression baseline, using the same 0.30 threshold and test set, produced TP = 128, FP = 246, FN = 88, TN = 1,938. Precision = 128 / 374 = 0.342, recall = 128 / 216 = 0.593, and F1 = 2 × (0.342 × 0.593) / (0.342 + 0.593) = 0.4057 / 0.935 = 0.434. The gradient-boosted model’s F1 score of 0.515 therefore represents a meaningful improvement over the logistic regression baseline’s 0.434, driven mainly by higher precision at an equivalent recall level – that is, the boosted model identifies a similar proportion of true defaulters while raising substantially fewer false alarms among genuinely creditworthy applicants.

Finally, the Matthews correlation coefficient (MCC), a metric considered particularly robust to class imbalance because it uses all four confusion matrix cells symmetrically (Chicco and Jurman, 2020), was calculated for the gradient-boosted model:

MCC = (TP×TN − FP×FN) / √[(TP+FP)(TP+FN)(TN+FP)(TN+FN)]
= (151×1,965 − 219×65) / √[(370)(216)(2,184)(2,030)]
= (296,715 − 14,235) / √[370 × 216 × 2,184 × 2,030]
= 282,480 / √(3.549 × 10¹¹)
= 282,480 / 595,700 ≈ 0.474

An MCC of 0.474 confirms a moderate-to-strong positive relationship between predicted and actual default status that is not an artefact of the class imbalance, corroborating the F1-based comparison and supporting the gradient-boosted model as the stronger of the two candidates on this test set. Figure 1 summarises the three headline metrics for both models side by side.

Examining the gradient-boosted model’s feature importances, using SHAP values as recommended by Lundberg and Lee (2017) for explaining tree-ensemble predictions, showed that debt-to-income ratio, credit utilisation and months since most recent delinquency were the three strongest predictors, together accounting for approximately 58% of total feature importance, broadly consistent with variables identified as influential in prior credit-scoring studies (Lessmann et al., 2015). Annual income and employment length contributed moderately, while loan purpose contributed only marginally once the other variables were accounted for, suggesting that behavioural and repayment-history variables carry more signal than static demographic or purpose fields in this dataset.

To connect the statistical comparison to business decision-making, an expected-cost calculation was also performed, translating each confusion-matrix cell into an approximate financial impact using illustrative cost assumptions of £3,000 average loss per missed default (false negative) and £150 average lost margin per wrongly declined creditworthy applicant (false positive), figures consistent in order of magnitude with loss-given-default estimates reported in retail lending literature (Crook, Edelman and Thomas, 2007). For the gradient-boosted model:

Expected cost = (FN × £3,000) + (FP × £150) = (65 × 3,000) + (219 × 150) = £195,000 + £32,850 = £227,850

The equivalent calculation for logistic regression, using its confusion matrix (FN = 88, FP = 246), gives:

Expected cost = (88 × 3,000) + (246 × 150) = £264,000 + £36,900 = £300,900

On this illustrative cost basis, the gradient-boosted model would be expected to save approximately £73,050 in expected losses relative to logistic regression across the 2,400 applications in the test set – a reduction of just over 24%, driven mainly by correctly identifying 23 more of the 216 true defaulters. This cost-based comparison reinforces the metric-based conclusion above, while making explicit that the practical significance of the performance gap depends heavily on the assumed cost ratio, which should be replaced with the lender’s own loss-given-default and margin figures before any deployment decision is finalised.

Evaluation

The worked comparison above shows a consistent, if not dramatic, advantage for the gradient-boosted model across F1 and MCC, both of which are more appropriate than accuracy or a naive ROC-AUC comparison for this class distribution. However, several evaluation limitations should qualify any deployment recommendation. First, the analysis reports performance at a single decision threshold (0.30); a full evaluation would examine the precision-recall curve across the full threshold range, since the relative cost of false positives and false negatives depends on business context that this assignment’s simulated dataset only approximates, and the optimal threshold could differ from 0.30 once real cost estimates are available.

Second, the gradient-boosted model’s superior discrimination comes with a well-documented interpretability cost relative to logistic regression (Lessmann et al., 2015). In UK consumer credit, the Consumer Credit Act 1974 and FCA (2022) Consumer Duty rules require lenders to be able to explain adverse credit decisions to applicants; a logistic regression model’s coefficients map directly onto explainable statements about which factors drove a decision, whereas the gradient-boosted ensemble requires post-hoc explanation methods such as SHAP values (Lundberg and Lee, 2017) to achieve comparable transparency. This is not a reason to reject the higher-performing model outright, but it is a material deployment cost that a purely metric-driven comparison omits.

Third, both models were evaluated on a single train-test split drawn from one historical period; a fuller evaluation would use out-of-time validation, testing the model on applications from a later period than the training data, since credit risk relationships can drift as macroeconomic conditions change (Crook, Edelman and Thomas, 2007). Without this check, there is a risk that both models’ apparent performance reflects patterns specific to the period sampled rather than stable, generalisable relationships. Fourth, fairness across protected characteristics was not assessed in this exercise; a production-grade evaluation would additionally test whether false negative and false positive rates differ materially across demographic groups, since a model can achieve strong aggregate metrics while performing unevenly across subgroups, a concern central to the FCA’s (2022) Consumer Duty and to broader algorithmic fairness literature (Barocas, Hardt and Narayanan, 2019).

A further consideration is probability calibration, which was not directly assessed by the threshold-based metrics reported above. Gradient-boosted models are known to produce predicted probabilities that can be poorly calibrated – systematically over- or under-confident relative to observed default rates – particularly when class weighting is applied during training, whereas logistic regression’s outputs are, by construction, well calibrated on the training distribution (Lessmann et al., 2015). This distinction matters in lending because predicted probabilities are often used directly in downstream pricing and provisioning models, not merely to produce a binary accept/decline decision at a single threshold. Before deployment, a reliability diagram comparing predicted probability deciles against observed default rates should be produced for the gradient-boosted model, with recalibration via Platt scaling or isotonic regression applied if systematic miscalibration is found.

Area under the ROC curve, though argued above to be less informative than precision-recall-based metrics for this class distribution, was nonetheless calculated as a supplementary check, since it remains widely reported in industry practice and allows comparison with published benchmarks. The gradient-boosted model achieved an AUC of 0.81 against 0.76 for logistic regression, a gap consistent with, though somewhat smaller than, the difference implied by the F1 and MCC comparisons; this is expected, since ROC-AUC evaluates ranking ability across all thresholds and is less sensitive than F1 or MCC to performance specifically in the region of the imbalanced minority class that matters most for this business problem (Saito and Rehmsmeier, 2015).

Conclusion and Recommendation

On the basis of the worked comparison, the gradient-boosted ensemble is recommended over logistic regression as the primary candidate for further development, given its materially higher F1 score (0.515 versus 0.434) and Matthews correlation coefficient (0.474), both calculated using metrics appropriate to the dataset’s 9% default rate rather than misleading accuracy figures. This recommendation is, however, conditional rather than final. Before deployment, three further steps are recommended: full threshold optimisation using business-specified cost weights for false positives and false negatives; out-of-time validation on a later applicant cohort to test temporal stability; and a subgroup fairness audit against protected characteristics, paired with SHAP-based explanation tooling to meet FCA transparency expectations. Logistic regression should be retained as an interpretable benchmark and fallback model throughout this process, since its transparent coefficient structure remains valuable for regulatory explainability even where it is not the primary decisioning model. In practice, a hybrid governance approach is often most defensible at this stage of model development: run the gradient-boosted model as the primary scoring engine while retaining logistic regression scores as a parallel reference, flag cases where the two models disagree substantially for manual underwriter review, and revisit the comparison once out-of-time data, subgroup fairness testing and calibration diagnostics are available. This staged approach captures the boosted model’s performance advantage identified above while managing the interpretability, stability and fairness risks that a purely metric-driven recommendation would otherwise overlook.

References

• Barocas, S., Hardt, M. and Narayanan, A. (2019) Fairness and Machine Learning: Limitations and Opportunities. Cambridge, MA: MIT Press.
• Chawla, N.V., Bowyer, K.W., Hall, L.O. and Kegelmeyer, W.P. (2002) ‘SMOTE: Synthetic minority over-sampling technique’, Journal of Artificial Intelligence Research, 16, pp. 321-357.
• Chicco, D. and Jurman, G. (2020) ‘The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation’, BMC Genomics, 21, article 6.
• Crook, J.N., Edelman, D.B. and Thomas, L.C. (2007) ‘Recent developments in consumer credit risk assessment’, European Journal of Operational Research, 183(3), pp. 1447-1465.
• Financial Conduct Authority (2022) PS22/9: A New Consumer Duty. London: FCA.
• He, H. and Garcia, E.A. (2009) ‘Learning from imbalanced data’, IEEE Transactions on Knowledge and Data Engineering, 21(9), pp. 1263-1284.
• Lessmann, S., Baesens, B., Seow, H.V. and Thomas, L.C. (2015) ‘Benchmarking state-of-the-art classification algorithms for credit scoring: An update of research’, European Journal of Operational Research, 247(1), pp. 124-136.
• Lundberg, S.M. and Lee, S.I. (2017) ‘A unified approach to interpreting model predictions’, Advances in Neural Information Processing Systems, 30, pp. 4765-4774.
• Saito, T. and Rehmsmeier, M. (2015) ‘The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets’, PLOS ONE, 10(3), e0118432.