Statistical Analysis Sample: One-Way ANOVA Comparing Three Training Methods

Model Answer

Introduction and Research Question

Organisations investing in sales-training programmes need evidence on which delivery format produces the strongest post-training performance before committing budget at scale. This analysis addresses a scenario in which a UK retail employer trialled three training interventions for a new product line with newly recruited sales staff: self-paced e-learning modules, traditional classroom instruction, and a blended format combining short e-learning modules with one-to-one manager coaching. All employees completed the same post-training assessment, a simulated sales scenario scored out of 100 by trained assessors using a standardised rubric, allowing performance to be compared directly across the three groups.

The organisational context motivating this pilot is a common one in UK retail: the employer had recently launched a new, technically complex product line and needed to bring a cohort of 60 newly recruited sales staff up to a consistent standard of product knowledge and consultative selling skill within a compressed six-week onboarding window, across several regional offices. Historically, the employer had relied almost exclusively on classroom instruction delivered by regional trainers, but rising headcount and geographic dispersion made this format increasingly expensive and difficult to schedule consistently, prompting the training and people analytics team to pilot e-learning and a blended coaching alternative before recommending a single default format for the wider organisation. This analysis was commissioned specifically to give the training team statistically defensible evidence, rather than anecdotal manager impression, on which format to scale.

The research question is: does post-training performance differ significantly according to the training method received? The null hypothesis (H₀) states that the population mean performance score is equal across all three training methods. The alternative hypothesis (H₁) states that at least one population mean differs from the others. A one-way, between-subjects analysis of variance (ANOVA) is the appropriate test because the independent variable, training method, is categorical with three levels, the dependent variable, performance score, is continuous, and each employee received only one training method, satisfying the requirement for independent groups.

ANOVA was selected over multiple separate t-tests because running three pairwise t-tests to compare three groups would inflate the family-wise Type I error rate; ANOVA instead tests the overall difference across all groups in a single omnibus test, with follow-up post-hoc comparisons used only if that omnibus test is significant (Field, 2018).

This analysis sits within Kirkpatrick and Kirkpatrick’s (2006) well-established four-level training evaluation model, which distinguishes reaction (how trainees felt about the training), learning (measurable knowledge or skill gain), behaviour (transfer to the workplace) and results (impact on business outcomes). The post-training performance score analysed here corresponds to Kirkpatrick’s learning level: a controlled, standardised measure of skill acquisition immediately after training, rather than a direct measure of subsequent sales behaviour or revenue impact. This distinction matters for how the findings should be used: a significant difference at the learning level provides useful, decision-relevant evidence for comparing training formats, but it is not, on its own, proof that the same ranking of formats would hold for on-the-job sales results, a point returned to in the limitations section below. The significance threshold for all inferential tests in this analysis was set at the conventional α = .05.

Data and Variables

The dataset comprises post-training performance scores for 60 newly recruited sales employees, randomly allocated in equal numbers (n = 20 per group) to one of the three training conditions as part of the employer’s internal training pilot, which supports comparability across groups by removing the self-selection concern that would arise if employees had chosen their own training route. Table 1 summarises the variables entered into SPSS.

The dataset was screened prior to analysis using SPSS frequencies and a boxplot of performance score split by training method. No missing data were present, and no case exceeded a standardised score of ±3.29, so no cases were treated as extreme outliers or removed (Field, 2018; Tabachnick and Fidell, 2019).

Random allocation to condition was achieved using a computer-generated randomisation sequence stratified by the employee’s prior sales experience band (none, under one year, one year or more), so that each training group contained a comparable mix of experience levels; this stratification reduces the risk that any observed difference between groups reflects pre-existing experience rather than the training intervention itself. The sample size of 20 per group was set following an a priori power analysis in G*Power, which indicated that a total sample of 60 (n = 20 per group) would provide approximately 80% power to detect a large effect (f = 0.40) at α = .05 for a three-group one-way ANOVA, meaning the study was adequately, though not generously, powered to detect an effect of the size ultimately observed. The post-training performance assessment was a fifteen-minute simulated sales scenario, scored against a ten-item standardised rubric by two independent assessors who were blind to each employee’s training condition; inter-rater agreement between the two assessors was high (intraclass correlation = .89), and the average of the two assessors’ scores was used as the final performance score entered into SPSS.

Preliminary Checks and Assumptions

One-way ANOVA rests on four assumptions: independence of observations, a continuous dependent variable, approximate normality of the dependent variable within each group, and homogeneity of variance across groups. Independence was satisfied by the between-subjects design, since each employee appears in exactly one training condition. Performance score is continuous by definition.

Normality within each group was assessed using the Shapiro–Wilk test, appropriate given the group sizes of 20 (Pallant, 2020). Table 2 reports the results.

None of the three tests was significant at the .05 level (all p > .05), so the assumption of normality was retained for all three groups, further supported by broadly symmetric boxplots for each condition; group sizes of 20 also fall within the range Howell (2017) identifies as adequate for the F-test to remain robust to minor departures from normality provided variances are reasonably homogeneous, as confirmed below.

Homogeneity of variance was assessed using Levene’s test, reported in Table 3.

Levene’s test was not significant, F(2, 57) = 0.42, p = .659, so the assumption of equal population variances was retained and the standard (Fisher) F-statistic, rather than a Welch-adjusted alternative, was used to report the ANOVA result. Had Levene’s test been significant, the appropriate remedy would have been to report the Welch F-ratio in place of the standard F-ratio, and to substitute the Games–Howell post-hoc procedure for Tukey’s HSD, since Games–Howell does not assume equal variances or equal group sizes; because both assumptions were satisfied here, the more familiar Fisher-and-Tukey combination was appropriate and is reported throughout. Independence of observations across the three groups was further supported by the study design: employees worked in different regional offices during the training period and were assessed individually rather than as intact teams, which reduces the risk of clustering effects that might otherwise inflate the apparent precision of the group means.

Test Results

Table 4 presents descriptive statistics for post-training performance score by training method.

Mean performance rose across the three conditions, from e-learning (M = 72.5, SD = 8.1) through classroom instruction (M = 76.8, SD = 7.4) to blended coaching (M = 81.3, SD = 6.9). Table 5 presents the ANOVA summary table in the standard layout recommended for student reporting by Laerd Statistics (2021), as it would appear in SPSS output.

The omnibus test was statistically significant, F(2, 57) = 6.92, p = .002, indicating that post-training performance differed significantly across at least one pair of training methods. Conceptually, the F-ratio expresses how much of the total variation in performance scores sits between the three group means, relative to how much variation sits within each group around its own mean; an F-ratio well above 1, as observed here, indicates that the groups differ from one another by more than would be expected from within-group measurement noise alone. The effect size, eta squared, was calculated as the between-groups sum of squares divided by the total sum of squares (774.53 ÷ 3966.15 = 0.195), indicating a large effect by Cohen’s (1988) conventions for eta squared, where values above approximately .14 are considered large.

Because the omnibus test was significant, Tukey’s Honestly Significant Difference (HSD) post-hoc test was run to identify which specific group pairs differed, controlling the family-wise error rate across all three pairwise comparisons (Tukey, 1949). Tukey’s HSD was preferred here over a Bonferroni correction because Bonferroni tends to be conservative as the number of comparisons grows, increasing the risk of a Type II error, whereas Tukey’s procedure is specifically designed for the case of pairwise comparisons following a significant one-way ANOVA with equal group sizes, offering better statistical power while still controlling the family-wise error rate at .05 (Field, 2018). Table 6 reports the results.

The post-hoc comparisons show that only the difference between e-learning and blended coaching was statistically significant (mean difference = −8.80, p = .002, 95% CI [−14.49, −3.11]); neither the e-learning versus classroom comparison nor the classroom versus blended-coaching comparison reached significance, and both associated confidence intervals span zero.

Figure 1 displays the three group means with standard error bars, illustrating the clear separation between e-learning and blended coaching alongside the overlapping intervals for the classroom condition, consistent with the post-hoc pattern reported above.

Interpretation

Consistent with American Psychological Association (2020) reporting conventions, the result is summarised below as a single statistical sentence followed by a plain-English explanation for a non-specialist audience. A one-way ANOVA was conducted to compare the effect of training method (e-learning, classroom, blended coaching) on post-training performance score. There was a statistically significant effect of training method on performance, F(2, 57) = 6.92, p = .002, η² = .20. Tukey post-hoc comparisons indicated that mean performance for the blended-coaching group (M = 81.3, SD = 6.9) was significantly higher than for the e-learning group (M = 72.5, SD = 8.1), a mean difference of 8.8 marks (p = .002). The classroom group (M = 76.8, SD = 7.4) did not differ significantly from either the e-learning group (p = .294) or the blended-coaching group (p = .241).

In practical terms, this pattern suggests that adding structured, one-to-one manager coaching to a short e-learning programme produces a meaningfully larger performance gain than either format alone appears to achieve, while the fully classroom-based format sits statistically between the two without being clearly distinguishable from either. For an HR audience, the headline finding is not simply that ‘training method matters’, but specifically that the coaching component of the blended format is the element most strongly associated with the observed performance advantage, since blended coaching is the only condition that differs significantly from the lowest-performing group.

The large effect size (η² = .20) indicates that training method accounts for a substantial proportion, around one-fifth, of the variation in post-training performance scores observed in this sample, which is a practically meaningful effect for a single, modifiable input into an employer’s training design, even though the majority of variation in performance remains attributable to other factors not captured by training method alone.

This pattern is consistent with the training and development literature on the value of individualised feedback and deliberate practice. Noe (2020) argues that coaching adds value beyond content delivery because it provides immediate, personalised feedback on performance and allows a trainee’s specific weaknesses to be targeted directly, something that a standardised e-learning module or a one-to-many classroom session cannot easily replicate. This is consistent with deliberate-practice research more broadly, which emphasises that skill improvement depends less on repeated exposure to content and more on targeted feedback against a specific performance gap, precisely the mechanism that a manager coach, but not a static e-learning module, is positioned to provide during onboarding. Viewed through this lens, the present results are consistent with the interpretation that it is the individualised feedback loop of coaching, rather than content exposure alone, that drives the performance advantage, since e-learning and classroom instruction primarily differ from one another in delivery format and pacing rather than in the presence or absence of individualised feedback, and it is precisely this pair that fails to reach statistical significance.

From a cost-benefit perspective relevant to the employer commissioning this analysis, one-to-one coaching is typically the most resource-intensive of the three formats to deliver at scale, since it requires trained manager time on a one-to-one basis rather than a single facilitator reaching a full cohort simultaneously. The results reported here provide evidence that this additional investment is associated with a statistically and practically meaningful performance advantage over e-learning alone, but they do not, on their own, establish whether that advantage justifies the additional cost once delivered at the scale of a full national sales-force rollout; a formal cost-per-percentage-point-of-improvement analysis, sitting outside the scope of this statistical report, would be a natural next step for the HR analytics team.

Conclusion and Limitations

This analysis found a statistically significant and practically large effect of training method on post-training sales performance, F(2, 57) = 6.92, p = .002, η² = .20, with Tukey post-hoc testing indicating that blended coaching produced significantly stronger performance than e-learning alone, while classroom instruction was not clearly distinguishable from either alternative. The assumptions of normality and homogeneity of variance were both satisfied, supporting the validity of the standard ANOVA and Tukey HSD results reported above.

Several limitations should be considered when applying these findings. First, and returning to the Kirkpatrick framework introduced earlier, the assessment was conducted immediately after training and measures learning rather than sustained on-the-job behaviour or business results; a simulated sales scenario score, however well validated, may not translate directly into real-world sales performance once employees return to the variability of live customer interactions, seasonal demand and individual account portfolios. Second, the sample was drawn from a single employer, a single product line and a single geographic market, limiting the generalisability of both the ranking of training methods and the size of the observed effect to other sectors, products or sales contexts, where the relative value of coaching versus content delivery may differ. Third, while random allocation strengthens internal validity by balancing prior experience across groups, the blended condition combined two active ingredients, e-learning content and manager coaching, so this design cannot isolate which of the two components, or their interaction, drives the observed advantage over e-learning alone.

A fourth limitation concerns statistical power for the specific pairwise comparisons rather than the omnibus test. Although the overall study was powered to detect a large omnibus effect, the individual pairwise comparisons inevitably carry somewhat less power once the family-wise error correction inherent in Tukey’s procedure is applied, which may partly explain why the classroom condition, sitting numerically between the other two groups, did not reach significance against either comparator despite a difference of four to five marks in each case; a larger per-group sample would allow a more confident conclusion as to whether classroom instruction is genuinely equivalent to e-learning and blended coaching, or simply insufficiently powered to detect a true, smaller difference. A factorial design that varied e-learning and coaching independently, together with a longer-term, on-the-job follow-up assessment linked to actual sales outcomes, would allow considerably firmer conclusions to be drawn about which specific element of blended training employers should prioritise when designing future programmes at scale.

References

American Psychological Association (2020) Publication Manual of the American Psychological Association. 7th edn. Washington, DC: APA.

Cohen, J. (1988) Statistical Power Analysis for the Behavioral Sciences. 2nd edn. Hillsdale, NJ: Lawrence Erlbaum Associates.

Field, A. (2018) Discovering Statistics Using IBM SPSS Statistics. 5th edn. London: Sage.

Howell, D.C. (2017) Statistical Methods for Psychology. 9th edn. Boston, MA: Cengage Learning.

Kirkpatrick, D.L. and Kirkpatrick, J.D. (2006) Evaluating Training Programs: The Four Levels. 3rd edn. San Francisco, CA: Berrett-Koehler.

Laerd Statistics (2021) One-Way ANOVA Using SPSS Statistics. Lund Research Ltd.

Noe, R.A. (2020) Employee Training and Development. 8th edn. New York, NY: McGraw-Hill.

Pallant, J. (2020) SPSS Survival Manual. 7th edn. Maidenhead: Open University Press.

Tabachnick, B.G. and Fidell, L.S. (2019) Using Multivariate Statistics. 7th edn. Boston, MA: Pearson.

Tukey, J.W. (1949) ‘Comparing individual means in the analysis of variance’, Biometrics, 5(2), pp. 99–114.