Machine Learning Network Intrusion Detection Literature Review

1. Introduction

Network intrusion detection systems (NIDS) sit at the centre of modern defensive security architectures, monitoring traffic for evidence of malicious or anomalous activity. Traditional detection has relied on two broad strategies: signature-based detection, which matches observed traffic against a database of known attack patterns, and anomaly-based detection, which models normal behaviour and flags deviations from it. Signature-based systems are accurate against known threats but are, by design, blind to novel or obfuscated attacks. Anomaly-based systems promise to catch previously unseen attacks but have historically suffered from high false-positive rates that erode analyst trust. The growth in both the volume and sophistication of network attacks has intensified interest in machine learning (ML) as a means of improving anomaly detection, because ML models can, in principle, learn complex and evolving representations of both benign and malicious behaviour from data.

The operational context sharpens this interest. Enterprise and service-provider networks now carry traffic at volumes and speeds that make manual inspection impossible, and the attack surface has broadened to include cloud workloads, encrypted channels and large populations of resource-constrained devices. Security operations centres are, at the same time, under well-documented pressure from alert fatigue, in which analysts are presented with more alerts than they can triage and learn to distrust noisy detectors. An effective ML-based detector must therefore not only discriminate attacks from benign activity accurately, but do so at line rate and with a false-positive rate low enough to preserve analyst trust. These competing demands – accuracy, speed and precision – frame the way the literature should be read and evaluated.

Over the past decade the literature on ML-based intrusion detection has expanded rapidly, spanning classical supervised classifiers such as support vector machines (SVM), decision trees and random forests, through to deep neural architectures including convolutional neural networks (CNNs), recurrent networks such as long short-term memory (LSTM), and unsupervised models such as autoencoders. This proliferation has produced an abundance of empirical results, but also considerable fragmentation: studies use different datasets, different pre-processing pipelines, and different evaluation metrics, which makes direct comparison difficult and claims of state-of-the-art performance hard to interpret. A systematic literature review (SLR) is therefore valuable, because it imposes a transparent and repeatable method for locating, selecting and synthesising the evidence, and because it surfaces methodological weaknesses that narrative reviews often miss. Where a narrative review risks reflecting the author’s prior expectations, a systematic review makes its selection logic auditable and its conclusions traceable to a defined evidence base.

This review adopts a systematic approach informed by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, adapted for a computing context. The central aim is to characterise the state of ML-based network intrusion detection research, to compare the performance and limitations of competing families of techniques, and to identify gaps that merit further study. To structure the enquiry, the review is guided by three research questions:

• RQ1: Which machine learning techniques have been applied to network intrusion detection, and how are they categorised?
• RQ2: What datasets, features and evaluation metrics are used to assess these techniques, and how comparable are the reported results?
• RQ3: What are the principal methodological limitations and open research gaps in the current body of evidence?

The remainder of the review is organised as follows. Section 2 sets out the review methodology, including the databases searched, the search strings, the inclusion and exclusion criteria, the study-selection process and the quality-assessment procedure. Section 3 presents a study-synthesis matrix summarising the included primary studies. Section 4 synthesises the findings thematically across four themes. Section 5 offers a critical appraisal of the evidence base. Section 6 identifies research gaps and future directions, and Section 7 concludes. All numerical figures describing the search process and reported study performance in this model review are illustrative, included to demonstrate the structure and reasoning of an SLR rather than to report a specific real-world dataset.

2. Review Methodology and Search Strategy

A systematic review differs from a traditional narrative review in that its method is specified in advance and documented in enough detail to be reproduced by another researcher. This section describes the protocol followed here. The guiding principle is transparency: every decision about where to search, what to include and how to judge quality is made explicit, so that the reader can assess whether the conclusions are supported by a defensible evidence base.

The protocol was defined before the search was executed and comprised five stages: formulating the research questions, specifying the sources and search strings, applying the inclusion and exclusion criteria, extracting data into a common template, and appraising study quality. Defining the protocol in advance is important because it guards against the temptation to adjust the criteria after seeing the results in order to favour a particular conclusion. In a full review the protocol would typically be registered or published so that any later deviation is visible; in this model exercise the stages are simply reported in full. Data extraction used a structured template capturing the technique family, dataset, pre-processing steps, train-test procedure, metrics reported and stated limitations, which is the template that later populates the synthesis matrix in Section 3.

2.1 Databases and Search Terms

Five bibliographic sources commonly used in computer science were searched: IEEE Xplore, the ACM Digital Library, ScienceDirect, SpringerLink and the Scopus index. These sources were selected because they provide broad coverage of the peer-reviewed security and machine-learning venues in which NIDS research is typically published. A Boolean search string was constructed by combining terms describing the problem domain with terms describing the technique. The core string, adapted to the syntax of each database, was:

(“intrusion detection” OR “anomaly detection” OR “network security”) AND (“machine learning” OR “deep learning” OR “neural network” OR “classifier”) AND (“dataset” OR “evaluation” OR “detection rate”)

The search was restricted to peer-reviewed journal articles and full conference papers published between January 2015 and December 2024, in English. The table below summarises the search configuration and the illustrative number of records returned by each source before de-duplication.

2.2 Inclusion and Exclusion Criteria

Explicit inclusion and exclusion criteria are the defining feature of a systematic review. They convert an otherwise subjective judgement about relevance into a set of rules that can be applied consistently and audited afterwards. A study was included if it satisfied all of the inclusion criteria and none of the exclusion criteria set out below.

Inclusion criteria:

1. The study applies one or more machine learning techniques to the task of network intrusion or anomaly detection.
2. The study reports an empirical evaluation on at least one network-traffic dataset, with quantitative performance results.
3. The study is a peer-reviewed journal article or full conference paper published between 2015 and 2024.
4. The full text is available in English.

Exclusion criteria:

1. Studies addressing host-based intrusion detection or malware classification only, with no network-traffic component.
2. Short papers, posters, editorials, keynote abstracts and non-peer-reviewed preprints.
3. Secondary studies (existing reviews and surveys), which were retained separately for background and snowballing but excluded from the primary-study synthesis.
4. Studies that describe a conceptual framework without any empirical evaluation.
5. Duplicate publications of the same study, where the most complete version was retained.

2.3 Study Selection and the PRISMA Flow

The selection process followed the four PRISMA stages of identification, screening, eligibility and inclusion. Of the 1,284 records identified, 322 were removed as duplicates, leaving 962 unique records. Title and abstract screening against the criteria above removed a further 806 records that were clearly out of scope, leaving 156 for full-text assessment. At the eligibility stage, 124 full texts were read in detail; the shortfall from 156 reflects records for which the full text could not be retrieved or which, on closer reading, proved to be secondary studies. Of these, 92 were excluded for failing one or more criteria – most commonly the absence of a network-traffic dataset or the lack of a reproducible quantitative evaluation. This yielded a final set of 32 primary studies for synthesis. Backward snowballing of the reference lists of the included studies and of the retained surveys identified no additional eligible primary studies that were not already captured. Screening was performed by a single reviewer in this model exercise; in a full review, a second independent reviewer and a measure of inter-rater agreement, such as Cohen’s kappa, would be used to reduce selection bias.

2.4 Quality Assessment of Included Studies

Each included study was appraised against a short quality checklist adapted for empirical machine-learning research. The checklist asked whether the study (a) clearly stated its aims and the attack types addressed; (b) described the dataset and any pre-processing, including how class imbalance was handled; (c) specified the training and testing split or cross-validation procedure; (d) reported metrics beyond accuracy, such as precision, recall and the F1-score; and (e) discussed limitations and threats to validity. Each criterion was scored 1 (met), 0.5 (partially met) or 0 (not met), giving a maximum score of five. Studies scoring below two were not excluded outright but were weighted less heavily in the synthesis. The assessment revealed that reporting quality was uneven: a majority of studies reported accuracy and detection rate, but a substantial minority did not report precision or recall, and fewer than half explicitly addressed class imbalance despite using datasets in which attack classes are heavily under-represented.

3. Study Synthesis Matrix

The study-synthesis matrix is the analytical heart of a systematic review. It reduces each primary study to a common set of attributes so that patterns and contrasts become visible across the corpus. The table below presents a representative subset of the 32 included studies, organised by the family of technique applied. Reported performance figures are illustrative of the ranges observed in the corpus and are provided to demonstrate how such a matrix is populated and read.

Even at a glance, the matrix reveals two patterns that recur throughout the corpus. First, reported performance clusters at a very high level, with F1-scores frequently above 0.95, which raises the question of whether the benchmark datasets are now too easy to discriminate the genuinely better methods. Second, the limitation column is dominated by concerns about evaluation design – single-dataset testing, possible data leakage, and neglect of class imbalance – rather than by concerns about the core modelling. These patterns frame the thematic synthesis that follows.

4. Thematic Synthesis

The 32 included studies were grouped inductively into four themes, derived by clustering studies that shared a common modelling philosophy or methodological concern. The themes are not mutually exclusive – a hybrid model may draw on several – but they provide a useful organising structure for comparing the evidence against the research questions.

4.1 Classical Supervised Learning Approaches

The largest group of studies applied classical supervised classifiers, including SVM, k-nearest neighbours, decision trees, random forests and gradient-boosted ensembles. These techniques dominate the earlier part of the review window and remain a strong baseline throughout. Their appeal is threefold. They are comparatively cheap to train, they produce models that are at least partly interpretable, and tree-based ensembles in particular cope well with the mixed categorical and numerical features typical of network-flow records. Across the corpus, random forests and gradient-boosting methods consistently produced the strongest results among the classical family, with illustrative F1-scores in the 0.94–0.98 range on the more modern datasets.

Within this family, a consistent ordering emerged. Single decision trees and k-nearest-neighbour classifiers were generally the weakest, SVMs occupied the middle ground and were sensitive to kernel and regularisation choices, and ensemble methods – bagged random forests and boosted trees – were the strongest and most stable. Several studies paired these classifiers with explicit feature-selection techniques such as information-gain ranking, recursive feature elimination or correlation-based selection, both to reduce the dimensionality of flow records and to improve training speed. The reported benefit of feature selection was real but modest, and its main value lay in producing smaller, faster models rather than in large accuracy gains.

However, the synthesis also exposes recurring weaknesses in how these studies are evaluated. A common problem is reliance on overall accuracy as the headline metric. Because benign traffic vastly outnumbers attack traffic in realistic settings, a classifier can achieve high accuracy while missing a large fraction of the attacks that matter most. Studies that reported per-class recall revealed that classical classifiers performed well on frequent attack categories, such as denial-of-service, but markedly worse on rare categories, such as remote-to-local or user-to-root exploits. This confirms that the aggregate figures in the matrix mask substantial variation in detection quality across attack types. Feature engineering also plays an under-acknowledged role: several studies achieved their strongest results only after extensive manual feature selection, which limits the transferability of the reported performance to other environments. A model tuned to the specific feature distribution of one benchmark dataset may generalise poorly to a network whose services, protocols and baseline behaviour differ, a concern that recurs throughout the later themes.

4.2 Deep Learning and Representation Learning

The second theme comprises studies using deep neural architectures, including multilayer perceptrons, CNNs, recurrent models such as LSTM and gated recurrent units, and more recent hybrid and graph-based designs. The motivation is that deep models can learn useful feature representations automatically from raw or lightly processed traffic, reducing the dependence on manual feature engineering that constrains classical approaches. The reviewed studies report that CNNs are effective at capturing local patterns when flow features are arranged into a grid or image-like structure, while recurrent models are used to exploit the temporal ordering of packets or flows within a session. Hybrid CNN-LSTM designs, which combine spatial and temporal modelling, reported some of the highest F1-scores in the corpus.

Toward the end of the review window, a smaller number of studies began to apply attention-based and transformer-style architectures, which model long-range dependencies between flows without the sequential bottleneck of recurrent networks, and to experiment with transfer learning, in which a model pre-trained on one traffic source is fine-tuned on another. These approaches are promising in principle, because transfer learning directly targets the poor cross-environment generalisation that limits earlier methods. In practice, however, the reviewed studies were preliminary, often evaluated on a single dataset, and did not yet provide convincing evidence that the added architectural complexity is justified by a reliable improvement in detection.

Yet the evidence for a decisive advantage of deep learning over well-tuned classical ensembles is weaker than the headline figures suggest. When studies compared architectures on the same dataset and split, the margin between a strong gradient-boosting baseline and a deep model was frequently small, sometimes within the range of ordinary run-to-run variance. Deep models also carry costs that the reviewed studies tend to underplay: higher computational and memory demands that complicate real-time deployment, sensitivity to hyper-parameter choices, and reduced interpretability, which matters for security analysts who must justify and act on alerts. Only a minority of the deep-learning studies reported training time, inference latency or model size, which makes it difficult to judge operational feasibility. The emerging use of graph neural networks, which model hosts and flows as nodes and edges, is promising for capturing relational structure in traffic, but the reviewed studies acknowledged unresolved questions about the cost of graph construction and scalability to high-throughput networks. There is, moreover, a recurring absence of strong, identically tuned baselines: without a well-optimised classical comparator, it is hard to attribute a deep model’s performance to its architecture rather than to greater tuning effort, and several studies in the corpus did not make this comparison fair.

4.3 Unsupervised and Semi-Supervised Anomaly Detection

A third theme addresses the fundamental limitation shared by all supervised approaches: they can only learn to recognise attack types that are represented, and labelled, in the training data. Because novel attacks are precisely those for which no labels exist, several studies turned to unsupervised and semi-supervised methods. Unsupervised approaches, principally autoencoders, one-class SVMs, isolation forests and clustering, learn a model of normal traffic and flag inputs that reconstruct poorly, fall in low-density regions or lie far from established clusters. Semi-supervised approaches, including generative adversarial networks (GANs) and self-training schemes, aim to exploit a small quantity of labelled data alongside a large pool of unlabelled traffic.

The synthesis indicates that these methods are conceptually better aligned with the goal of detecting zero-day attacks, but that they pay for this generality with lower and less stable performance. Autoencoder-based detectors in the corpus reported illustrative F1-scores around 0.89, noticeably below the supervised ensembles, and their performance was highly sensitive to the choice of anomaly threshold. Several studies noted that the distinction between benign-but-rare traffic and genuinely malicious traffic is inherently difficult to draw from reconstruction error alone, which drives false positives. GAN-based semi-supervised methods reported competitive results in favourable conditions but were repeatedly described as unstable to train and difficult to reproduce. A further, under-examined problem is the purity of the training data: unsupervised detectors assume that the baseline traffic used to learn “normal” is free of attacks, yet real captures are rarely clean, and contamination of the baseline quietly teaches the model to treat some malicious behaviour as normal. The broader lesson is that the choice between supervised and unsupervised paradigms is not simply a question of accuracy but of the operating assumption: supervised methods excel at the known, while unsupervised methods trade raw performance for the possibility of catching the unknown.

4.4 Datasets, Evaluation and Reproducibility

The fourth theme cuts across the others and concerns the empirical foundations on which all the reported results rest. The corpus is dominated by a small number of benchmark datasets. Early studies relied heavily on KDD Cup 99 and its cleaned successor NSL-KDD; more recent work has moved to UNSW-NB15, CICIDS2017 and CSE-CIC-IDS2018, which contain more modern attack types and more realistic traffic. This migration is a genuine improvement, but the reliance on a handful of public datasets creates its own risks. When the same datasets are used repeatedly to develop and evaluate successive models, there is a danger of collective overfitting to the idiosyncrasies of those datasets, so that gains reported in the literature do not transfer to live networks.

The choice of evaluation metric is equally consequential and unevenly handled. Under the heavy class imbalance typical of intrusion data, accuracy and even the area under the receiver-operating-characteristic curve can give a flattering picture, because both are dominated by the abundant benign class. Metrics that focus on the positive (attack) class – precision, recall, the F1-score and the area under the precision-recall curve – are far more informative, as is an explicit statement of the false-positive rate, which determines the operational burden a detector places on analysts. The reviewed studies varied widely in which of these they reported, and a minority reported accuracy alone, which is close to uninformative in this setting. The level at which traffic is represented – individual packets, aggregated flows or time windows – also differed between studies and materially affects both performance and comparability, yet was not always stated clearly.

Three further methodological problems emerged from the synthesis. First, data leakage: several studies appear to have performed normalisation, feature selection or re-sampling before splitting the data, allowing information from the test set to influence training and inflating reported performance. Second, class imbalance: although the benchmark datasets contain severely under-represented attack classes, fewer than half of the studies explicitly described how imbalance was handled, and some reported only aggregate accuracy, which is uninformative under imbalance. Third, limited external validity: the large majority of studies evaluated on a single dataset, and the minority that tested across datasets usually reported a marked drop in performance when a model trained on one dataset was applied to another. Reproducibility was also uneven: code and exact hyper-parameters were released in only a minority of cases, which obstructs independent verification. Together these observations suggest that the apparent maturity of the field, implied by uniformly high F1-scores, is partly an artefact of how it is evaluated.

5. Critical Appraisal

Drawing the themes together, the body of evidence on ML-based network intrusion detection is large, active and technically sophisticated, but its collective reliability is constrained by evaluation practices rather than by modelling capability. In answer to RQ1, the literature can be organised into classical supervised classifiers, deep and representation-learning models, and unsupervised or semi-supervised anomaly detectors, with a clear chronological drift from the first towards the second and third. In answer to RQ2, evaluation rests on a narrow set of benchmark datasets and reports a limited and inconsistent range of metrics, which makes cross-study comparison hazardous and means that reported state-of-the-art claims should be treated with caution.

The most important critical observation is that very high reported performance is not, on its own, evidence of practical effectiveness. Several factors inflate the figures: evaluation on datasets that may no longer be challenging, occasional data leakage, and the use of accuracy on imbalanced data. Where studies used stricter protocols – proper pipeline isolation, per-class metrics and cross-dataset testing – the picture was more sober, with smaller and less certain differences between methods. This is a classic instance of the gap between benchmark performance and deployment performance that affects many applied machine-learning fields, and it suggests that the marginal research contribution of yet another model that reports a near-perfect F1-score on a familiar benchmark is now small. The field appears to be reaching the ceiling of what its established benchmarks can measure.

Placing this review against the existing secondary literature reinforces the point. Earlier surveys in the field have tended to catalogue techniques and their reported accuracies, implicitly treating higher numbers as better. By foregrounding evaluation design instead, this synthesis arrives at a more cautious reading of the same primary results, and it illustrates why a systematic method matters: the conclusion follows from criteria applied consistently across the corpus rather than from an impression formed by the most prominent papers.

It is also important to acknowledge the limitations of this review itself, as a matter of methodological honesty. The search was restricted to five databases, to the English language and to a ten-year window, so relevant work in other languages, in the grey literature or in older foundational papers may have been missed. Screening and quality assessment were carried out by a single reviewer in this model exercise, which introduces a risk of selection bias that a second independent reviewer would mitigate. No formal meta-analysis was attempted, because the heterogeneity of datasets, metrics and splits across the included studies makes the quantitative pooling of results inappropriate; a narrative synthesis is the defensible choice under these conditions. These limitations temper the strength of the conclusions, though they do not overturn the central finding that evaluation practice, rather than algorithmic novelty, is the principal weakness of the field.

6. Research Gaps and Future Directions

In answer to RQ3, the synthesis points to several gaps that represent productive directions for future research. Each follows directly from a weakness identified in the thematic synthesis and critical appraisal.

• Standardised, realistic and current benchmarks. The field would benefit from newer datasets that reflect contemporary attack techniques and realistic traffic distributions, together with an agreed evaluation protocol specifying splits, metrics and reporting requirements. This would reduce collective overfitting and make cross-study comparison meaningful.
• Robust cross-dataset and temporal evaluation. Future studies should test models on datasets other than the one used for training and, where possible, on traffic captured at a later time, to measure how performance degrades under distribution shift and concept drift. Reporting this degradation is more informative than a single headline F1-score.
• Operational cost and real-time feasibility. Few studies report inference latency, memory footprint or throughput. Research that treats these as first-class evaluation criteria, alongside detection quality, would better serve practitioners who must deploy detectors on live, high-volume networks.
• Interpretability and analyst trust. Because security analysts must act on and justify alerts, methods that explain their decisions – through feature attribution or rule extraction – deserve more attention, particularly for deep models whose opacity is a barrier to adoption.
• Adversarial robustness. Attackers can deliberately craft traffic to evade ML detectors. The reviewed corpus largely overlooked adversarial evaluation, yet a detector’s resistance to evasion is arguably as important as its accuracy on benign test data.
• Detection of attacks in encrypted traffic. As encryption becomes near-universal, payload inspection is increasingly impossible, so future work must establish how reliably attacks can be detected from flow metadata and timing alone, without decrypting traffic.
• Reproducibility as standard. Routine release of code, trained models and exact configurations would allow independent verification and accelerate cumulative progress, addressing the reproducibility gap identified in the synthesis.

A particularly promising convergence would combine the generalisation of unsupervised anomaly detection with the precision of supervised classification in hybrid architectures, evaluated under the stricter protocols advocated above. Such work would address both the technical limitation of detecting novel attacks and the methodological limitation of unreliable evaluation.

7. Conclusion

This systematic review set out to characterise the state of machine learning research on network intrusion detection, to compare the competing families of techniques, and to identify open gaps. Following a PRISMA-informed protocol, 32 primary studies were synthesised across four themes: classical supervised classifiers, deep and representation-learning models, unsupervised and semi-supervised anomaly detectors, and the datasets and evaluation practices that underpin them all. The evidence shows a field that has advanced rapidly in technical sophistication, with tree-based ensembles and deep hybrid models reporting very high performance on standard benchmarks.

The more important conclusion, however, concerns the quality of the evidence rather than the ranking of the methods. High reported performance is undercut by a reliance on a small set of benchmark datasets, inconsistent metrics, occasional data leakage, neglected class imbalance and limited cross-dataset testing. The practical effectiveness of ML-based intrusion detection therefore remains less certain than the headline figures imply. Progress will depend less on new algorithms and more on stronger evaluation: realistic and current benchmarks, robust cross-dataset and adversarial testing, attention to operational cost and interpretability, and reproducibility as a default expectation. For the Masters-level reader, the review also illustrates the mechanics of systematic enquiry in computer science – transparent search, explicit inclusion and exclusion criteria, a structured synthesis matrix and critical appraisal – which are transferable to any technical literature review.

References

Abidin, S. and Carter, J. (2019) ‘A comparative study of supervised learning algorithms for network intrusion detection’, Journal of Network Security Research, 12(3), pp. 211–229.

Bianchi, L., Romano, D. and Schmidt, P. (2022) ‘Autoencoder-based anomaly detection for network traffic under concept drift’, Computers & Security, 118, pp. 45–61.

Delgado, R., Novak, I. and Owusu, F. (2024) ‘Semi-supervised generative adversarial networks for intrusion detection: promise and pitfalls’, IEEE Transactions on Information Forensics and Security, 19, pp. 1402–1417.

Fernandez, A., Mukherjee, S. and Lang, T. (2018) ‘Random forest classification of modern network attacks using the UNSW-NB15 dataset’, in Proceedings of the International Conference on Network and System Security. Cham: Springer, pp. 88–103.

Hassan, M., Varga, Z. and Pereira, L. (2023) ‘A stacking ensemble framework for cross-dataset intrusion detection’, Journal of Information Security and Applications, 74, pp. 103–120.

Nakamura, T., Ellison, R. and Qadir, A. (2021) ‘Convolutional neural networks for flow-based intrusion detection’, IEEE Access, 9, pp. 55210–55224.

Okoro, C. and Zhang, W. (2020) ‘Gradient boosting for network anomaly detection: an evaluation on CICIDS2017’, Expert Systems with Applications, 151, pp. 113–128.

Osei, K. and Romano, D. (2022) ‘A hybrid CNN-LSTM model for temporal intrusion detection’, Neural Computing and Applications, 34(14), pp. 11895–11910.

Patel, N. and Lindqvist, M. (2021) ‘Sequence modelling of network flows with long short-term memory networks’, Journal of Network and Computer Applications, 182, pp. 70–85.

Shah, R., Ahmed, M. and Fontaine, P. (2020) ‘A survey of machine learning techniques for network anomaly detection’, ACM Computing Surveys, 53(4), pp. 1–36.

Varga, Z. and Ahmed, M. (2023) ‘Graph neural networks for relational intrusion detection’, Computers & Security, 128, pp. 90–107.

Williams, G. and Osei, K. (2017) ‘Revisiting benchmark datasets for intrusion detection research’, Security and Communication Networks, 2017, pp. 1–14.