Table of Contents
Type: Full Masters Dissertation | Subject: Computer Science | Level: Masters | Word Count: ~14,000 words
This model Full Masters Dissertation was produced by an Essays UK specialist as reference material for learning purposes only. For support in this field, see our Computer Science writers.
Overview: This model MSc Computer Science dissertation presents a complete, structurally faithful example of an applied machine learning project: the design, implementation and evaluation of a framework for detecting phishing URLs. It demonstrates how a full dissertation is assembled — abstract, introduction, literature review, requirements and design, implementation, evaluation, discussion and conclusion — with a worked feature-engineering pipeline, a comparison of several classifiers and illustrative experimental results. All datasets, metrics and figures are illustrative and generated for teaching purposes; the document is intended solely as a learning reference for how a Masters-level computing dissertation is organised, argued and evidenced.
Phishing attacks continue to impose substantial costs on individuals and organisations by exploiting human trust to harvest credentials and financial information. Because the uniform resource locator (URL) is the first artefact a victim encounters, and one that can be inspected before any page content is retrieved, URL-level detection offers an attractive early line of defence. This dissertation designs, implements and evaluates a machine learning framework that classifies URLs as phishing or legitimate using only lexical and lightweight host-based features, avoiding the latency and privacy costs of downloading page content.
A feature-engineering pipeline was developed to extract thirty interpretable attributes from each URL, spanning length statistics, character composition, token-level signals, and host characteristics such as domain age and the presence of a raw IP address. Four supervised models — logistic regression, random forest, gradient-boosted trees and a character-level one-dimensional convolutional neural network — were trained and compared on an illustrative balanced dataset of 50,000 URLs using stratified cross-validation. Models were assessed with accuracy, precision, recall, F1 score and the area under the receiver operating characteristic curve, with particular attention to the false-positive rate because wrongly blocking legitimate sites carries a high operational cost.
In the experiments reported here, the gradient-boosted tree model achieved the strongest overall performance, with an illustrative accuracy of 97.4% and an F1 score of 0.973, while retaining full interpretability through feature-importance analysis. The character-level network performed comparably without manual feature engineering but at greater computational cost and reduced transparency. Feature-importance analysis indicated that URL length, the use of a raw IP address, domain age and the count of suspicious tokens were the most informative signals. The framework was wrapped in a lightweight inference service capable of classifying a URL in a few milliseconds, demonstrating feasibility for near-real-time deployment. The dissertation concludes by discussing the trade-off between accuracy and interpretability, the threat of adversarial evasion through feature manipulation, concept drift as attacker behaviour evolves, and the ethical considerations of automated blocking. These findings are illustrative and intended to show how such a study is designed, executed and critically appraised.
Phishing is a form of social-engineering attack in which an adversary impersonates a trusted entity to persuade a victim to disclose sensitive information or perform an action against their own interests. Unlike attacks that exploit purely technical vulnerabilities, phishing targets the human user, which makes it persistent and difficult to eliminate through software patches alone. The classic vector is a deceptive message — most often an email, but increasingly a text message or social-media post — that directs the recipient to a fraudulent web page designed to resemble a legitimate service such as a bank, a government portal or a cloud productivity suite. When the victim enters their credentials, those details are captured by the attacker and used for account takeover, financial fraud or onward attacks.
Central to this chain of deception is the uniform resource locator. The URL is the address the victim is asked to visit, and it is the one component of the attack that can be examined before any potentially malicious content is loaded. This property makes the URL a valuable object of study: a defence that can judge a URL accurately, quickly and without rendering the destination page offers protection at the earliest possible moment, before credentials are ever requested. Browsers, email gateways and endpoint security products all attempt to make this judgement, traditionally by consulting blacklists of known malicious addresses.
The limitation of blacklists is structural rather than incidental. A blacklist can only contain URLs that have already been observed, reported and verified. Modern phishing campaigns are engineered precisely to defeat this reactive model: attackers register large numbers of short-lived domains, rotate addresses rapidly, and use automatically generated subdomains and path components so that each victim may encounter a distinct URL. By the time a phishing URL has been reported and added to a blacklist, the campaign may already have moved on. The operational lifetime of many phishing pages is short, which means that the window in which a blacklist is useful is correspondingly narrow. A defence that could generalise from the characteristics of known phishing URLs to recognise previously unseen ones would address this gap directly.
Machine learning offers exactly this capacity for generalisation. Rather than memorising specific addresses, a supervised classifier learns statistical regularities that distinguish phishing URLs from legitimate ones, and can then apply those regularities to URLs it has never seen. This is the premise on which a growing body of research and a number of commercial products are built. The appeal is considerable: a well-trained model can flag a novel phishing URL in milliseconds, using only the text of the address and a small number of cheaply obtained host attributes. The challenge is equally real. Attackers adapt, the statistical properties of legitimate and malicious URLs overlap, and the cost of wrongly blocking a legitimate site is high. Building a detector that is accurate, fast, interpretable and robust is therefore a non-trivial engineering and research problem, and it is the problem this dissertation addresses.
It is worth situating URL-level detection within the wider ecology of anti-phishing defences, because the choice to work at the level of the URL is a considered one rather than a default. Defences can be positioned at several points in the attack chain. At the delivery stage, email authentication protocols and spam filters attempt to stop the lure from reaching the victim at all. At the point of click, URL reputation services and browser safe-browsing features attempt to judge the destination. At the page stage, content analysis inspects the rendered page for login forms, brand logos and scripts characteristic of credential harvesting. And after the fact, takedown services attempt to have the malicious infrastructure removed. Each layer has a different cost, latency and failure mode, and a mature defence combines several. URL-level detection occupies a particularly useful position in this chain: it acts earlier than content analysis, is cheaper and more private because it needs no page fetch, and generalises better than reputation lookups because it reasons about the structure of the address rather than its identity. Positioning the contribution in this way clarifies that the detector developed here is intended to be one component of a layered defence rather than a complete solution, a framing that recurs in the ethical discussion of Chapter 6.
The economics of the problem also explain its persistence and shape the design priorities. Because a phishing campaign is profitable even at a very low response rate, and because the marginal cost of additional lures and domains is negligible, attackers can afford to operate at volume and to treat individual domains as disposable. This asymmetry means a defender cannot win by raising the attacker’s cost on any single URL; the defender must instead generalise across the population of phishing URLs faster than the attacker can vary them. That observation is the deepest justification for a learned, feature-based detector over any approach that reasons about specific addresses, and it motivates the attention paid later to whether the learned signal is durable or merely captures the signature of current campaigns.
The practical problem this project confronts is that reactive, list-based defences do not protect users against newly created phishing URLs during the critical early period of a campaign. There is a clear need for a predictive detector that can classify a URL as phishing or legitimate on the basis of its intrinsic characteristics, without depending on prior knowledge of that specific address. Such a detector must satisfy several competing requirements simultaneously. It must be accurate enough to be trusted, yet it must keep false positives extremely low, because blocking a legitimate site erodes user confidence and can disrupt business. It must be fast enough to operate inline, within the time budget of a page load or an email scan. It should ideally be interpretable, so that security analysts can understand and audit its decisions. And it should degrade gracefully as attackers change their behaviour over time.
From a research perspective, the problem is to understand which characteristics of a URL carry the most discriminative signal, how different classes of model trade accuracy against interpretability and cost, and how a detector trained on a snapshot of data can be expected to behave when confronted with evolving threats. This dissertation treats the problem as a supervised binary classification task and investigates it end to end, from feature design through to deployment feasibility, so that the engineering decisions and their consequences are made explicit.
The aim of this dissertation is to design, implement and critically evaluate a machine learning framework that detects phishing URLs using lexical and lightweight host-based features, and to establish which modelling approach offers the most favourable balance of accuracy, interpretability and operational cost. To achieve this aim, the following objectives were defined.
| Objective | Description |
|---|---|
| O1 | Review the literature on phishing detection, establishing the strengths and weaknesses of blacklist, heuristic and machine learning approaches, and identifying a defensible research gap. |
| O2 | Specify functional and non-functional requirements for a URL classifier suitable for near-real-time deployment. |
| O3 | Design a feature set and a modular system architecture covering data acquisition, feature extraction, model training and inference. |
| O4 | Implement a reproducible pipeline and train several candidate classifiers representing different points on the accuracy–interpretability spectrum. |
| O5 | Evaluate the classifiers on a held-out dataset using an appropriate suite of metrics, with explicit attention to the false-positive rate. |
| O6 | Critically discuss the results, including feature importance, error patterns, robustness to adversarial manipulation, concept drift and ethical considerations. |
The objectives are framed around three research questions that the evaluation and discussion chapters return to explicitly.
The scope of this project is deliberately constrained to make it tractable within the time and resources of a Masters dissertation. The detector operates on the URL and a small set of cheaply obtained host attributes only; it does not download, render or analyse the content of the destination page, nor does it inspect network traffic, TLS certificates in depth, or email headers. This restriction is a design choice as much as a limitation: content-based analysis is slower, raises privacy concerns because it requires fetching the page, and can itself be a vector for attack. By confining the detector to the URL, the project isolates a clearly defined and practically useful sub-problem.
Three limitations should be stated at the outset. First, the dataset used is illustrative and assembled for teaching purposes; although it is constructed to resemble the characteristics of real phishing and legitimate URLs, the absolute performance figures reported here should be read as indicative of method rather than as benchmark results. Second, the evaluation is offline and does not include a live deployment against active campaigns, so the effect of concept drift is discussed analytically rather than measured longitudinally. Third, the treatment of adversarial robustness is exploratory: the project reasons about plausible evasion strategies and conducts a small feature-perturbation experiment, but it does not develop a formal adversarial training regime. These limitations are revisited in the discussion and conclusion.
A word is warranted on what would constitute success for a project of this kind, because an applied dissertation can be misjudged if it is held to the wrong standard. The measure of success here is not a record accuracy figure, which, as the discussion will argue, is largely incomparable across studies and easily inflated. It is rather whether the project poses a well-defined question, designs an appropriate and reproducible method to answer it, executes that method correctly, reports the results with the metrics that matter, and appraises them honestly, including their limits. Framing success in these terms at the outset keeps the emphasis on method and reasoning, which is where the learning value of the document lies, and it guards against the temptation to present the work as more conclusive than the evidence allows.
The remainder of the dissertation is organised as follows. Chapter 2 reviews the literature on phishing and its detection, surveying list-based, heuristic and machine learning approaches, the feature-engineering tradition in URL classification, and the methodological challenges of datasets and benchmarking, before stating the research gap. Chapter 3 elicits requirements and presents the system design, including the feature set and the rationale for the models chosen. Chapter 4 describes the implementation of the data pipeline, feature extraction, model training and the inference service. Chapter 5 sets out the dataset, experimental protocol and evaluation metrics, and reports the comparative results, feature importance and error analysis. Chapter 6 interprets the findings, compares them with prior work, examines threats to validity, and considers ethical and deployment issues. Chapter 7 summarises the contributions, answers the research questions and proposes directions for future work.
The academic and industry literature consistently characterises phishing as a problem of scale, adaptation and economics. Phishing persists not because the underlying technical trick is sophisticated — frequently it is not — but because it is cheap to mount, easy to automate and difficult to defend against comprehensively. The attacker needs only a fraction of recipients to respond for a campaign to be profitable, and the marginal cost of sending additional lures is negligible. Monitoring organisations that track phishing activity report sustained growth in the number of distinct phishing sites observed over successive reporting periods, alongside a shift towards shorter-lived infrastructure (Anti-Phishing Working Group, 2023). The literature also notes a diversification of channels beyond email, including SMS-based “smishing” and voice-based “vishing”, and a professionalisation of the ecosystem through phishing-as-a-service offerings that lower the technical barrier to entry (Oest et al., 2020).
From a defensive standpoint, two features of this landscape are especially consequential. The first is the brevity of a phishing site’s useful life, which undermines any purely reactive defence, as discussed in the introduction. The second is the deliberate mimicry of legitimate services: attackers craft URLs and pages to resemble trusted brands, which means that superficial signals can mislead both users and naive detectors. The literature therefore emphasises detectors that can capture subtle structural regularities rather than surface resemblance (Sahingoz et al., 2019).
The literature also distinguishes several variants of phishing that, while they share the core deception, differ in targeting and infrastructure in ways relevant to detection. Mass or “bulk” phishing casts a wide net with generic lures and is the variant most amenable to statistical detection because its high volume produces recognisable regularities. Spear phishing and whaling target specific individuals or senior figures with tailored messages, trading volume for plausibility; these are harder to detect from the URL alone because the attacker invests in making the destination appear legitimate. Clone phishing reuses a genuine message, substituting a malicious link, and pharming manipulates name resolution so that even a correctly typed address leads to a fraudulent host. A URL-based detector is best suited to the high-volume variants and offers least against pharming, where the URL itself may be entirely legitimate; recognising this boundary is important for an honest account of what the detector in this project can and cannot do.
Before machine learning became dominant, phishing detection relied principally on two families of technique: blacklisting and rule-based heuristics. Blacklisting, as already discussed, consults a curated database of known malicious URLs. Its great strength is precision: because entries are verified, a blacklist produces very few false positives, and this reliability explains its continued use in browsers and gateways. Its weakness is coverage and timeliness. Studies that measured the latency between a phishing URL first appearing and its addition to major blacklists found that a substantial proportion of the harm occurs before listing, precisely because of the short operational lifetime of campaigns (Sheng et al., 2009, as summarised in subsequent reviews). Blacklists also generalise poorly: a trivial change to a URL produces an entry the list has never seen.
Rule-based heuristics attempt to address this by encoding expert knowledge about what phishing URLs tend to look like. Typical rules flag the presence of a raw IP address in place of a domain name, an unusually large number of subdomains, the use of the “@” symbol in the authority component, excessive URL length, or the appearance of a well-known brand name in a position where it does not belong. Heuristics generalise better than blacklists because they respond to properties rather than exact strings, and they are transparent and auditable. However, hand-crafted rules are brittle. Thresholds must be chosen manually, rules interact in ways that are hard to reason about, and attackers who understand the rules can craft URLs that avoid triggering them. The literature broadly concludes that heuristics are valuable as components of a layered defence but insufficient on their own (Mohammad, Thabtah and McCluskey, 2014).
A crucial insight drawn from this body of work is that the features used by heuristics — IP-address usage, subdomain counts, length, suspicious characters — are themselves good candidate inputs for a learned model. Machine learning can be seen as a principled way to combine and weight these signals, replacing manually chosen thresholds and brittle conjunctions with parameters fitted to data. This observation motivates much of the feature engineering described later in this dissertation.
The application of supervised machine learning to phishing detection is now well established in the literature. The standard formulation treats detection as binary classification: each URL is represented as a feature vector and labelled phishing or legitimate, and a classifier is trained to predict the label of unseen URLs. A wide range of algorithms has been applied, including logistic regression, support vector machines, decision trees, random forests, gradient-boosted ensembles and, more recently, deep neural networks (Sahingoz et al., 2019; Verma and Das, 2017).
A recurring finding across comparative studies is that ensemble tree methods — random forests and gradient boosting in particular — tend to perform strongly on tabular URL features, often outperforming linear models and sometimes rivalling more complex neural approaches, while remaining comparatively efficient to train and query (Subasi and Kremic, 2020). This is consistent with the broader machine learning literature, in which gradient-boosted trees are frequently competitive on structured, heterogeneous feature sets. Linear models such as logistic regression are reported to be fast, highly interpretable and a useful baseline, but to lose ground when the decision boundary between phishing and legitimate URLs is non-linear, as it generally is.
The literature also stresses the importance of evaluation methodology. Because phishing datasets are sometimes imbalanced, and because the cost of a false positive differs from that of a false negative, accuracy alone is widely regarded as an inadequate metric. Reviews consistently recommend reporting precision, recall, F1 score and the area under the ROC curve, and inspecting the confusion matrix, so that the behaviour of a detector on each class is visible (Das et al., 2020). This methodological guidance directly informs the evaluation design in Chapter 5.
A related theme concerns class imbalance and cost sensitivity. In deployment, legitimate traffic overwhelmingly outnumbers phishing, so a detector that appears excellent on a balanced research dataset can perform disappointingly in practice: at a fixed false-positive rate, the sheer volume of legitimate URLs means that even a small percentage of false positives produces many wrongly blocked sites in absolute terms. The literature responds with two complementary techniques. The first is cost-sensitive learning, in which the training objective penalises false positives and false negatives differently to reflect their real costs, steering the model towards the operating point the deployer actually wants. The second is resampling, through either over-sampling the minority class or under-sampling the majority, to counteract the imbalance during training. Reviews caution that these techniques must be applied inside the cross-validation loop to avoid optimistic bias, and that the evaluation must still be conducted at the realistic base rate if the reported precision is to be meaningful (Das et al., 2020). This project uses a balanced dataset for tractability and transparency but returns to the base-rate issue explicitly when discussing external validity in Chapter 6.
Feature engineering has historically been the centre of gravity in URL-based phishing detection, and the literature offers a rich catalogue of candidate features. These are conventionally grouped into categories. Lexical features are derived purely from the text of the URL and include its total length, the length of individual components (scheme, host, path, query), counts of specific characters such as dots, hyphens, digits, slashes and special symbols, the number of subdomains, the depth of the path, and the presence of tokens such as “login”, “secure”, “verify” or “account” that frequently appear in phishing URLs. Lexical features are attractive because they are instantaneous to compute and require no network access.
Host-based features describe the infrastructure behind the URL and include the age of the domain as recorded in WHOIS data, the time until registration expiry, the presence and configuration of DNS records, the autonomous system or hosting provider, and whether the host is a raw IP address rather than a registered domain. Host-based features add discriminative power — newly registered domains, for example, are disproportionately associated with phishing — but they require external lookups that introduce latency and potential points of failure. Content-based features, which require fetching and parsing the destination page, are reported to be powerful but are excluded from this project for the reasons given in the scope.
A methodological theme in the feature-engineering literature is the tension between informativeness and robustness. Some highly informative features are also easy for an attacker to manipulate. The presence of an HTTPS scheme, for instance, was historically treated as a signal of legitimacy, but the near-universal availability of free certificates has rendered it far less discriminative and in some analyses actively misleading. Similarly, URL length can be reduced by an attacker using a shortening service. The literature therefore cautions that feature selection should consider not only statistical importance on historical data but also the ease with which an adversary could alter a feature (Tian et al., 2018). This dissertation adopts that perspective, and the discussion chapter examines the manipulability of the most important features.
The question of how many and which features to use has itself been studied extensively. Feature-selection methods are conventionally grouped into filter methods, which rank features by a statistical association with the label independently of any model; wrapper methods, which search over feature subsets using a model’s performance as the objective; and embedded methods, in which selection is a by-product of training, as with the importance scores and regularisation of tree ensembles and linear models. The literature reports that a modest, well-chosen set of lexical and host features frequently captures most of the available signal, and that very large feature sets risk over-fitting and inflate computational cost without commensurate benefit. This finding supports the decision in the present project to work with a deliberately compact set of thirty interpretable features and to rely on the embedded importance measures of the tree models rather than an elaborate separate selection stage.
The temporal stability of features is a further, under-emphasised consideration. A feature that is highly discriminative in one period may lose or reverse its value in another as attacker tooling changes; the HTTPS example is the clearest case, but the risk applies more broadly. The literature’s recommendation of time-aware evaluation is partly a device for detecting precisely this instability, because a feature whose value depends on a passing campaign will contribute to a large gap between random-split and time-split performance. This dissertation treats feature stability as an explicit object of study rather than an afterthought, which is one of the ways it responds to the research gap identified at the end of the chapter.
A more recent strand of the literature applies deep learning to raw URL strings, aiming to reduce or eliminate manual feature engineering. The characteristic approach represents the URL as a sequence of characters, embeds each character into a dense vector, and processes the sequence with a convolutional or recurrent network that learns discriminative patterns directly from the data (Le et al., 2018; Saxe and Berlin, 2017, as discussed in later surveys). Character-level convolutional networks are reported to capture local patterns such as suspicious substrings and unusual character n-grams without those patterns being specified in advance, and some studies report accuracy competitive with, or slightly exceeding, carefully engineered feature-based models.
The advantages claimed for deep models are reduced reliance on human feature design and the ability to adapt to novel patterns. The costs reported are equally clear: deep models require more data and more computation to train, they are slower at inference unless optimised, and, most importantly for a security application, they are far less interpretable. A security analyst can read the decision path of a tree or the coefficients of a linear model, but cannot straightforwardly explain why a convolutional network flagged a particular URL. The literature increasingly frames this as a trade-off to be managed rather than a question with a single answer (Aleroud and Zhou, 2017). This dissertation engages with the trade-off directly by including a character-level convolutional network alongside the feature-based models and comparing them on accuracy, cost and interpretability.
Interpretability deserves emphasis because in a security context it is not a mere convenience but an operational and sometimes a regulatory requirement. When a detector flags a URL, an analyst must often decide whether to act, and that decision is far easier to make and to defend when the reasons for the flag are visible. Post-hoc explanation techniques such as feature-attribution methods have been proposed to open up otherwise opaque models, attributing a prediction to the contributions of individual features and thereby offering a degree of transparency even for complex classifiers. The literature notes, however, that such explanations are approximations and can themselves be unstable or misleading, and that an intrinsically interpretable model avoids this additional layer of uncertainty. There is also a governance dimension: where automated decisions affect access to services, accountability frameworks increasingly expect that a meaningful explanation can be provided. These considerations reinforce the project’s leaning towards models whose reasoning is directly inspectable, and they frame interpretability as a first-class design criterion rather than a desirable extra.
A persistent methodological problem identified throughout the literature is the difficulty of constructing and comparing on good datasets. Phishing URLs are commonly drawn from community feeds such as PhishTank and OpenPhish, while legitimate URLs are sampled from popularity rankings such as the Tranco list or from directories of established sites. Several difficulties arise. The first is label noise: community feeds can contain stale, misclassified or duplicate entries. The second is sampling bias: legitimate URLs drawn from popularity rankings tend to be short, clean, well-established domains, while phishing URLs are freshly harvested, so a classifier may inadvertently learn to distinguish “old and popular” from “new and obscure” rather than “legitimate” from “phishing”. The third is temporal leakage: if training and test sets are drawn from overlapping time periods, performance can be optimistically inflated because the model sees features of a campaign in training that recur in testing (Oest et al., 2020).
The literature recommends several mitigations: deduplication and careful cleaning, drawing legitimate and phishing URLs so that their superficial distributions overlap where possible, and, most importantly, time-based splitting in which the model is trained on earlier data and tested on later data to approximate real deployment. These recommendations shape the experimental protocol in Chapter 5, where a time-aware evaluation is discussed alongside the standard cross-validation, and the limitations of the illustrative dataset are stated plainly.
Synthesising the literature, three observations define the gap this dissertation addresses. First, while many studies report high accuracy, fewer foreground the false-positive rate and the operational cost of wrongly blocking legitimate sites, even though this is the dominant practical concern for a deployed detector. Second, the accuracy–interpretability trade-off between feature-based ensembles and character-level deep models is frequently asserted but less often demonstrated within a single, controlled comparison that also measures inference cost. Third, the robustness of the most important features to deliberate manipulation is often discussed in the abstract but seldom examined empirically within the same project that establishes those features’ importance.
This dissertation responds to that gap by conducting a controlled, end-to-end study that (a) treats the false-positive rate as a first-class metric, (b) compares feature-based and character-level models on accuracy, cost and interpretability within one pipeline, and (c) connects feature-importance analysis to an explicit consideration of adversarial manipulability. In doing so it does not claim to advance the state of the art in absolute accuracy; rather, it provides a transparent, reproducible worked example of how such a detector should be designed, measured and critically appraised — which is the appropriate contribution for a Masters-level engineering dissertation.
Because the project does not have an external client, requirements were elicited from three sources: the literature reviewed in Chapter 2, which establishes what a credible detector must measure and report; a consideration of realistic deployment contexts such as a browser extension or an email gateway, which constrains latency and resource use; and the research questions, which determine what the system must make observable for analysis. The requirements were then organised into functional requirements, describing what the system must do, and non-functional requirements, describing the qualities it must exhibit. Each requirement was given a priority using the MoSCoW scheme (must, should, could, won’t) so that trade-offs under time pressure could be made transparently.
| Ref | Requirement | Priority |
|---|---|---|
| FR1 | The system shall accept a URL as input and return a binary classification (phishing or legitimate) together with a calibrated probability score. | Must |
| FR2 | The system shall extract a defined set of lexical and host-based features from a URL deterministically. | Must |
| FR3 | The system shall train and persist multiple candidate models from a labelled dataset. | Must |
| FR4 | The system shall evaluate models and output a report of the agreed metrics, including a confusion matrix. | Must |
| FR5 | The system shall expose feature-importance information for the selected model. | Should |
| FR6 | The system shall provide an inference endpoint callable programmatically for near-real-time classification. | Should |
| FR7 | The system could allow the decision threshold to be adjusted to tune the balance between false positives and false negatives. | Could |
| Ref | Quality | Requirement |
|---|---|---|
| NFR1 | Performance | Inference on a single URL, excluding any external WHOIS lookup, should complete within approximately 10 milliseconds on commodity hardware. |
| NFR2 | Reliability | The false-positive rate on the held-out set should remain below 2% to be viable for inline use. |
| NFR3 | Interpretability | The primary deployed model should permit human inspection of the factors behind a decision. |
| NFR4 | Reproducibility | Random seeds, data splits and hyperparameters shall be fixed and recorded so that results can be regenerated. |
| NFR5 | Maintainability | The pipeline shall be modular so that features, models and datasets can be changed independently. |
| NFR6 | Privacy | The detector shall not fetch or store the content of destination pages, operating only on the URL and lightweight host metadata. |
Two requirements deserve emphasis because they shaped the design more than the others. NFR2, the ceiling on false positives, reflects the asymmetric cost structure of the problem and justifies treating the false-positive rate as a primary metric in evaluation. NFR3, interpretability, biases the choice of deployed model towards tree ensembles and linear models that expose their reasoning, even if a less transparent model were marginally more accurate.
The framework was designed as a modular pipeline of loosely coupled components, so that each stage could be developed, tested and replaced independently in keeping with NFR5. The architecture comprises five stages, arranged as a flow from raw data to a deployed decision.
A clean separation between the feature extractor and the models is the architectural keystone. Because the identical extractor object is serialised and reused at inference, the features computed when a model is queried are guaranteed to be consistent with those on which it was trained. The character-level neural network bypasses the tabular feature extractor and instead consumes a tokenised character sequence, but it is integrated behind the same inference interface so that models can be compared and swapped without changes to the calling code.
The data flow through the architecture can be described concretely. A raw URL string enters the system and is first normalised, then either routed to the tabular feature extractor, which emits a thirty-element numeric vector, or tokenised into a padded character sequence for the neural model. The vector or sequence is passed to the loaded model, which emits a probability; the probability is compared against the configured decision threshold to yield a label; and the label, probability and, optionally, the extracted feature values are returned to the caller. During training this same flow is applied in batch to the whole dataset, with the labels attached, so that the training and serving paths differ only in whether a label is present and whether the output is a fitted model or a prediction. Keeping the two paths structurally identical is a deliberate defence against training–serving skew, a failure mode in which subtle differences between how features are computed in training and in production silently degrade a deployed model.
Robustness of the pipeline itself, as distinct from robustness of the model, was treated as a design concern. Each stage was specified to fail safely: a URL that cannot be parsed is rejected with a clear error rather than being assigned arbitrary feature values; a WHOIS lookup that times out yields a sentinel rather than blocking; and the inference service validates its input before extraction. These choices reflect the principle that a security component must behave predictably on malformed or adversarial input, because such input is to be expected rather than treated as exceptional. The modular design also means that a fault in one stage — a change in an external WHOIS service, say — is contained and diagnosable rather than propagating opaquely through the system.
Thirty features were specified, drawn from the lexical and host-based categories identified in the literature and chosen to balance informativeness against cost and robustness. The table below lists a representative subset, grouped by category, with a brief rationale for each. The full set extends these with additional character counts and ratios.
| Feature | Category | Rationale |
|---|---|---|
| url_length | Lexical | Phishing URLs are frequently longer, padding paths and queries to obscure the destination. |
| host_length | Lexical | Long host strings can indicate brand-stuffing or generated subdomains. |
| num_dots | Lexical | Many dots imply many subdomains, a common obfuscation device. |
| num_hyphens | Lexical | Hyphens are used to construct look-alike domains such as secure-bank-login. |
| num_digits | Lexical | Digit-heavy hosts correlate with automatically generated phishing domains. |
| has_at_symbol | Lexical | An “@” in the authority causes browsers to ignore text before it, a classic deception. |
| num_subdomains | Lexical | Excess subdomains can place a trusted brand in a misleading position. |
| path_depth | Lexical | Deep paths can hide the true nature of the resource. |
| suspicious_token_count | Lexical | Tokens such as login, verify, secure and account recur in phishing URLs. |
| url_entropy | Lexical | High character entropy can indicate randomly generated strings. |
| has_ip_address | Lexical/Host | A raw IP in place of a domain is strongly associated with phishing. |
| domain_age_days | Host | Recently registered domains are disproportionately malicious. |
| tld_risk | Host | Certain top-level domains are over-represented in phishing feeds. |
| uses_https | Lexical | Retained but down-weighted, as HTTPS is now near-universal and weakly discriminative. |
Several design decisions in the feature set reflect the literature’s warnings about robustness. The uses_https feature was retained for completeness but anticipated to carry little weight, and this expectation is tested in the feature-importance analysis. The domain_age_days feature, while powerful, requires a WHOIS lookup and so is isolated so that the latency-critical lexical features can operate without it if a lookup is unavailable or too slow. All features were defined to be deterministic functions of the input so that extraction is reproducible, in accordance with NFR4.
Four models were selected to span the accuracy–interpretability spectrum identified in the literature, so that the comparison demanded by RQ2 could be conducted meaningfully. Logistic regression serves as a transparent linear baseline whose coefficients can be read directly; it establishes the performance achievable with the simplest credible model. Random forest introduces non-linearity and feature interactions while remaining interpretable through feature-importance measures and amenable to fast inference. Gradient-boosted trees were chosen because the literature repeatedly reports them as the strongest performers on tabular URL features, and because modern implementations are efficient and expose importance scores. Finally, a character-level one-dimensional convolutional neural network represents the deep-learning alternative that learns from the raw URL without manual features, allowing the project to test empirically whether the additional complexity and reduced interpretability are justified by a gain in accuracy. Together these four models make the central trade-off of the dissertation observable rather than assumed.
Two candidate approaches were deliberately excluded, and recording the reasoning is part of making the design defensible. Support vector machines were considered, as they appear frequently in the phishing literature, but were set aside because they offer little interpretability advantage over the tree ensembles while being harder to calibrate into reliable probabilities and slower to train at the chosen data scale. Large pre-trained language models applied to URLs were also considered and rejected for this project: although they can be powerful, their inference cost is incompatible with the latency target NFR1, their opacity conflicts with the interpretability requirement NFR3, and their use would shift the project’s centre of gravity away from the transparent, reproducible comparison that is its purpose. Documenting these exclusions shows that the four chosen models represent a reasoned sampling of the design space rather than an arbitrary selection.
The framework was implemented in Python, which was chosen for its mature machine learning ecosystem and its suitability for reproducible scientific work. Tabular models were built with a standard machine learning library providing logistic regression, random forests and a consistent cross-validation and metrics interface; gradient boosting used a dedicated high-performance boosting library; and the character-level network was implemented with a deep learning framework supporting GPU acceleration. Data manipulation used a dataframe library, and numerical work a standard array library. All dependencies and their versions were pinned in a requirements file, and a fixed random seed was set at every stochastic step — data shuffling, splitting, model initialisation and any sampling — so that the entire study could be regenerated deterministically, satisfying NFR4. The code was organised into modules mirroring the architecture of Section 3.4: a data module, a features module, a models module, an evaluation module and a service module, each independently testable.
Phishing URLs for the illustrative dataset were conceptually sourced from community feeds of reported phishing sites, and legitimate URLs from a popularity-ranked list of established domains, following the common practice described in Section 2.6. Because the dataset used here is illustrative and prepared for teaching, it was constructed to reproduce the structural properties of such sources rather than to reproduce any specific real feed. Pre-processing proceeded in four steps. First, URLs were normalised: the scheme was lower-cased, default ports were removed, and trailing fragments after a hash were stripped where they carried no routing meaning. Second, exact duplicates and near-duplicates differing only in volatile query parameters were removed to prevent the same campaign appearing in both training and test sets. Third, malformed entries that could not be parsed into a valid authority and path were discarded. Fourth, each URL was labelled, with phishing encoded as the positive class because the detection of phishing is the event of interest.
A deliberate effort was made to reduce the sampling bias discussed in Section 2.6. Rather than pairing very short, famous legitimate domains against long, obscure phishing URLs — which would let a classifier cheat by learning “famous versus obscure” — the legitimate sample was drawn to include a spread of lengths and structures, including legitimate URLs with long paths and query strings. This makes the classification problem harder and the reported results more honest, at the cost of some headline accuracy.
The feature extractor was implemented as a single class exposing a transform method that accepts a URL string and returns a fixed-length numeric vector. Internally it parses the URL into its components using a standard URL-parsing routine, then computes each feature as a pure function of those components. Counting features such as num_dots and num_hyphens are simple character tallies; num_subdomains is derived by splitting the host on dots and subtracting the registrable domain and public suffix; url_entropy is the Shannon entropy of the character distribution of the URL; and suspicious_token_count matches the host and path against a curated list of tokens frequently seen in phishing. The has_ip_address feature applies a regular expression for dotted-quad and bracketed IPv6 forms. The domain_age_days feature performs an optional WHOIS lookup, with a short timeout and a graceful fallback to a sentinel value if the lookup fails, so that the extractor never blocks indefinitely — an essential property for the latency target NFR1.
The extractor was unit-tested against hand-constructed URLs with known feature values, including awkward edge cases such as URLs with no path, internationalised domain names, embedded credentials, and the “@” deception. These tests guard against silent changes in feature semantics and were run as part of the pipeline so that any regression in extraction would be caught before training. Because the same extractor instance is serialised and loaded by the inference service, the features seen at prediction time are by construction identical to those seen during training, eliminating a common and insidious source of error.
A worked example makes the extraction concrete. Consider the illustrative phishing-style URL http://secure-login.paypa1-account.verify-session.com/webscr?cmd=update. The extractor would normalise the scheme to lower case, parse the authority as secure-login.paypa1-account.verify-session.com, and compute, among others: a total url_length of around seventy characters; a num_dots of three and a num_subdomains of two, reflecting the stacked subdomains that push the look-alike brand token into a misleading position; two num_hyphens; a has_at_symbol of zero; a has_ip_address of zero; a uses_https of zero; and a suspicious_token_count of three, matching secure, login and verify. The digit substitution in paypa1 contributes to num_digits and raises url_entropy slightly. By contrast, a legitimate URL such as https://www.gov.uk/log-in yields a short length, one subdomain, no suspicious tokens beyond a single login-related term, and an HTTPS scheme. Laying the two side by side shows how the feature vector encodes the structural differences the models exploit, and also why long, token-heavy but genuinely legitimate URLs are the natural source of false positives.
A design decision worth recording concerns the suspicious_token_count feature. The curated token list was kept deliberately short and generic, drawn from terms that recur across the phishing literature, rather than being tuned aggressively on the training data. Over-fitting this list to the particular dataset would inflate reported performance while producing a feature that generalises poorly and is trivial for an attacker to avoid simply by renaming tokens. Keeping the list small and principled trades a little headline accuracy for a feature that is more honest and more durable, a trade consistent with the project’s emphasis on robustness over leaderboard performance.
Training followed a uniform protocol so that the four models could be compared fairly. The labelled dataset was split into a development set and a held-out test set using a stratified split that preserved the class balance, with the test set reserved untouched until final evaluation. Within the development set, five-fold stratified cross-validation was used both to estimate generalisation during tuning and to produce robust validation metrics. Continuous features were standardised for the logistic regression and the neural network, for which feature scale matters, while the tree-based models were left unscaled because they are invariant to monotonic transformations of individual features.
Hyperparameters were tuned by grid search over small, sensible ranges, selecting the configuration that maximised the cross-validated F1 score, because F1 balances precision and recall and is appropriate given the asymmetric costs. The principal hyperparameters explored are summarised below. The character-level network’s architecture — an embedding layer, two convolutional blocks with max-pooling, a dense layer and a sigmoid output — was fixed after preliminary experiments, with training length controlled by early stopping on the validation loss to prevent over-fitting.
| Model | Key Hyperparameters Tuned | Selected Values (Illustrative) |
|---|---|---|
| Logistic Regression | Regularisation strength, penalty type | C = 1.0, L2 penalty |
| Random Forest | Number of trees, max depth, min samples per leaf | 400 trees, depth 20, leaf 2 |
| Gradient-Boosted Trees | Learning rate, number of estimators, max depth, subsample | lr 0.05, 600 estimators, depth 6, subsample 0.8 |
| Char-Level 1D-CNN | Embedding size, filters, kernel size, dropout | embed 32, 128 filters, kernel 5, dropout 0.3 |
To support FR1’s requirement for a calibrated probability rather than a bare label, the probabilities emitted by the selected tree model were checked for calibration, and the decision threshold was retained as a tunable parameter (FR7) so that an operator could move along the precision–recall curve according to their tolerance for false positives.
Although the dataset used in these experiments was balanced, the training code was written to accommodate class weighting so that the same pipeline could be applied to an imbalanced, deployment-realistic dataset without modification. Class weights allow the training objective to penalise errors on the rarer class more heavily, which is the mechanism by which a model trained on imbalanced data can still achieve useful recall on the minority class. Providing this facility in the implementation, even though it was not exercised in the balanced experiments, means the framework is ready for the cost-sensitive, imbalanced evaluation identified as future work, and it embodies the principle that the engineering should anticipate the realistic deployment setting rather than being tied to the convenient research setting. The cross-validation was implemented so that any resampling or weighting is applied only within each training fold and never leaks into the validation fold, which is the standard guard against the optimistic bias that improper resampling introduces.
To demonstrate deployment feasibility (FR6, NFR1) the selected model and its feature extractor were wrapped behind a minimal service exposing a single classification endpoint. On start-up the service loads the serialised extractor and model once into memory; on each request it extracts features from the supplied URL, passes them to the model, and returns the predicted label and probability as a structured response. The optional WHOIS lookup for domain age is guarded by a short timeout and can be disabled entirely for latency-sensitive callers, in which case the model falls back to the lexical features alone. The service was deliberately kept small and stateless so that it could be replicated behind a load balancer, and so that the measurement of per-request latency reported in the evaluation reflects the model and feature extraction rather than incidental framework overhead.
Because the credibility of the results depends on the correctness of the pipeline, testing was treated as integral rather than incidental. Three levels of testing were employed. Unit tests verified individual feature functions against hand-computed expected values, as described in Section 4.3, and covered the error-handling paths for malformed URLs and failed lookups. Integration tests exercised the full flow from a raw URL through extraction to a prediction, confirming that the serialised extractor and model loaded correctly and produced the same outputs after a save-and-reload cycle, which directly validates the defence against training–serving skew. A small set of regression tests pinned the metrics produced on a fixed seed and a fixed data sample, so that an unintended change anywhere in the pipeline would alter a known number and be noticed.
Reproducibility was validated by regenerating the headline results from a clean state using the recorded seeds and configuration and confirming that the figures matched, satisfying NFR4 in practice and not merely in intent. This discipline is particularly important in machine learning work, where stochastic training and data shuffling can otherwise make results subtly irreproducible and where that irreproducibility can mask genuine errors. Documenting and testing for reproducibility is therefore presented here as part of the methodological contribution of the dissertation rather than as mere housekeeping.
The illustrative dataset comprised 50,000 URLs, balanced equally between the phishing and legitimate classes to give the models sufficient positive examples and to make accuracy interpretable alongside the other metrics. The balanced design is a modelling convenience rather than a claim about real-world prevalence, where legitimate traffic vastly outnumbers phishing; the implications of this choice for deployment are examined in the discussion. The data were split into a development set of 40,000 URLs and a held-out test set of 10,000, with the class balance preserved in both. The composition is summarised below.
| Partition | Phishing | Legitimate | Total |
|---|---|---|---|
| Development (train + cross-validation) | 20,000 | 20,000 | 40,000 |
| Held-out test | 5,000 | 5,000 | 10,000 |
| All | 25,000 | 25,000 | 50,000 |
Two complementary evaluation regimes were used. The primary regime was five-fold stratified cross-validation on the development set, which produces a mean and a variance for each metric and so indicates both the level and the stability of performance. The held-out test set, untouched during all tuning, was then used for a single final evaluation of each model, providing an unbiased estimate of generalisation. In addition, and in direct response to the temporal-leakage concern raised in Section 2.6, a time-aware variant was conducted in which the data were ordered by their notional collection date and the model was trained on the earlier portion and tested on the later portion. This time-aware split is the more realistic proxy for deployment, and the gap between its results and those of the random split is itself an informative measure of how much the detector relies on transient campaign-specific regularities.
All models were evaluated through the identical pipeline and metrics code, so that differences in reported performance are attributable to the models rather than to measurement. Inference latency was measured as the median per-URL time over the test set, excluding the optional WHOIS lookup, on a single commodity CPU, to assess NFR1.
To avoid over-interpreting small differences between models, the comparison was supported by a measure of uncertainty. The cross-validation produced a standard deviation for each metric across the five folds, and these were used to judge whether the gap between two models was comfortably larger than fold-to-fold variation or within the noise. In addition, the predictions of the two leading models — the gradient-boosted trees and the character-level network — were compared on the held-out set using McNemar’s test, which is appropriate for comparing two classifiers on the same test instances because it examines the cases where the models disagree. Reporting this kind of paired comparison, rather than simply declaring the model with the higher accuracy the winner, is part of a defensible evaluation and guards against drawing strong conclusions from differences that could easily arise by chance.
Following the methodological guidance in the literature, a suite of metrics was reported rather than a single figure. Accuracy is the proportion of correct predictions; it is meaningful here because the dataset is balanced, but it is insufficient alone. Precision is the proportion of URLs flagged as phishing that truly are phishing, and it bears directly on the false-positive problem. Recall (sensitivity) is the proportion of true phishing URLs that are caught. F1 score is the harmonic mean of precision and recall, summarising the balance between them. The area under the ROC curve (AUC) measures the model’s ability to rank phishing above legitimate across all thresholds and is insensitive to the particular threshold chosen. Finally, the false-positive rate — the proportion of legitimate URLs wrongly flagged — was reported explicitly and treated as a primary acceptance criterion in line with NFR2, because it captures the dominant operational cost.
The table below reports the illustrative performance of the four models on the held-out test set. The figures are generated for teaching purposes and should be read as indicative of the relationships between models rather than as absolute benchmarks.
| Model | Accuracy | Precision | Recall | F1 | AUC | FPR |
|---|---|---|---|---|---|---|
| Logistic Regression | 93.1% | 0.930 | 0.928 | 0.929 | 0.975 | 7.0% |
| Random Forest | 96.8% | 0.971 | 0.965 | 0.968 | 0.994 | 2.9% |
| Gradient-Boosted Trees | 97.4% | 0.978 | 0.969 | 0.973 | 0.996 | 2.2% |
| Char-Level 1D-CNN | 97.1% | 0.972 | 0.969 | 0.970 | 0.995 | 2.8% |
Three patterns stand out. First, every model comfortably beats a naive baseline, confirming that the engineered features carry real signal. Second, the three non-linear models substantially outperform the linear baseline, which is consistent with the literature’s view that the phishing–legitimate boundary is non-linear; the logistic regression’s false-positive rate of 7.0% alone would disqualify it under NFR2. Third, and most importantly for RQ2, the gradient-boosted trees and the character-level network are close in accuracy, but the boosted trees achieve a marginally lower false-positive rate while remaining fully interpretable and cheaper to query. On the time-aware split, all models lost several points of accuracy and gained false positives — the boosted model fell to an illustrative 94.6% accuracy with a 4.1% false-positive rate — which quantifies the reliance on transient regularities and underscores why a time-aware evaluation is the more honest one.
The confusion matrix for the selected gradient-boosted model on the held-out test set is shown below, making the error distribution concrete.
| Predicted Phishing | Predicted Legitimate | |
|---|---|---|
| Actual Phishing | 4,845 (true positive) | 155 (false negative) |
| Actual Legitimate | 110 (false positive) | 4,890 (true negative) |
Measured latency was well within the target: the tree models classified a URL in a median of roughly two milliseconds excluding any WHOIS lookup, comfortably satisfying NFR1, while the character-level network was several times slower per query on CPU, a cost that would matter at scale.
The paired comparison of the two leading models found that their disagreements on the held-out set were roughly balanced — each model was right where the other was wrong on a similar number of URLs — so that McNemar’s test did not indicate a statistically convincing superiority of one over the other in raw accuracy. This is an important nuance: it means the preference for the gradient-boosted model rests not on a decisive accuracy advantage but on its lower false-positive rate, its interpretability and its far lower inference cost, which are exactly the criteria the requirements elevated. Reporting the absence of a significant accuracy gap, rather than quietly presenting the boosted model as simply better, is a more honest account of the evidence.
Beyond the single operating point reported in the table, the models were examined across the full range of decision thresholds. The receiver operating characteristic curves of the three non-linear models were close to the top-left corner and to one another, consistent with their similar AUC values, whereas the logistic regression’s curve was visibly inferior. More informative for this problem is the precision–recall curve, because it focuses on the positive (phishing) class and is more sensitive at the high-precision end that matters for deployment; there the gradient-boosted model retained the highest precision as recall increased, which is the behaviour a deployer seeking a low false-positive operating point would want. The probabilities emitted by the boosted model were also checked for calibration by comparing predicted probabilities against observed frequencies; they were reasonably well calibrated, meaning a predicted probability of, say, 0.9 corresponded to roughly a nine-in-ten chance of the URL being phishing, which makes the score usable as a confidence signal rather than merely an ordering. Where calibration is imperfect, the literature recommends a post-hoc calibration step, and this is noted as a straightforward refinement.
Feature importance was examined for the selected gradient-boosted model to answer RQ1, using the model’s built-in gain-based importance scores and cross-checked against a permutation-importance analysis on the test set, which is less prone to inflating the importance of high-cardinality features. The two methods broadly agreed on the ranking of the top features, which are reported below with their illustrative relative importance.
| Rank | Feature | Relative Importance |
|---|---|---|
| 1 | url_length | 0.19 |
| 2 | has_ip_address | 0.15 |
| 3 | domain_age_days | 0.13 |
| 4 | suspicious_token_count | 0.11 |
| 5 | num_subdomains | 0.09 |
| 6 | num_hyphens | 0.07 |
| 7 | url_entropy | 0.06 |
| … | uses_https (near-zero) | < 0.01 |
The analysis supports several interpretations. The dominance of url_length, has_ip_address and domain_age_days is consistent with both the heuristic tradition and the machine learning literature, and it confirms that a small number of features carry much of the signal. The negligible importance of uses_https empirically vindicates the design decision to down-weight it and corroborates the literature’s observation that HTTPS is no longer a useful indicator of legitimacy. The prominence of domain_age_days is a double-edged result: it is highly informative but requires an external lookup and is therefore both a latency liability and, as the discussion notes, a feature an attacker could partly neutralise by ageing domains before use.
A manual inspection of the misclassified URLs was conducted to understand where and why the selected model failed, because aggregate metrics conceal the character of errors. The false negatives — phishing URLs the model judged legitimate — were dominated by two patterns. The first was phishing hosted on compromised but otherwise reputable domains, where the registrable domain is old and clean and the malicious content sits on a deep path; here the host-based features actively mislead the model, and the project’s exclusion of content-based features leaves little signal to recover. The second was short, clean phishing URLs that deliberately avoid the structural tells the model relies on, demonstrating the evasion strategy discussed in the next chapter.
The false positives — legitimate URLs wrongly flagged — clustered around long, parameter-heavy legitimate URLs, such as tracking links and search-result pages, whose length and token composition superficially resemble phishing. This pattern explains why the deliberate inclusion of long legitimate URLs in the dataset (Section 4.2) depressed headline accuracy: it exposed a genuine weakness that a biased dataset would have hidden. The error analysis therefore not only characterises the model’s limitations but also validates the dataset-construction choices, and it motivates the future-work proposal to add a small number of carefully chosen content-light features to disambiguate these hard cases.
A closer reading of the errors suggests that the two error classes have different remedies, which is useful for prioritising future effort. The false negatives on compromised reputable domains are fundamentally a limitation of the URL-only design: no amount of additional lexical cleverness can recover a signal that is not present in the address, and only a cautious widening of the feature set towards cheap host or certificate signals is likely to help. The false positives on long legitimate URLs, by contrast, look more tractable, because the offending URLs share recognisable structure — recognisable query-parameter patterns, well-known tracking domains, and reserved paths — that a modest allow-list or an additional feature capturing the reputation of the registrable domain could address without compromising the detection of genuine phishing. Distinguishing errors that reflect a design boundary from errors that reflect a fixable gap is itself a contribution of the error analysis, because it prevents effort being spent on the former in the mistaken belief that it is the latter.
It is also notable that the model’s confidence on its errors was informative. Many of the false negatives were assigned probabilities close to the decision threshold rather than confidently legitimate, which means that a deployment operating at a more cautious threshold, or escalating borderline cases to a secondary check, would catch a meaningful share of them at the cost of a modest rise in false positives. This observation connects the error analysis back to the threshold-tuning requirement (FR7) and shows that the raw confusion matrix understates the room a deployer has to trade the two error types against each other.
Because the feature-importance analysis identified which signals the model relies on, a small adversarial experiment was conducted to test how easily those signals could be subverted, giving an empirical grounding to the robustness discussion. A sample of correctly detected phishing URLs was taken, and plausible, low-cost manipulations of the kind an attacker could realistically perform were applied: shortening the URL to reduce url_length and the token counts, removing hyphens and extra subdomains, and simulating the use of an aged rather than freshly registered domain by setting domain_age_days to a larger value. The proportion of these manipulated URLs that the model then misclassified as legitimate was recorded. The illustrative outcome is summarised below.
| Manipulation (Illustrative) | Phishing URLs Flipped to “Legitimate” |
|---|---|
| Shorten URL and remove suspicious tokens | ~22% |
| Remove hyphens and extra subdomains | ~9% |
| Simulate an aged domain (raise domain_age_days) | ~17% |
| All manipulations combined | ~41% |
The result confirms the central robustness concern: the very features that make the model accurate are, to varying degrees, within an attacker’s power to alter, and a determined adversary combining several cheap manipulations could evade the detector on a substantial fraction of cases. The domain-age manipulation is instructive because it has a real cost — the attacker must register and hold domains in advance — which illustrates that not all manipulations are equally cheap, and that a detector can usefully lean on features that are expensive to fake. This experiment is exploratory and its figures are illustrative, but it demonstrates how a robustness claim can be grounded in evidence rather than asserted, and it motivates the adversarial-training direction proposed in the future work.
The results support a coherent narrative about the problem. A detector built on cheap, interpretable URL features can reach a high level of discrimination, but only if it uses a model capable of representing non-linear interactions among those features; the linear baseline’s inadequate false-positive rate makes this concrete. Within the non-linear models, the gain from moving to a character-level deep network was negligible in accuracy and negative once interpretability and inference cost were weighed, which answers RQ2 in favour of gradient-boosted trees for this setting. This is not a claim that deep models are inferior in general — with far larger datasets, raw URL strings, and an operational need to avoid feature engineering, the balance could shift — but within the constraints of this project the engineered ensemble is the better engineering choice.
The feature-importance findings answer RQ1 directly: URL length, raw IP usage, domain age and suspicious-token count are the most informative signals, while HTTPS usage is effectively worthless. The gap between the random-split and time-aware results is perhaps the most instructive finding of all, because it quantifies something the single-number results hide: a meaningful fraction of the detector’s apparent skill comes from recognising the signatures of specific, contemporaneous campaigns rather than durable properties of phishing. This directly informs RQ3 and the treatment of concept drift below.
Taken together, the findings caution against reading any single accuracy figure as the measure of a detector’s worth. A model can post an impressive number on a random split and a balanced dataset while being fragile to drift, sensitive to the realistic base rate, and evadable through cheap feature manipulation — all of which this study made visible precisely because it looked beyond the headline metric. The methodological lesson, which is arguably the most transferable outcome of the project, is that the evaluation of a security classifier should be multidimensional: accuracy under realistic splitting, cost-weighted error rates, behaviour across thresholds, stability over time, and resistance to manipulation are all part of the picture, and a study that reports only the first of these risks presenting a flattering but misleading result.
The relative ordering of the models observed here is consistent with the broad consensus in the literature that ensemble tree methods are strong performers on tabular URL features and that linear models trail them (Subasi and Kremic, 2020; Sahingoz et al., 2019). The finding that a character-level network matches but does not clearly exceed a well-tuned boosted ensemble echoes the more measured recent studies that frame deep learning as a trade-off rather than an unambiguous advance (Aleroud and Zhou, 2017). The empirical demotion of the HTTPS feature aligns with observations in more recent work that the signal value of transport encryption has collapsed as free certificates became ubiquitous. Where this project departs from much of the literature is in its explicit prioritisation of the false-positive rate and its direct connection of feature importance to adversarial manipulability; these are not novel ideas, but foregrounding them within a single controlled study is the project’s modest contribution and is appropriate to its level.
It is important to be candid about how the absolute figures reported here relate to published results. Many studies report accuracies above 97% or even 99%, and the illustrative numbers in this dissertation sit comfortably within that band on the random split. However, direct numerical comparison across studies is largely meaningless, because results depend heavily on the particular datasets, the degree of cleaning, whether the split is random or time-aware, and the base rate assumed. A headline accuracy quoted without these details conveys little, and part of the argument of this dissertation is that the field would benefit from reporting practices that make such comparison meaningful — shared time-aware splits, realistic base rates, and the false-positive rate alongside accuracy. Rather than claiming to beat prior work, the project positions itself as an illustration of those better practices, which is the more useful stance for a reference artefact and avoids the common pitfall of over-claiming on the strength of a single, favourably measured number.
Several threats to validity must be acknowledged, structured as internal, external and construct validity. Internal validity concerns whether the measured differences between models are real; this was protected by using an identical pipeline and metrics for all models, fixing seeds, holding out a test set untouched during tuning, and reporting cross-validated variances. The principal residual internal threat is the illustrative nature of the dataset, which means the absolute figures cannot be taken as benchmark results. External validity concerns whether the findings generalise. The balanced dataset does not reflect the real class imbalance, so precision in deployment, where legitimate URLs dominate, would be worse than reported even at the same false-positive rate; this is a well-known consequence of base rates and is the strongest reason to treat the headline numbers cautiously. The time-aware evaluation partially addresses external validity but cannot substitute for a live trial. Construct validity concerns whether the metrics measure what matters; reporting a full suite including the false-positive rate, and conducting an error analysis, were intended to protect it.
Although the project used only publicly available, non-personal URL data and therefore raised limited data-protection concerns, deploying an automated blocking detector carries ethical weight that the dissertation treats seriously. The central issue is the consequence of a false positive: wrongly blocking a legitimate site can deny people access to services, damage the reputation of the blocked site, and erode trust in the security system itself. This asymmetry is precisely why the false-positive rate was elevated to a primary criterion. A responsible deployment would therefore favour a high-precision operating point, provide a clear and fast appeals or override mechanism, and treat the model’s output as advisory to a layered defence rather than as an unaccountable automatic block. Transparency matters too: the interpretability of the chosen model means that a flagged decision can be explained to an analyst and, where appropriate, to an affected site operator, which is far harder with an opaque model. Finally, there is a dual-use dimension — publishing the features that most betray phishing URLs also informs attackers — which is mitigated here by the fact that these signals are already well known in the literature and by framing the work as a defensive reference.
A second ethical theme concerns fairness and unintended harm. A URL-based detector could systematically disadvantage certain legitimate actors whose addresses happen to resemble the phishing distribution — small organisations using long shared-hosting URLs, services on less common top-level domains, or newly launched legitimate sites whose domains are genuinely young. Because the detector leans on features such as domain age and top-level domain risk, there is a real possibility of a disparate impact on exactly these groups, who are also often the least able to appeal or to have a block reversed quickly. A responsible deployment would monitor for such patterns, avoid treating any single high-risk feature as decisive, and ensure that the override mechanism is genuinely accessible rather than nominal. Framing this as a fairness question, and not merely an accuracy question, reflects the fact that the costs of errors fall unevenly across the population of legitimate sites.
A third consideration is data governance. Although the project used only URLs and public host metadata, a deployed system that logs the URLs users are about to visit would be processing information that can be highly revealing of individuals’ interests, health, finances and associations. Any real deployment would therefore need a clear lawful basis, strict minimisation and retention limits, and careful access controls over the logs, consistent with data-protection obligations. Noting this explicitly, even though the research itself did not handle personal data, is part of taking the deployment context seriously rather than treating the detector as a purely technical artefact divorced from its consequences.
The gap between the random-split and time-aware results identified in Chapter 5 is the clearest evidence in this project of concept drift: the tendency for the statistical relationship between features and labels to change over time, so that a model trained on past data becomes progressively less accurate on future data. In phishing detection, drift arises from two directions. Attacker behaviour evolves as tooling, hosting choices and evasion strategies change, which alters the distribution of phishing URLs; and the legitimate web changes too, as new services, URL conventions and marketing practices shift the distribution of benign URLs. A detector that is not maintained will therefore decay, and the rate of decay is itself uncertain and campaign-dependent.
Several strategies for managing drift are available, each with trade-offs. Periodic retraining on fresh data is the simplest and is often sufficient, but it requires a continuing supply of reliably labelled recent URLs, which is precisely the resource that is scarce and expensive. Online or incremental learning, in which the model updates continuously as new labelled examples arrive, can track drift more closely but risks instability and makes the model harder to audit because it is always changing. Drift detection, in which statistical monitors watch for changes in the input distribution or in the model’s error rate and trigger retraining only when needed, offers a middle path that conserves labelling effort while still responding to genuine change. A robust deployment would most plausibly combine drift monitoring with scheduled retraining and a held-back recent test set used to verify that each new model is genuinely better before it is promoted.
A subtler point is that not all of the detector’s skill is equally perishable. Features tied to durable structural properties of phishing — the use of a raw IP address, or the economic reality that much phishing infrastructure is freshly registered — are likely to remain informative longer than features tied to the surface vocabulary of current campaigns, such as particular suspicious tokens. This suggests that a detector deliberately weighted towards the more durable features might decay more slowly, trading a little peak accuracy for greater longevity. The project could not test this longitudinally, but the analysis indicates it as a principled design lever and a worthwhile direction for future evaluation, and it reframes concept drift from a nuisance to be patched into a property to be designed for.
Beyond the threats to validity, the project has concrete limitations that bound its claims. The exclusion of content-based features, while justified on latency and privacy grounds, caps the achievable accuracy and is directly responsible for the hardest error category, phishing on compromised reputable domains. The adversarial analysis is exploratory: a small feature-perturbation experiment showed that lengthening a legitimate-looking phishing URL or ageing its domain could flip some predictions, confirming the manipulability of the top features, but the project did not implement adversarial training or a formal robustness evaluation. The offline evaluation cannot capture concept drift as it unfolds, so the durability of the detector is argued rather than demonstrated. And the balanced, illustrative dataset limits the external validity of the absolute figures, as already discussed. These limitations are the natural boundary of a single Masters project and define its future-work agenda rather than undermining its conclusions about method.
This dissertation designed, implemented and evaluated a modular machine learning framework for detecting phishing URLs from lexical and lightweight host-based features alone. It contributed a transparent, reproducible worked example of the full engineering lifecycle: a thirty-feature extraction pipeline grounded in the literature and the heuristic tradition; a fair, controlled comparison of four models spanning the accuracy–interpretability spectrum; an evaluation that treats the false-positive rate as a first-class metric and includes a time-aware split; a feature-importance analysis connected explicitly to adversarial manipulability; and a lightweight inference service demonstrating that classification is feasible within a few milliseconds. The principal empirical result, on the illustrative data used here, is that a well-tuned gradient-boosted tree model offers the most favourable overall balance, matching a character-level neural network on accuracy while remaining interpretable and cheaper to query.
Beyond the specific result, the dissertation makes a methodological contribution that is arguably more durable than any accuracy figure. It demonstrates, within a single coherent study, an evaluation practice appropriate to security machine learning: treating the false-positive rate as a primary criterion, complementing random splitting with a time-aware split that exposes concept drift, supporting model comparison with a paired statistical test rather than a bare ranking, examining behaviour across decision thresholds and calibration rather than at one operating point, and grounding a robustness claim in an explicit perturbation experiment. Each of these is individually standard, but assembling them into one transparent, reproducible worked example is precisely what makes the document useful as a reference, and it embodies the argument running through the whole dissertation that a security classifier must be judged on several dimensions at once.
RQ1 is answered by the feature-importance analysis: URL length, the presence of a raw IP address, domain age and the count of suspicious tokens are the most informative features, whereas HTTPS usage is essentially uninformative. RQ2 is answered by the model comparison: non-linear models decisively outperform the linear baseline, and among them gradient-boosted trees are preferable to a character-level network once the near-equal accuracy is weighed against the trees’ superior interpretability and lower inference cost. RQ3 is answered by the combination of the time-aware evaluation, the error analysis and the discussion: the principal threats are base-rate-driven precision loss in deployment, concept drift as campaigns evolve, and adversarial manipulation of the most important features, each of which admits partial mitigation but none of which is fully solved by a URL-only detector.
Several avenues follow directly from the findings and limitations.
These directions are not independent wish-list items but follow logically from the specific weaknesses the evaluation exposed: the hardest errors, the base-rate problem, the measured evasion, the inferred drift, and the unverified value of explanations. Presenting future work in this way — as the direct continuation of what the evidence revealed — rather than as a generic list, is itself part of the discipline the dissertation aims to demonstrate.
In closing, this model dissertation has shown how an applied machine learning project in computer security is specified, built, measured and critically appraised. Its value as a reference lies less in the particular accuracy figures — which are illustrative — than in the explicit reasoning that connects requirements to design, design to implementation, and results to a sober account of what has and has not been established.
Aleroud, A. and Zhou, L. (2017) ‘Phishing environments, techniques, and countermeasures: A survey’, Computers & Security, 68, pp. 160–196.
Anti-Phishing Working Group (2023) Phishing Activity Trends Report. Available at: https://apwg.org/trendsreports/ (Accessed: 18 October 2026).
Das, A., Baki, S., El Aassal, A., Verma, R. and Dunbar, A. (2020) ‘SoK: A comprehensive reexamination of phishing research from the security perspective’, IEEE Communications Surveys & Tutorials, 22(1), pp. 671–708.
Le, H., Pham, Q., Sahoo, D. and Hoi, S.C.H. (2018) ‘URLNet: Learning a URL representation with deep learning for malicious URL detection’, arXiv preprint arXiv:1802.03162.
Mohammad, R.M., Thabtah, F. and McCluskey, L. (2014) ‘Predicting phishing websites based on self-structuring neural network’, Neural Computing and Applications, 25(2), pp. 443–458.
Oest, A., Safaei, Y., Zhang, P., Wardman, B., Tyers, K., Shoshitaishvili, Y. and Doupé, A. (2020) ‘PhishTime: Continuous longitudinal measurement of the effectiveness of anti-phishing blacklists’, in Proceedings of the 29th USENIX Security Symposium. Berkeley: USENIX Association, pp. 379–396.
Sahingoz, O.K., Buber, E., Demir, O. and Diri, B. (2019) ‘Machine learning based phishing detection from URLs’, Expert Systems with Applications, 117, pp. 345–357.
Saxe, J. and Berlin, K. (2017) ‘eXpose: A character-level convolutional neural network with embeddings for detecting malicious URLs, file paths and registry keys’, arXiv preprint arXiv:1702.08568.
Sheng, S., Wardman, B., Warner, G., Cranor, L.F., Hong, J. and Zhang, C. (2009) ‘An empirical analysis of phishing blacklists’, in Proceedings of the Sixth Conference on Email and Anti-Spam (CEAS). Mountain View: CEAS.
Subasi, A. and Kremic, E. (2020) ‘Comparison of adaboost with multiboosting for phishing website detection’, Procedia Computer Science, 168, pp. 272–278.
Tian, K., Jan, S.T.K., Hu, H., Yao, D. and Wang, G. (2018) ‘Needle in a haystack: Tracking down elite phishing domains in the wild’, in Proceedings of the Internet Measurement Conference. New York: ACM, pp. 429–442.
Verma, R. and Das, A. (2017) ‘What’s in a URL: Fast feature extraction and malicious URL detection’, in Proceedings of the 3rd ACM International Workshop on Security and Privacy Analytics. New York: ACM, pp. 55–63.
You May Also Like
All work is written by human writers. 100% AI free, guaranteed.
100% money back guarantee if you find plagiarism in our work.
COMPANY DETAILS
