Understanding the Phish: Using Judgment Analysis to Evaluate the Human Judgment of Phishing Emails
MetadataShow full item record
Phishing emails, malicious messages designed to appear legitimate in an attempt to get individuals to conduct compromising actions, pose a continuously growing threat to cybersecurity. Phishing campaigns are responsible for around 90% of all identified data breaches and result in billions of dollars lost each year. Existing user training and automatic filtering techniques are not grounded in cognitive theory and thus have limited effectiveness. As such, there is a real need to understand how users synthesize information to identify phishing emails. The lens model, a judgment analysis (JA) technique, uses symmetric statistical models of the environment (also called the criterion) and the judgment values made by the human to evaluate human judgment performance. Because the lens model provides a means of analyzing both the environment and the human users, it was hypothesized that it would be a more effective way of understanding the phishing problem than conventional approaches. Further, recent literature suggests that cognitive automaticity plays a critical role in phishing victimization. The overlap between the lens model and the cognitive continuum theory (CCT; a human judgment theory that places cognitive modes along a continuum from intuitive to analytical cognition) enables the effect of automaticity (intuitive cognition in CCT terms) on phishing detection to be studied at a higher fidelity than was previously possible. This research focused on applying the lens model and the CCT to phishing. This aimed to satisfy three objectives: validate the lens model approach for the analysis of phishing email judgments, explore the differences in lens model approaches within this domain, and apply and extend the lens model's analysis capabilities with the CCT to better understand the phishing problem.Because the lens model had not been previously applied to phishing, it was necessary to assess the effectiveness of this analysis method in this domain. This included whether the statistical assumptions of a lens model analysis with multiple linear regression were upheld. We hypothesized that phishing cues are linearly combinable because each adds additional evidence that an email is phishing, meaning a lens model analysis should be appropriate for evaluating phishing judgments. To test this, ten participants, who judged whether or not emails were phishing, were analyzed using the double system lens model. Results showed that the lens model is an effective means of analyzing phishing judgments. This was indicated by a high environmental predictability value, which showed that the judgment environment was well represented by a linear model. Thus, the non-linearity of the environment was not a performance limiting factor for the judge. High cognitive control values indicated that humans do use linear judgment strategies, meaning a linear regression model adequately captured the human's judgment policy. Establishing the lens model as an effective analysis technique enables future work to capitalize on the lens model's afforded analysis capabilities. Although traditional lens model analyses use multiple linear regression, other regression types may be more appropriate for certain variable types. Both the criterion and judgment in the data used throughout this work were dichotomous, but there is little work focused on understanding how to use the four most appropriate lens model approaches for handling dichotomous variables (linear, logistic, confidence-adjusted, and hybrid). Thus, it was necessary to investigate the statistical and practical differences between these four lens model methods when applying JA to phishing. Partially modeled by previous literature, comparisons included lens model statistics, cue weight rankings, and prediction accuracy using cross-validation. A second, larger dataset with 74 judges was analyzed. Results indicated differences between the lens model statistics computed for the four methods based on the type of regression used to evaluate the environment. Specifically, the approaches that fit a logistic model to the criterion provided the best decomposition of the phishing judgment domain. Because there were no significant differences between statistics from the logistic method and the hybrid method, and the hybrid method had other practical disadvantages, it was concluded that the logistic method was the most appropriate for evaluating phishing judgments. Logistic regression also exhibited the most accurate predictions for both the criterion and for individual's judgments.The previous results of this work were used to apply analysis capabilities afforded by the lens model to gain insights into numerous aspects of the phishing problem, including cognition. While recent phishing literature identified cognitive automaticity as an important aspect of phishing victimization, more research was needed. The cognitive continuum theory was used with the lens model to not only investigate judge cognition, but also the cognitive implications of the email sorting task. This was done by calculating a cognitive continuum index score for each participant and a task continuum index score for the environment. The lens model results from the logistic method of the previous objective were used for this objective. The results indicated that the task was better suited for more analytical cognition. There also was a positive relationship between judge cognition and achievement, meaning participants who exhibited more analytical cognition performed better on the task. Cluster analysis results identified three general judgment policies across participants. The cluster with significantly lower achievement values also had significantly lower cognition scores, indicating more intuitive cognition. While there were three different judgment policies, two did not have any differences in achievement, highlighting the role of vicarious functioning. Although the amount of achievement explained by the modeled and unmodeled components differs between clusters, the lack of a significant difference in explained achievement (the proportion of achievement that the judgment and criterion models explain) means the models consistently explained achievement regardless of cluster. When comparing the specific odds ratios for each cue from the clusters, the main difference between the highest and lowest achieving clusters was in the utilization of the suspicious link cue. These results, combined with the significant differences in CCI score, seem to suggest that for that cue to be appropriately utilized, more analytical cognition was required from the judge.Overall, this research builds upon the lens model, CCT, and phishing literature by combining established and novel measures and analysis techniques to provide a more comprehensive understanding of the phishing domain. Results indicated that the lens model, especially the logistic method, is effective for evaluating phishing judgments. CCT analyses evaluated the cognitive implications of the task and judgments. Results supported the posited relationship between automaticity and victimization and highlighted differences between judgment policy groups. This gives analysts the ability to understand how to apply judgment analysis to the phishing domain, creating many new avenues of future research. This will be vital for providing a theoretically grounded basis for phishing mitigation and training approaches.