This article will be available soon. Please check back.


Following a series of high profile miscarriages of justice in the UK linked to questionable expert evidence, the post of the Forensic Science Regulator was created in 2008. The main objective of this role is to improve the standard of practitioner competences and forensic procedures. One of the key strategies deployed to achieve this is the push to incorporate a greater level of scientific conduct in the various fields of forensic practice. Currently there is no statutory requirement for practitioners to become accredited to continue working with the Criminal Justice System of England and Wales. However, the Forensic Science Regulator is lobbying the UK Government to make this mandatory. This paper focuses upon the challenge of incorporating a scientific methodology to digital forensic investigations where malicious software (‘malware’) has been identified. One aspect of such a methodology is the approach followed to both select and evaluate the tools used to perform dynamic malware analysis during an investigation. Based on the literature, legal, regulatory and practical needs we derive a set of requirements to address this challenge. We present a framework, called the ‘Malware Analysis Tool Evaluation Framework’ (MATEF), to address this lack of methodology to evaluate software tools used to perform dynamic malware analysis during investigations involving malware and discuss how it meets the derived requirements.


[1] Adam, C. (2016). Forensic Evidence in Court: Evaluation and Scientific Opinion. John Wiley & Sons.

[2] Akinrolabu, O., Agrafiotis, I., & Erola, A. (2018). The challenge of detecting sophisticated attacks: Insights from SOC Analysts. Proceedings of the 13th International Conference on Availability, Reliability and Security, 1–9. https://doi.org/10.1145/3230833.3233280

[3] Bahnsen, A. C., Torroledo, I., Camacho, L. D., & Villegas, S. (2018). DeepPhish: Simulating Malicious AI. 2018 APWG Symposium on Electronic Crime Research (ECrime), 1–8.

[4] Bayer, U., Habibi, I., Balzarotti, D., Kirda, E., & Kruegel, C. (2009). A view on current malware behaviors. Proceedings of the 2nd USENIX Conference on Large-Scale Exploits and Emergent Threats: Botnets, Spyware, Worms, and More, 8–8. Retrieved from http://portal.acm.org/citation.cfm?id=1855676.1855684

[5] Beckett, J. (2010). Forensic Computing: A Deterministic Model for Validation and Verification through an Ontological Examination of Forensic Functions and Processes (PhD, University of South Australia). Retrieved from Personal communication from author, September 2011

[6] Beckett, J., & Slay, J. (2007). Digital Forensics: Validation and Verification in a Dynamic Work Environment. System Sciences, 2007. HICSS 2007. 40th Annual Hawaii International Conference On, 266a–266a. https://doi.org/10.1109/HICSS.2007.175

[7] Bikeev, I., Kabanov, P., Begishev, I., & Khisamova, Z. (2019). Criminological risks and legal aspects of artificial intelligence implementation. Proceedings of the International Conference on Artificial Intelligence, Information Processing and Cloud Computing, 1–7. https://doi.org/10.1145/3371425.3371476

[8] Bowcott, O. (2018, January 15). London rape trial collapses after phone images undermine case. The Guardian. Retrieved from https://www.theguardian.com/law/2018/jan/15/london-rape-trial-collapses-after-phone-images-undermine-case

[9] Bowles, S., & Hernandez-Castro, J. (2015). The first 10 years of the Trojan Horse defence. Computer Fraud & Security, 2015(1), 5–13. https://doi.org/10.1016/S1361-3723(15)70005-9

[10] Brown, C. S. (2015). Investigating and Prosecuting Cyber Crime: Forensic Dependencies and Barriers to Justice. International Journal of Cyber Criminology, 9(1), 55.

[11] Burnay, C. (2016). Are Stakeholders the Only Source of Information for Requirements Engineers? Toward a Taxonomy of Elicitation Information Sources. ACM Transactions on Management Information Systems, 7(3), 8:1–8:29. https://doi.org/10.1145/2965085

[12] Carrier, B. (2010, August 11). Digital (Computer) Forensics Tool Testing Images. Retrieved 23 March 2016, from Digital Forensics Tool Testing Images website: http://dftt.sourceforge.net/

[13] Carvey, H. (2012). Windows Forensic Analysis Toolkit, Third Edition: Advanced Analysis Techniques for Windows 7 (3 edition). Waltham, MA: Syngress.

[14] Casey, E. (2019). The chequered past and risky future of digital forensics. Australian Journal of Forensic Sciences, 51(6), 649–664. https://doi.org/10.1080/00450618.2018.1554090

[15] Chen, P., Huygens, C., Desmet, L., & Joosen, W. (2016). Advanced or Not? A Comparative Study of the Use of Anti-debugging and Anti-VM Techniques in Generic and Targeted Malware. In J.-H. Hoepman & S. Katzenbeisser (Eds.), ICT Systems Security and Privacy Protection (pp. 323–336). https://doi.org/10.1007/978-3-319-33630-5_22

[16] Christensen, A. M., Crowder, C. M., Ousley, S. D., & Houck, M. M. (2014). Error and its Meaning in Forensic Science. Journal of Forensic Sciences, 59(1), 123–126. https://doi.org/10.1111/1556-4029.12275

[17] Clarke, S. (2009). Good Practice and Advice Guide for Managers of e-Crime Investigations. Association of Chief Police Officer of England, Wales and Northern Ireland.

[18] CPS. (2014, September 26). Evidence from Computer Records: Legal Guidance: The Crown Prosecution Service. Retrieved 23 May 2016, from The Crown Prosecution Service website: http://www.cps.gov.uk/legal/a_to_c/computer_records_evidence/

[19] CPS. (2019, October 9). Expert Evidence | The Crown Prosecution Service. Retrieved 27 February 2020, from https://www.cps.gov.uk/legal-guidance/expert-evidence

[20] Cuckoo Foundation. (2016). Automated Malware Analysis - Cuckoo Sandbox. Retrieved 14 March 2016, from https://cuckoosandbox.org/

[21] D’Elia, D. C., Coppa, E., Palmaro, F., & Cavallaro, L. (2020). On the Dissection of Evasive Malware. IEEE Transactions on Information Forensics and Security, 15, 2750–2765. https://doi.org/10.1109/TIFS.2020.2976559

[22] Deng, X., & Mirkovic, J. (2018). Malware Analysis Through High-level Behavior. Presented at the 11th {USENIX} Workshop on Cyber Security Experimentation and Test ({CSET} 18). Retrieved from https://www.usenix.org/conference/cset18/presentation/deng

[23] Digital Corpora. (2017, February 15). Digital Corpora. Retrieved 15 February 2017, from Digital Corpora website: http://digitalcorpora.org/

[24] Douglas, J. (2007, October 11). Trojan defence: the old chestnut... [Closed Law Enforcement forum]. Retrieved 8 April 2011, from Digital Detective website: http://www.digital-detective.co.uk/cgi-bin/digitalboard/YaBB.pl?num=1191330237/15

[25] Duranti, L., & Rogers, C. (2012). Trust in digital records: An increasingly cloudy legal area. Computer Law & Security Review, 28(5), 522–531. https://doi.org/10.1016/j.clsr.2012.07.009


[27] Elisan, C. C. (2015). Advanced Malware Analysis. McGraw-Hill Osborne.

[28] Fang, Y., Zhang, W., Li, B., Jing, F., & Zhang, L. (2020). Semi-Supervised Malware Clustering Based on the Weight of Bytecode and API. IEEE Access, 8, 2313–2326. https://doi.org/10.1109/ACCESS.2019.2962198

[29] Ferrie, P. (2007). Attacks on more virtual machine emulators. Symantec Technology Exchange, 55, 369.

[30] Forensic control. (2011). What is IT forensics? Retrieved 8 December 2011, from Forensic Control website: http://forensiccontrol.com/resources/beginners-guide-computer-forensics/

[31] Forensic Science Regulator. (2020a, February 25). Forensic Science Regulator highlights threats to criminal justice. Retrieved 27 February 2020, from GOV.UK website: https://www.gov.uk/government/news/forensic-science-regulator-highlights-threats-to-criminal-justice

[32] Forensic Science Regulator. (2020b, April 22). Forensic science providers: codes of practice and conduct, Issue 5. Retrieved from https://www.gov.uk/government/publications/forensic-science-providers-codes-of-practice-and-conduct-2020

[33] F-Secure. (2011, April). F-Secure Sample Analysis System. Retrieved 8 April 2011, from F-Secure Sample Analysis System website: https://analysis.f-secure.com/portal/login.html

[34] Garfinkel, S., Farrell, P., Roussev, V., & Dinolt, G. (2009). Bringing science to digital forensics with standardized forensic corpora. Digital Investigation, 6, S2–S11. https://doi.org/10.1016/j.diin.2009.06.016

[35] Guidance Software Inc. (2014, March 24). EnCase Legal Journal 5th Edition. Retrieved from https://www.guidancesoftware.com/docs/default-source/document-library/publication/encase-legal-journal---5th-edition.pdf?sfvrsn=aa3e8bad_16

[36] Horsman, G. (2019a). Formalising investigative decision making in digital forensics: Proposing the Digital Evidence Reporting and Decision Support (DERDS) framework. Digital Investigation, 28, 146–151. https://doi.org/10.1016/j.diin.2019.01.007

[37] Horsman, G. (2019b). Tool testing and reliability issues in the field of digital forensics. Digital Investigation, 28, 163–175. https://doi.org/10.1016/j.diin.2019.01.009

[38] Horsman, G. (2020). Part 1:- quality assurance mechanisms for digital forensic investigations: Introducing the Verification of Digital Evidence (VODE) framework. Forensic Science International: Reports, 2, 100038. https://doi.org/10.1016/j.fsir.2019.100038

[39] Hubbard, D. W. (2014). How to Measure Anything: Finding the Value of Intangibles in Business. John Wiley & Sons.

[40] Hughes, N., & Varol, C. (2020). The Critical Need for Tool Validation before Using Malware Scanners in Digital Forensics. ICCWS 2020 15th International Conference on Cyber Warfare and Security, 228. Academic Conferences and publishing limited.

[41] Hungenberg, T., & Eckert, M. (2016). INetSim: Internet Services Simulation Suite. Retrieved 10 April 2016, from INetSim: Internet Services Simulation Suite website: http://www.inetsim.org/

[42] Ianelli, N., Kinder, R., & Roylo, C. (2007). The Use of Malware Analysis in Support of Law Enforcement. Retrieved from CERT Coordination Center, Carnegie Mellon University website: http://www.securitynewsportal.com/securitynews/article.php?title=The_Use_of_Malware_Ana lysis_in_Support_of_Law_Enforcement

[43] ISO. (2005). ISO/IEC 17025:2005 - General requirements for the competence of testing and calibration laboratories. Retrieved 11 March 2011, from International Standards Organisation website: http://www.iso.org/iso/catalogue_detail.htm?csnumber=39883

[44] JCGM. (2008, September). JCGM - GUM. Retrieved 24 March 2011, from http://www.bipm.org/en/publications/guides/gum.html

[45] Joe Security. (2020). Joe Sandbox Cloud Basic. Retrieved 28 February 2020, from https://www.joesandbox.com/

[46] Kat, C.-J., & Els, P. S. (2012). Validation metric based on relative error. Mathematical and Computer Modelling of Dynamical Systems, 18(5), 487–520. https://doi.org/10.1080/13873954.2012.663392

[47] Kennedy, I. (2017). A Framework for the Systematic Evaluation of Malware Forensic Tools (PhD, The Open University). Retrieved from http://oro.open.ac.uk/50521/

[48] Kim, A. C., Kim, S., Park, W. H., & Lee, D. H. (2014). Fraud and financial crime detection model using malware forensics. Multimedia Tools and Applications, 68(2), 479–496. https://doi.org/10.1007/s11042-013-1410-3

[49] Kirillov, I., Beck, D., Chase, P., & Martin, R. (2010, February). Malware Attribute Enumeration and Characterization. Retrieved 8 January 2011, from http://maec.mitre.org/about/docs/Introduction_to_MAEC_white_paper.pdf

[50] Law Commission. (2011). Expert Evidence in Criminal Proceedings in England and Wales (No. 325).

[51] Lee, J. Y., Chang, J. Y., & Im, E. G. (2019). DGA-based malware detection using DNS traffic analysis. Proceedings of the Conference on Research in Adaptive and Convergent Systems, 283–288. https://doi.org/10.1145/3338840.3355672

[52] Levitin, D. (2016). A field guide to lies and statistics: A neuroscientist on how to make sense of a complex world. Penguin UK.

[53] Liu, J., Kammar, R., Sasaki, R., & Uehara, T. (2017). Malware Behavior Ontology for Digital Evidence. 2017 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C), 585–586. https://doi.org/10.1109/QRS-C.2017.105

[54] Lloyd, I. J. (2020). Information Technology Law (New Edition, Ninth Edition). Oxford, New York: Oxford University Press.

[55] Malin, C. H., Casey, E., & Aquilina, J. M. (2008). Malware forensics: investigating and analyzing malicious code. Syngress Publishing.

[56] Malin, Cameron H., Casey, E., & Aquilina, J. M. (2012). Malware Forensics Field Guide for Windows Systems: Digital Forensics Field Guides. Elsevier.

[57] Malin, Cameron H., Casey, E., & Aquilina, J. M. (2013). Malware Forensics Field Guide for Linux Systems: Digital Forensics Field Guides. Elsevier.

[58] McLinden, S. (2009, November 9). Child Porn Virus [Closed forum]. Retrieved 5 April 2011, from Guidance Software Inc. website: https://support.guidancesoftware.com/forum/showthread.php?t=36363&highlight=child+porn+virus

[59] Microsoft. (2020, September 17). Windows Sysinternals - Windows Sysinternals. Retrieved 18 September 2020, from https://docs.microsoft.com/en-us/sysinternals/

[60] Ministry of Justice. (2015, October 5). Criminal Procedure Rules 2015 [Procedure rules]. Retrieved 21 February 2017, from http://www.justice.gov.uk/guidance/courts-and-tribunals/courts/procedure-rules/criminal/rulesmenu.htm

[61] Murali, R., Ravi, A., & Agarwal, H. (2020). A Malware Variant Resistant To Traditional Analysis Techniques. 2020 International Conference on Emerging Trends in Information Technology and Engineering (Ic-ETITE), 1–7. https://doi.org/10.1109/ic-ETITE47903.2020.264

[62] Nataraj, L., Karthikeyan, S., Jacob, G., & Manjunath, B. S. (2011). Malware images: visualization and automatic classification. Proceedings of the 8th International Symposium on Visualization for Cyber Security, 1–7. https://doi.org/10.1145/2016904.2016908

[63] NIST. (2016, March 2). The CFReDS Project. Retrieved 23 March 2016, from The CFReDS Project website: http://www.cfreds.nist.gov/

[64] Palkmets, L., Ciobanu, C., Leguesse, Y., & Sidiropoulos, C. (2014, November). Building artifact handling and analysis environment toolset. Retrieved from https://www.enisa.europa.eu/topics/trainings-for-cybersecurity-specialists/online-training-material/documents/building-artifact-handling-and-analysis-environment-toolset/view

[65] Parkour, M. (2020, February 5). contagio. Retrieved 27 March 2020, from http://contagiodump.blogspot.com/

[66] Phu, T. N., Dang, K. H., Quoc, D. N., Dai, N. T., & Binh, N. N. (2019). A Novel Framework to Classify Malware in MIPS Architecture-Based IoT Devices [Research Article]. https://doi.org/10.1155/2019/4073940

[67] Provataki, A., & Katos, V. (2013). Differential malware forensics. Digital Investigation, 10(4), 311–322. https://doi.org/10.1016/j.diin.2013.08.006

[68] Royal Statistical Society. (2001, October 23). The Royal Statistical Society. Retrieved 13 November 2011, from http://www.rss.org.uk/site/cms/contentviewarticle.asp?article=527

[69] Shosha, A. F., James, J. I., Hannaway, A., Liu, C.-C., & Gladyshev, P. (2013). Towards Automated Malware Behavioral Analysis and Profiling for Digital Forensic Investigation Purposes. In M. Rogers & K. C. Seigfried-Spellar (Eds.), Digital Forensics and Cyber Crime (pp. 66–80). https://doi.org/10.1007/978-3-642-39891-9_5

[70] Sikorski, M., & Honig, A. (2012). Practical Malware Analysis: The Hands-On Guide to Dissecting Malicious Software (1 edition). San Francisco: No Starch Press.

[71] Singh, J., & Singh, J. (2018). Challenge of Malware Analysis: Malware obfuscation Techniques. International Journal of Information Security Science, 7(3), 100–110.

[72] Smith, M. (2012, June). Factors Influencing Power. Retrieved 23 August 2016, from Common Mistakes in using statistics website: https://www.ma.utexas.edu/users/mks/statmistakes/FactorsInfluencingPower.html

[73] State of Florida v. Casey Marie Anthony. , No. 48-2008-CF-015606-O (Ninth Judicial Circuit Court, Orlando, Florida, USA 5 July 2011).

[74] Szor, P. (2005). The Art of Computer Virus Research and Defense (01 edition). Upper Saddle River, NJ: Addison-Wesley Professional.

[75] Talib, M. A. (2018). Testing closed source software: computer forensic tool case study. Journal of Computer Virology and Hacking Techniques, 14(2), 167–179. https://doi.org/10.1007/s11416-017-0302-x

[76] Tank, D., Aggarwal, A., & Chaubey, N. (2019). Virtualization vulnerabilities, security issues, and solutions: a critical study and comparison. International Journal of Information Technology. https://doi.org/10.1007/s41870-019-00294-x

[77] Thanh, C. T., & Zelinka, I. (2019). A Survey on Artificial Intelligence in Malware as Next-Generation Threats. MENDEL, 25(2), 27–34. https://doi.org/10.13164/mendel.2019.2.027

[78] Truong, T. C., Diep, Q. B., & Zelinka, I. (2020). Artificial Intelligence in the Cyber Domain: Offense and Defense. Symmetry, 12(3), 410. https://doi.org/10.3390/sym12030410

[79] Tully, G., Cohen, N., Compton, D., Davies, G., Isbell, R., & Watson, T. (2020). Quality standards for digital forensics: Learning from experience in England & Wales. Forensic Science International: Digital Investigation, 200905. https://doi.org/10.1016/j.fsidi.2020.200905

[80] University of London. (2020). Digital forensics - IYM015. Retrieved 27 February 2020, from University of London website: https://london.ac.uk/courses/digital-forensics

[81] University of Portsmouth. (2019). Malware Forensics - U23524. Retrieved 27 February 2020, from MALWARE FORENSICS Academic Session:2020 - 2021 v.2 website: https://register.port.ac.uk/ords/f?p=111:19:::NO::P19_UNIT_ID:1861507637

[82] van Ruth, E. M., & Smithuis, M. M. A. (2019). On Forensic Science Expertise. In P. L. Patrick, M. S. Schmid, & K. Zwaan (Eds.), Language Analysis for the Determination of Origin: Current Perspectives and New Directions (pp. 243–252). https://doi.org/10.1007/978-3-319-79003-9_14

[83] VirusShare. (2020, March 27). VirusShare.com. Retrieved 27 March 2020, from https://virusshare.com/

[84] VirusTotal. (2010). VirusTotal - Free Online Virus and Malware Scan. Retrieved 14 November 2011, from VirusTotal - Free Online Virus, Malware and URL Scanner website: http://www.virustotal.com/

[85] Wagener, G., Dulaunoy, A., & Engel, T. (2008). An Instrumented Analysis of Unknown Software and Malware Driven by Free Libre Open Source Software. IEEE International Conference on Signal Image Technology and Internet Based Systems, 2008. SITIS ’08, 597–605. https://doi.org/10.1109/SITIS.2008.57

[86] Williams, J. (2012). ACPO Good Practice Guide for Digital Evidence v5.0. ACPO.

[87] Wueest, C. (2014). Threats to virtual environments. Symantec Research. Mountain View. Symantec, 1–17.

[88] YARA. (n.d.). Retrieved 26 September 2020, from YARA - The pattern matching swiss knife for malware researchers website: https://virustotal.github.io/yara/





To view the content in your browser, please download Adobe Reader or, alternately,
you may Download the file to your hard drive.

NOTE: The latest versions of Adobe Reader do not support viewing PDF files within Firefox on Mac OS and if you are using a modern (Intel) Mac, there is no official plugin for viewing PDF files within the browser window.