Applying Software Engineering Black-Box Methods for Testing Machine Learning Models

Author

Timothy Elvira, Embry-Riddle Aeronautical UniversityFollow

Date of Award

Spring 2025

Access Type

Dissertation - Open Access

Degree Name

Doctor of Philosophy in Electrical Engineering & Computer Science

Department

Electrical Engineering and Computer Science

Committee Chair

Omar Ochoa

First Committee Member

Massood Towhidnejad

Second Committee Member

Nicholas Del Rio

Third Committee Member

Alex Vargas

Fourth Committee Member

Kenji Yoshigoe

College Dean

James W. Gregory

Abstract

This dissertation proposes researching an approach to incorporate and align Software black-box testing methods into Machine Learning (ML) applications, specifically in the context of computer vision models. Typically, testing methods within Software Engineering (SE) encompass a range of test types that assess levels of a software system, such as Unit, Integration, Functional, and System testing [1]. The testing spectrum offers two perspectives on the system: black-box, where the system’s code is hidden, and white-box, where the system's code is exposed for testing. Software Quality pairs testing with requirements, in a many-to-one relationship, to ensure proper validation of the software system. Within ML, models are often tested in a black-box manner, with the focus on the input-output classification. However, the results are quantified by metrics such as, F-1 score, accuracy, precision, and recall, which do not capture the nuanced behavior of the ML model [2]. Interestingly, ML Non-Functional Requirements (NFR) are a growing research area with ML Functional Requirements (FR) remaining relatively unexplored [3]. This dissertation proposes using the growing ML FR domain to invigorate the testing of ML. The goal of this research is to leverage the expanding functional requirements research to introduce black-box testing methods of equivalence partition testing, decision table testing, and boundary value analysis, to uncover the behavioral qualities of an ML model. This dissertation accomplishes the exploration of ML testing through four case studies, which iteratively examine software engineering-based black-box testing methods and the adjustments required to accommodate the stochastic and non-deterministic nature of ML. The new SE-inspired ML testing strategies were evaluated on their abilities to fulfill requirements, which are a staple for traditional SE-based verification. The results showed that the strategies, by fulfilling requirements linked to specific behaviors, are effective in testing and identifying ML model behaviors in both industry-grade and academic object detection models.

Scholarly Commons Citation

Elvira, Timothy, "Applying Software Engineering Black-Box Methods for Testing Machine Learning Models" (2025). Doctoral Dissertations and Master's Theses. 886.
https://commons.erau.edu/edt/886