Verification of Machine Learning Algorithms

Although Machine Learning (and Deep Learning) show great successes, their verification is often limited and rudimentary.
Master

Although Machine Learning (and Deep Learning) show great successes, their verification is often limited and rudimentary. By applying known software testing techniques, such as metamorphic testing, towards ML algorithms, the student will explore ways to verify the training and the generalization capabilities of supervised and unsupervised ML methods.

Usually, supervised ML is trained on a set of labeled examples and then verified on an unknown (but limited) set of similar examples, chosen randomly. Little is known about the relation of this test data to the real world or the potential input space.
In unsupervised ML the situation is even worse, verification with a test set is often not possible.

Metamorphic testing can help to overcome to oracle problem (i.e. there is no known correct answer to an input) in cases where it is known how the output changes if the input is changed in a certain way. This relation is called a metamorphic relation, and once it is known, it allows to automatically generate many additional test cases to verify a system.

In this thesis, the student should explore metamorphic relations for supervised and unsupervised machine learning techniques. This exploration includes both the identification of relations and their use in an experimental evaluation of different ML methods.

Potential ML methods include (deep) neural networks for classification, Reinforcement Learning or unsupervised clustering.

Goal

  • Identify metamorphic relations for supervised and unsupervised machine learning
  • Perform an experimental evaluation of representative algorithms based on the identified metamorphic relations

Qualifications

  • Programming Skills (preferably Python or Java), First experiences with Machine Learning

Supervisors

  • Arnaud Gotlieb
  • Helge Spieker

References

[1] S. Segura, G. Fraser, A. B. Sanchez, and A. Ruiz-Cortes, “A Survey on Metamorphic Testing,” IEEE Trans. Softw. Eng., vol. 42, no. 9, pp. 805–824, 2016.
[2] Ding, J., Kang, X., & Hu, X. H. (2017). Validating a Deep Learning Framework by Metamorphic Testing. Proceedings - 2017 IEEE/ACM 2nd International Workshop on Metamorphic Testing, MET 2017, 28–34. doi.org/10.1109/MET.2017.2