At IEEE SANER 2024 in Rovaniemi, Finland, we presented “On the Stability and Applicability of Deep Learning in Fault Localization” (Csuvik, Aszmann, Beszédes, Horváth and Gyimóthy).
The study trained deep learning fault localization models (MLP, CNN and RNN) on the Defects4J benchmark several times, with the same input but different random initialisations. The rankings varied widely between trainings: 90% of the produced ranks differed in subsequent runs. Meta-parameter optimisation, simpler models and resampling helped in part, but the models still were not stable enough to produce reliable results.
The finding that stays with us: a model that scores well once may not behave the same on the next run. Production AI needs repeatable results.
Read the paper: publisher page
