What BPM Pros Really Think About AI and A/B Testing Process Change

Table of Links

Abstract and 1. Introduction

2. Background and Related Work
3. Research Method
4. Results
5. Discussion
6. Conclusion and References

Abstract. In order to better facilitate the need for continuous business process improvement, the application of DevOps principles has been proposed. In particular, the AB-BPM methodology applies AB testing and reinforcement learning to increase the speed and quality of improvement efforts. In this paper, we provide an industry perspective on this approach, assessing requirements, risks, opportunities, and more aspects of the AB-BPM methodology and supporting tools. Our qualitative analysis combines grounded theory with a Delphi study, including semi-structured interviews and multiple follow-up surveys with a panel of ten business process management experts. The main findings indicate a need for human control during reinforcement learning-driven experiments, the importance of aligning the methodology culturally and organizationally with the respective setting, and the necessity of an integrated process execution platform.

1 Introduction

Business processes are crucial for creating value and delivering products and services. Improving these processes is essential for gaining a competitive edge and enhancing value delivery, as well as increasing efficiency and customer satisfaction. This makes business process improvement (BPI) a key aspect of business process management (BPM), which is described as “the art and science of overseeing how work is performed in an organization to ensure consistent outcomes and to take advantage of improvement opportunities” [6].

DevOps, an integration of development and operations, is “a set of practices intended to reduce the time between committing a change to a system and the change being placed into normal production, while ensuring high quality” [2] and widely applied in the software industry. A new line of research in the field of BPM proposes using DevOps principles, like AB testing, to facilitate continuous BPI with a method called AB-BPM [24]. AB testing assesses different software feature versions in the production environment with real users, usually in end user-facing parts of the software. The current version is only retired if the test data supports the improvement hypothesis. Applying such rapid validation of improvements to processes is a departure from the traditional BPM lifecycle, where the possibility of not meeting improvement hypotheses in production is rarely considered, leading to expensive do-overs [24,7]. Going beyond traditional AB testing, AB-BPM proposes the application of reinforcement learning (RL) to utilize performance measurements already while the experiments are conducted by dynamically routing incoming process instantiation requests to the more suitable version.

The AB-BPM method has not yet been systematically analyzed regarding the needs of BPM practitioners. To facilitate applicability, additional research is needed to increase confidence in the proposed approach. Furthermore, BPM practitioners’ insights on the AB-BPM methodology can be of value to the wider BPM community, since they can uncover hurdles and possibilities on the path to more automation in the field of process (re-)design. Therefore, this paper presents a qualitative study, with the overarching research question being: What do BPM experts think about AB-BPM, and which implications does this have for the further development of the methodology and supporting tools? To this end, we collected data on experts’ views regarding the impact, advantages, and challenges of the AB-BPM method in an industry setting. In particular, we study the overall sentiment, perceived risks, potential use cases, technical feasibility, and software support requirements of the AB-BPM method. To obtain the results, a panel of BPM experts from a large enterprise software company was interviewed and participated in follow-up surveys. We applied a mixture of the grounded theory [4] and Delphi [5] research methodologies.