Automated Incident Runbook Generation via LLMs: An Open Framework and Benchmark

Abstract

This is due to the fact that today's software architectures rely heavily on highly-distributed cloud-native systems, micro services architectures and automated operational environments which are constantly creating complex incidents. Incident runbooks are a critical component of production systems that lend structure to procedures and are of great help to engineers on diagnosis, mitigation and recovery. Traditional runbook creation is however still largely manual, involves lots of knowledge and effort from experts and the need to keep track of happenings in the infrastructure as it is integrated, configured and developed. This means that operational documentation is often ill-structured, out of date and hard to maintain. This study aims to create and assess a generative framework of incident runbooks (IRs) based on Large Language Models (LLMs). To achieve this, the following specific aims are set for this study: (1) To strip down incident artifacts (mortem, resolution transcript, or log from operational communication) into an actionable, systemized runbook and to provide an open benchmark for a systematic evaluation.

Design/methodology/approach:

The proposed framework has three elements, all linked together. Structured incident feature extraction techniques extract important operational elements like symptoms, triggering events, root causes, mitigation activities, etc., and also the resolution timelines from incident artifacts. Second, extracted features are then fed into LLM-based procedural synthesis mechanism to generate runbook automatically. Third, a rubric-based framework for the validation of generated outputs using an LLM-as-judge evaluates the outputs based on the specified rubric attributes: completeness, coherence, correctness and operational usefulness. An open benchmark of over 200 publishable incident reports was developed to enable experimentation by drawing on reports from cloud service providers and software platforms. Several different LLM configurations were tried with synthesized gold-standard runbooks.

Findings:

The results indicated that using a structured extraction along with an LLM-based synthesis approach enhances the procedural uniformity and contextual appropriateness of the generated runbooks. Results from the experiments showed that there are measurable differences in performance between the models evaluated and that the rubric-based approach to assessment is effective as an effective tool for assessing operational documentation quality.

Research limitations/implications:

Relying on data from incidents one of the basic public data sets available is used, and this data is reported differently and varies in how complete it is. In addition, the outputs produced are still dependent on model reasoning features and variations of prompts.

Practical implications:

The proposed approach can significantly decrease documentation maintenance expenses, enhance incident readiness, speed up operational knowledge sharing, and enable scalable incident response processes within an enterprise.

Originality/value/Novelty:

This research circumvents the need for human experts to evaluate the quality of OpsDocs and presents a new validation method for OpsDocs: rubric validation using LLM-as-judge.