Title:
Using Fine-tuned and Context-Aware Large Language Models for Requirements Technical Debt Evaluation in Systems Engineering Work Products
Abstract:
Technical Debt (TD) has evolved from a software engineering metaphor into a broader systems engineering concern, where Requirements Technical Debt (RTD) manifests through ambiguous, incomplete, or inconsistent requirements that propagate costly design rework and integration challenges. Despite its critical impact on project quality and lifecycle costs, the detection and management of RTD within Systems Engineering Work Products—particularly Systems Engineering Management Plans (SEMPs)—remain largely manual, subjective, and inconsistent across practitioners.
This research introduces a fine-tuned, context-aware Large Language Model (LLM) framework for automating RTD detection, interpretation, and explanation in SEMPs. The study employs a two-phase mixed-methods design that integrates human expertise with computational reasoning. In Phase I, qualitative insights from fifteen Subject Matter Experts (SMEs) at Draper were elicited through structured interviews and surveys to capture their cognitive heuristics for identifying and mitigating RTD. These insights were encoded into structured JSON representations and synthesized into a machine-readable Requirements Debt Detection Guide (RQDG)—a standardized knowledge artifact encoding linguistic cues, reasoning patterns, and mitigation strategies.
In Phase II, this expert-informed guide was operationalized into an intelligent agent—the SEMP Requirements Debt Analyzer—implemented within a Retrieval-Augmented Generation (RAG) framework hosted on AWS Bedrock. The system integrates authoritative Systems Engineering standards (INCOSE SE Handbook, ISO/IEC/IEEE 15288, NASA SEMP guidelines) and employs Chain-of-Thought (CoT) and In-Context Learning (ICL) prompting strategies to balance interpretability and contextual efficiency. The analyzer produces structured, traceable outputs identifying instances of RTD, their severity, explanatory reasoning, and suggested corrective actions.
Evaluation through post-interview feedback surveys demonstrated strong SME alignment, with 64% of participants confirming that the AI-generated reasoning accurately reflected their analytical process, and 73% agreeing that the system introduced valuable new insights. These results validate that a hybrid CoT + ICL framework can approximate expert judgment in a transparent and auditable manner, supporting the long-term goal of enhancing peer review efficiency and standardization in systems engineering practice.
This work contributes a replicable methodology for digitizing expert reasoning, a functioning prototype for AI-assisted RTD detection, and early empirical evidence supporting the feasibility of fine-tuned, context-aware LLMs as cognitive partners in SE lifecycle management. Future research will extend testing within Draper’s Controlled Unclassified Information (CUI) environment to quantify time savings, evaluate model generalization across SE domains, and refine explainability modules for operational deployment in engineering workflows.
Research Advisor:
Prof. Shams Bhada
ECE Department, WPI
Research Committee:
Prof. Bashima Islam
ECE Department, WPI
Brian Sheehan
Draper Corporation