Project R-16411

AssembLLeM: LLM-based expert knowledge elicitation and semi-automatic generation of (conversational) DWIs. (Research)

In industry, and particularly in High-Mix Low-Volume (HMLV) manual assembly, the creation and use of Digital Work Instructions (DWIs) must balance cost, accuracy, and flexibility. Authoring DWIs is a high-effort process that often requires input from various stakeholders like product experts, process engineers, and multimedia specialists. At the same time, plenty of documentation and tacit knowledge is typically available that however needs to be manually converted into usable DWIs. This all results in significant labor costs, with ratios of one hour of DWI authoring time for three minutes of work support not being uncommon in industry. As such, companies are often compelled to draft DWIs only once instead of keeping them up-to-date (e.g., when product changes happen). For the same reason, at most a handful of specialized versions are commonly created per instruction set (e.g., distinguishing between novice versus expert operators). On the shopfloor, however, operators need instructions that are clear, accurate, and responsive to their preferences and immediate context—whether that means different competency levels or product variants. The need from industry is thus to have solutions that reduce DWI authoring overhead while enabling more flexible, context-aware guidance for operators. While our validation centers on assembly work instructions, reflecting the consortium's customer base, the methods and RRs are domain-agnostic and intended to generalize to other instruction-centric tasks (e.g., installation, maintenance, service). AssembLLeM acknowledges Large Language Models (LLMs) as a timely and highly suitable technology to tackle the above described DWI challenges. On the authoring level, assembLLeM will research how LLMs can consolidate knowledge bases by correlating relevant dispersed information sources, such as legacy documents (e.g., product manufacturing information), datasheets, and safety regulations. Special attention will hereby be given to tacit knowledge extraction from domain experts, particularly via automatic processing of video recordings of expert demonstrations. The consolidated knowledge bases will then be exploited to accelerate DWI authoring, where LLM technology is applied to quickly draft new DWIs and to adapt existing DWIs (e.g., customization towards heterogeneous shopfloor settings). In this workflow, company-specific requirements will automatically be accounted for (e.g., DWI styling guidelines, safety regulations, …). Through eXplainable AI (XAI) and human-in-the-loop (HITL) principles, the human engineer will at all times retain full control over assembLLeM's LLM-powered DWI authoring. On the consumption side, assembLLeM will leverage LLM technology to realize a novel conversational DWI approach that is both flexible and bi-directional. Through natural interaction paradigms, the operator will be able to engage in Q&A sessions while receiving adaptive guidance. This will in turn foster upskilling opportunities by letting operators acquire conceptual knowledge during task executions.

01 June 2026 - 31 May 2028