GRATA: How AI is transforming training in robot-assisted eye surgery

When a robot operates on the eye, it is ultimately always the human surgeon who determines success or failure. But how do surgeons train to work with robotics – precisely, safely and reproducibly, without putting patients at risk?

This is exactly where GRATA comes in: the “GraphRAG-based training and education system for robot-assisted medical procedures”. In this research project, adesso is working with leading clinical and research partners to develop a novel training system for eye surgery. This post marks the start of a blog series in which we will follow GRATA over several years – from the initial concepts through prototypes to clinical demonstrators.

Why training in the operating theatre is under pressure

Robot-assisted eye surgery is among the most precise and delicate procedures in medicine. Even the slightest deviations can have lasting consequences for patients’ vision.

At the same time, operating theatre slots, highly specialised robotic systems and experienced staff are in short supply – and so are the opportunities to train for such complex procedures under real-world conditions. Traditional training methods such as shadowing in the operating theatre, work shadowing or isolated simulation modules are reaching their limits.

The next generation needs:

• a realistic yet risk-free training environment,
• repeatable training scenarios rather than one-off opportunities, and
• objective, data-driven feedback rather than purely subjective assessments.

The key question is: How do we design training and further education so that humans and robotics can truly operate as a well-coordinated team in the operating theatre?

GRATA: Intelligent training instead of isolated simulation

GRATA is developing an adaptive training system for robot-assisted eye surgery – initially focusing on the treatment of age-related macular degeneration (AMD).

At the heart of the approach is the integration of modern technologies:

• A 3D simulation realistically depicts the eye, the robot and the operating theatre.
• Semantic knowledge models precisely describe procedures, roles and instruments.
• GraphRAG technology links this specialist knowledge with language models to enable context-specific instructions. The system not only retrieves information but also recognises relationships within the specialist knowledge.

To ensure that a robot-assisted procedure can be practised safely, several components must work closely together, as shown in Figure 1.

At its core is a semantic knowledge base in which surgical procedures, domain knowledge, environmental information and process-relevant relationships are formally modelled. This knowledge base combines:

• a knowledge graph for structured knowledge and
• RAG-based knowledge sources such as medical literature, glossaries and process descriptions for unstructured knowledge.

On this basis, the training system utilises various AI components – basic, multimodal and domain-specific models. These help to understand training contexts, enrich information, generate instructions and evaluate training progress. The architecture provides for components such as:

• instruction generation,
• context understanding and enrichment,
• training evaluation,
• trainee monitoring and
• multimodal information processing.

A key aspect is the real-time processing of diverse data sources: speech, text, video data and robot sensor data. Through its connection to simulation environments, the real operating theatre, the trainee and the surgical robot, the system realistically replicates training situations – and evaluates them continuously.

This is complemented by perception and simulation components: sensors for recognising gestures and activities, microscopic image processing, and 3D simulators for the eye and the robot. This creates a training environment that replicates the real operating theatre setting as accurately as possible.

The aim: a training environment in which humans and robots can systematically practise and analyse complex procedures in a patient-safe environment – from preparation to follow-up. The continuous evaluation of sensor, video and interaction data enables context-specific feedback, an assessment of training progress and – where pedagogically appropriate – an adaptive adjustment of the training steps. For example, trainees can practise the positioning of an instrument and receive immediate feedback on deviations, movement execution or approach behaviour.

How GRATA interacts in operation

The interface view of GRATA (Figure 2) shows how this architecture interacts in actual operation.

The operating theatre forms the starting point for multimodal processing and supplies video, audio and robotics-related data to the central system components:

• A World Model processes visual information to form a situational understanding of the procedure – for example: Which instruments are located where, how are they moving, what is happening in the surgical field?
• A communication model interprets verbal and auditory interactions – i.e. questions, commands or feedback from those involved.

Both models access a shared data repository in which states, contextual information and relevant domain knowledge are consolidated. This information converges in the integrator: it links perception, the knowledge base and training logic into a consistent overall picture and generates situation-specific support from it.

The result is an adaptive assistance system that not only observes the training process but actively supports it through feedback, guidance and adaptive responses.

GRATA: fundamental research with broad potential

GRATA is a fundamental research project running until Q3 2028. The aim is not a finished product, but a demonstrator that tests new technical and pedagogical approaches.

We are deliberately taking an exploratory approach:

• some approaches are discarded,
• others are unexpectedly expanded if they prove particularly promising.

This flexibility is crucial, as AI technologies are evolving rapidly. Our architecture therefore remains deliberately open to enable the integration of new model types, forms of perception and interaction in the future.

At the same time, GRATA is not intended to be limited to eye surgery. The architecture is aimed at high-stakes environments where errors have serious consequences – and where practical, data-driven training is crucial. In addition to other surgical disciplines such as neurosurgery or orthopaedics, we see potential in, amongst other things:

• industrial robotics,
• logistics automation and
• complex simulation environments.

GRATA thus creates a building block for standardised, scalable training systems in these areas.

The strong consortium behind GRATA

GRATA thrives on interdisciplinary collaboration. The project brings together clinical practice, research, simulation and software development in close cooperation:

• The Klinikum rechts der Isar at the Technical University of Munich (MRI) leads the project and contributes world-leading expertise in robot-assisted retinal surgery as well as the clinical perspective.
• Fortiss develops semantic knowledge models and no-code approaches.
• The Technical University of Chemnitz is working on perception systems for gesture and activity recognition.
• SynthesEyes provides realistic eye simulations and microscopic image processing.
• adesso Life Sciences takes on key tasks at the interface between software development and AI: we are building the training platform, integrating GraphRAG technology and validating the system together with medical staff.

You can find a concise overview of the project’s objectives, partners, duration and contact details on our project page at adesso Life Sciences.

What adesso is developing in the GRATA project

Our responsibility lies at the heart of the training system – where training content, AI and robotic control converge.

We are developing a demonstrator that:

• manages and visualises training content both textually and graphically,
• enables training scenarios to be configured via a no-code approach,
• provides AI-supported instructions and feedback, and
• analyses robotics and sensor data in real time.

The no-code approach is central to this: surgeons should be able to configure training scenarios themselves – without any programming knowledge. This allows new surgical techniques or training levels to be quickly incorporated into the system.

Another key focus is the use of GraphRAG in a medical context. Traditional language models tend to produce inaccuracies (‘hallucinations’) when dealing with medical expertise. Our approach combines semantic knowledge graphs with domain-specific language models. This ensures that instructions and explanations can be clearly traced back to validated standards.

The interface to the operating theatre is also particularly important. Interaction with the information and command system takes place via microphone and loudspeaker or headset:

• Very fast and robust speech-to-text processing transcribes requests from the operating theatre in a fraction of a second.
• The requests pass through the AI pipeline – including GraphRAG and the integration of video and gesture data.
• The system outputs the response as visual and spoken feedback.

Only if this end-to-end chain remains well below perceptible waiting times will the system be viable in everyday clinical practice.

In addition, we rely on AR-supported interaction when setting up the robotic system and on natural language dialogues during training. An intelligent monitoring system evaluates sensor data in real time and makes learning progress objectively measurable.

Outlook: What the GRATA series brings

By Q3 2028, a demonstrator combining simulation, knowledge graphs, AI and robotics integration will be developed in stages. In this blog series, we will follow the project from a practical perspective and highlight, among other things:

• in around six months: initial findings on ontologies and user interfaces,
• subsequently: insights into domain-specific language models and the use of GraphRAG in an operating theatre context,
• later: validations with medical staff and experiences from clinical training sessions.