Spoken Manipulation of MRI Brain Images

Medical Issues

For pre-operational planning and surgery simulators the reconstruction and segmentation of medical multidimensional images become more and more important. This development rises a need of techniques for the representation of anatomical knowledge. Displaying such 3-dimensional data on screen is a topic of virtual reality. Interacting with the 3-dimensional virtual systems with conventional input devices, like the mouse or the keyboard is often difficult, because desired operations have to be described on flat 2-dimensional screen. Spoken dialogue enables more comfortable interaction with such systems as it is closer to human's way to describe 3-dimensional operations.

Brain Images

The 3D surface of the brain is extracted directly from a MR volumic image, by a thresholding operation. It is made of squares named surfels, which are the facets of the voxels (volume elements) of the volumic image. Arbitrary points are chosen on the surface and associated to an arbitrary colour. The colours are then diffused on the surface, and their intensity is modulated to take account that the area (darker) is inside a sulci, or at the top of a girus of the brain (lighter).

Requirements for Resolving Spoken Commands in a Virtual Space:

• Even if brain images are visualised on a 2-dimensional screen, the commands to a interacting system are nearly almost related to the users idea of the 3-dimensional world behind the screen. Commands like 'I would like to see the other side of the brain.' or 'Turn it right.' are some examples of user commands which give a proof to this. To enable this kind of commands a representation of 3-space has to be implemented, containing the position and the orientation of the user.
• Because fast switching pictures make orientation extremely difficult, animation sequences should visualise a smooth transition between the previous and the desired position.
• Some of the commands contain references to objects which are placed in 3-space: 'I would like to see the cerebellum'. As the system has to resolve these references, a representation of objects in the world is required, even if there is no relation between these and the brain image. Examples for information of a object can be its viewpoint and its centre.
• Commands like 'Stop!' and 'Continue!' suggest an implementation of an animation planning which is separated from its execution. In addition animation sequences should be reversible as far as commands 'Go back!', 'Go back to the right hemisphere!' are possible.

System Architecture

The primary system inputs are spoken user commands like 'Go forward.', 'I would like to see the right hemisphere.' 'Turn the brain to the left.', 'Stop' 'Go back!' 'Where am I?' These sentences are recognised by a commercial device, split into their syntactical structures by a CHART parser, and interpreted in their semantic context. To make the interaction more natural, a dialogue automata reacts to the user's input by generating answers and requests for further information. If the user requests a manipulation of the brain image, it calls the corresponding functions of the motion planning. To enable smooth transitions between previous and desired position, the motion planning prepares animation sequences, generating a motion list which is displayed by a separated execution unit.

Conclusion & Perspectives

We have implemented a system which visualises medical brain images in a 3-dimensional way. We currently implement a dialogue system for spoken navigation in this virtual world. We think this type of system can be useful in application ranges such as telemedicine and medical education.