Introduction

Scientific Goals

Neurosurgical practices are progressively guided by tomographic scanning technologies with increasing anatomical details, such as magnetic resonance imaging (MRI). These tomographic image technologies enable presurgical analysis and planning, as well as surgical navigation. However, a growing number of parameters and data are available to the surgeon in order to plan his surgery, broadening the process and demanding complex perception. The surgeon’s perception relies largely on imaging processes, which need to be supported by developing new designed interfaces to aid navigating through data and consolidating numerous inputs, e.g. to a range from patients’ medical histories to structural brain architectures. To advance such digital representations, we develop functional models and prototypes that introduce new paradigms both in image viewing and analysis in an interdisciplinary setting, consisting of graphic design, software design, cognitive anthropology, software engineering and neurosurgery. These interdisciplinary explorations lead to new perspectives on the neurosurgical task, and to new types of knowledge that are transferred through those tools.

The context of this project, and especially the necessities and restrictions of neurosurgical clinical environments, have been analyzed in a preceding interdisciplinary Cluster of Excellence project at Humboldt University Berlin.

The research project Brain Roads concatenates interdisciplinary concepts and aims to develop new approaches on design, humanities, material science and medicine. As a research group, it is building on fruitful encounters between sciences, humanities, art and design around a new approach of matter and materials, objects and techniques, based on the notion of materiality and activity of the digital and the brain. To provide conceivable visualizations of anatomical features, thus helping users to better interpret relevant diagnostic details within a neuroanatomical context, novel approaches for reformatting the manifold surfaces of image data are to be explored.

To enable the analysis and implementation of large datasets in real time, it is necessary to develop innovative data filtering. The Data Visualization Program, developed by the department of Graphic and Digital Design at ESAD de Reims with computer engineers of JIN (video games and digital interactions) department at Télécom SudParis (TSP), addresses a series of graphic and digital design challenges in relation to contemporary digital transformation. Indeed, the underlying reality of digital technologies is decreasingly perceived and understood by its users, due to the increasing complexity of technology. This research program makes visible and sensitive BigData and background computational algorithms that process and act on our socio-technical environments. The joint team of designers and engineers proposes effective, open visual concepts and interface prototypes based on state-of-the -art interfaces, such as Virtual Reality, Augmented Reality, Mixed Reality or Multi-modal interfaces to allow new interactions to be experienced.

Together, the newly constituted international research team will develop and compare several alternative image analysis software and design new software interfaces for advanced neurosurgical image analysis and manipulation, in the form of experimental prototypes produced by French MA students in Design and Engineering. Furthermore, display and visualization technologies will be investigated with the aim of spatially merging virtual and physical forms of human action in neurosurgical planning. Thus, these models will incorporate advanced fiber tracking data (probabilistic delineation of the human white matter, based on diffusion MRI) as well as connectome data (structural-functional brain network). New modes of interaction, based on an interdisciplinary analysis of historical and contemporary concepts of imaging data visualizations and various technical gestures will lead to a paradigm shift in presurgical image analysis and therefore refine presurgical planning. Furthermore, the exploration of neurosurgeons’ individual mapping techniques of, their use of mental models and visual landmarks, will add to our understanding of image viewing and how complex knowledge is being transferred.

Project Aims

1.Paradigm shift in medical image viewing and analysis
Numerous forms, including models and prototypes,design artefacts and scenarios will be taken into account to develop new tools for medical image analysis. Different data inputs derived from various imaging modalities and tools will be combined. Designers, Software Engineers and researchers will develop their work around the notion of Gestaltung and the form-giving processes, in close collaboration with the work of scientists and researchers at HU, Charité, ESAD, TSP and ENSSIB. Functional models and prototypes that rethink both medical image viewing and analysis in an interdisciplinary setting, consisting of graphic design, software engineering and neurosurgery.

2.Multi-modal image combination
Currently there are many different programs for different applications, such as fiber tractography (derived from diffusion MRI data), connectome analysis (different MRI modalities), tissue segmentation etc. One of our main aims is to bring these multiple tools together and rethink as well as reconstruct medical image viewing in order to make preoperative planning more efficient. Based on past projects exploring image viewing and analysis, we will investigate display, interaction and visualization technologies with the aim to spatially merge virtual and physical forms of human action and neuroanatomy.

3. Customizing complex models to general usability
Also aiming at facilitating the understanding of a medical environment by a non specialist audience, the prototypes will result from a mix of user-centered methods, psychological and anthropological enquiries. The purpose here is to develop new digital models strengthening the visual understandability of brain wiring for non experts.

4.Open source tool
Developments and prototypes in software engineering will be published publicly and provided as open source.

Workshop in Berlin. 24-26.Oct.19

The 3-day workshop, which brings together teams of designers, engineers and researchers, has two main objectives:

• understand the formative process of current images of the brain model.
• define visual design issues and sketch them out by visual means.

workshop information and program

Summary of the first analysis & visual questions

The kick-off workshop brought together all project participants a week ago in Berlin. Together we analysed the process of formative current images which are intended to inform the neurosurgical team about the location of tumours in the vicinity of functional networks of critical importance to patients. Several visual questions emerged from this first phase:

• At present, the model seems very affirmative and precise whereas reality is much more complex to interpret. How to represent the degrees of uncertainty of the model? How can we legitimately increase doctors’ confidence in imaging?
• How to deliver all critical information during the most important 10 to 15 minutes of the surgical procedure?
• Currently, each scale has different conventions. How to build an intuitive, informative and coherent visual language for each scale of representation, from sub-millimetric voxel to macroscopic mapping?
• Colour is currently used to locate in space the directions of functional networks. The practicability and habitability of the system – and its current implementation – are in question. How can it be improved?
• Is it possible to propose radical changes in the aesthetics of these images, by summoning other cultural references, while respecting the conventions currently in use? What acceptability for new images?
• Is it possible to propose more abstract forms to represent the brain and its functions?

Graphic Proposals

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

The Question of Color

• 

• 

• 

• 

First prototype

Technical process to produce tractographic images

• 

• 

Access to the open source code

Our entire work was focused on improving the software library Mrtrix, and more precisely mrview, a tool used to visualize IRM data. As we did a fork of the original project on GitHub, it is possible to review, download, and use the code we wrote by previous link. This is in the continuity of Mrtrix and Mrview, both of which are open source projects.

Github branch links:

• brainroads-master
  Fiber bundles colors and positionning tool (merged)
  git-hub link

• brainroads-uncertainty
  Uncertainty zone (applied to each objects)
  git-hub link

Install notes:

Within the readme, you have to follow the “quick install” part. Then you only have to follow these instructions :

Before launching mrview
export MRTRIX_IMAGE=xxxxx/out2.bmp
This is the link to the image, it can be anywhere but after a clean clone, it should be in your mrtrix3 directory root.
And it could be added to .bashrc/.zshrc to keep the variable when restarting the terminal.

How to use the software
The position helper is always available, provided visual data is being displayed on screen.
The new color set is available when visualizing tractography data (Display panel in the upper right hand corner). This applies to all .tck files.

To access an available branch
Execute the command “git checkout NAME” in the repository, then the command “./build” (as the readme explains). You need git command to do this.