VISUALIZATION OF ULTRASOUND IMAGES WITH AUGMENTED REALITY

Abstract

Ultrasound imaging is a diagnostic method that uses high frequency sound waves that come out of a transducer and bounce back at the interfaces of the tissues to create a cross sectional image with different characteristics depending on the wave energy. These images are further visualized on a monitor on the ultrasound station. On the other hand, Augmented Reality (AR) is a technology that is able to superpose digital objects in the real world by the use of electronic devices, being the most widely used the Head Mounted Displays (HMDs) which can be transparent or opaque, using cameras to map and/or display the real world, respectively.

Integration of ultrasound images on AR involved portable ultrasound systems and the HoloLens 2 HMD. Communication between them was achieved using Google Remote Procedure Call (gRPC), which is based on client-server communication where the client was a portable ultrasound system and the server was HoloLens 2 HMD. The client involved the development of an Android application on an Android tablet that is able to record its screen using Android¿s official MediaProjection API (Application Programming Interface) and process it to send it to the server. The server was developed using the Unity video game engine, where a scene that contains a 3D plane was created. Finally, a C# script was developed for receiving the ultrasound captures, processing them and visualizing them on the 3D plane.

The integration of ultrasound imaging with AR was found very useful in the daily clinical practice by the medical personnel of Hospital Universitario La Paz. Some limitations could also be found, the most notable one being the end-to-end latency of acquiring the ultrasound images with the transducer and displaying them on AR. Nonetheless, this approach is still a great advancement for improving the precision and efficiency in US diagnosis compared to traditional methods.