Augmented Reality assists surgeons in the operating theater

Artificial intelligence is taking on more and more tasks in our modern world. For example, we use it every day when we use online search engines. Translation programs are unimaginable without AI, as are speech recognition, face recognition, computer games and, in the future, autonomous driving. In medicine, AI is also becoming more widespread and has already found its way into the operating theater. Just a few days ago, Innovation Origins wrote about operating with live 3D image navigation inside the body.

The Karlsruhe Institute of Technology (KIT) has now gone one step further and has even been awarded the NEO 2019 Innovation Prize (worth €20,000) by the Karlsruhe TechnologyRegion for their ‘HoloMed’ system. The new system assists surgeons in the operating room via Artificial Intelligence (AI) and Augmented Reality (AR). It does this by creating a model from computer tomographic images of the patient. These reveal the hidden structures deep inside the body.

GPS for the brain

HoloMed’s main focus is on cranial punctures. This is a procedure whereby accumulated fluid is removed from the brain in order to reduce pressure. Frequently used for e.g. brain hemorrhages, craniocerebral trauma and strokes. In order to determine the optimal point of insertion and alignment for the puncture, the surgeon must measure and glean data from “various anatomical landmarks” from computer tomography (CT) and/or magnetic resonance imaging (MRI) scans.

“The difficulty lies in the fact that determining the angle of insertion only allows for a very small margin of error and the doctor isn’t able to see the target straightaway,” notes Professor Björn Hein. He oversees the project together with Professor Franziska Mathis-Ullrich at KIT. Determining this exact point is complicated as these images are only two-dimensional and the human head is three-dimensional. That’s why only about 60 percent of all free-hand incisions are able to pinpoint the best position.

Surgeons use HoloMed augmented reality glasses to assist them in determining this optimal insertion point and angle for the puncture needle. An AI developed at the AI by science staff member Christian Kunz uses the data from the patient’s digital file and their latest CT and/or MRI scans for creating a model that accurately depicts the structures deep inside the body that cannot be seen externally. This information is superimposed onto the surgeon’s AR glasses and shows the surgeon precisely where and how to guide the needle, much like a navigation system.

Easy to use and cost-efficient

Professor Hein states that machine learning methods are used in the automated generation of this information. “First of all, a segmented 3D model of the head is generated, which is used to determine the target position. However, the doctor is always able to make their own adjustments if appropriate,” Hein adds. The aim of the system is to provide an “innovative, novel and cost-effective solution that has a direct influence on the quality of these procedures”.

After its puncture method is successfully rolled out, HoloMed will also be used for other operations in the future. Since the system is, firstly, easy to use, and secondly, cost-efficient, the inventors say it is ideal for lowering healthcare costs. Plus it would also benefit poorly financed hospitals in emerging countries.

Cover photo: Dr. Michal Hlavac from the University Clinic for Neurosurgery Ulm and Christian Kunz from the “Health Robotics and Automation” (HERA) KIT team evaluating the HoloMed system during the initial surgery simulation with a dummy. (Photo: KIT-HERA).

Start-up of the week: clean your house with acorns

”Your sneak preview of the future” is the slogan of Innovation Origins, and that’s just what we will highlight with our Start-up of the Week column. Over the past few days, five start-ups of the day have been featured and on Saturday we will choose the week’s winner.

Innovation Origins presents a Start-up of the Day each weekday

We shall consider various issues such as sustainability, developmental phase, practical application, simplicity, originality and to what extent they are in line with the Sustainable Development Goals  of UNESCO. They will all pass by here and at the end of the week, the Start-Up of the Week will be announced.


PlaygroundVR – Playing outdoors while stuck in a hospital bed

A long-term stay in a hospital is no fun at all for a child. Outdoor play is often sorely missed. A Dutch team developed PlaygroundVR and this is exactly what you would expect from it. By using VR-glasses, children are able to take a trip to a colorful online playground where there is plenty to do. Fantasy and creativity play a prominent role and physical games are also a possibility thanks to the virtual reality feature. Gaming numbs your brain?!? – What a 20th century mindset!

The game is connected to the internet so that anyone, anywhere in the world, is able to join in the fun. It can also be a solution for healthcare providers because it offers a bit of a distraction when it is time for those unpleasant medical procedures. Lastly, it offers a good alternative to children who need some distraction but are no fans of Clini Clowns.

Axiles Bionics – prosthetic feet with joints

The technology for feet prosthetics has been standing still for fifty years according to the founders of the Brussels start-up Axiles Bionics. The wooden peg prosthetic as used by a stereotypical pirate is more accurate than you may think – even in the second decade of the 21st century. Of course it does have the shape of a foot, but the way in which it works has not changed over time.

Nevertheless, a human foot is a lot more complex than a piece of solid wood. As a result, wearers of these prosthetics will never be able to move as effortlessly as they used to and will experience back pain due to unnatural body posture. Yet developments are not standing still, robotics might very well prove to be the answer. A complex mechanism of springs and motors is able to accurately simulate the functioning of muscles and joints. The team is working towards an affordable prototype that should last for a good number of years.

Skoon Energy – A marketplace for large-scale batteries

The benefits of the Sharing Economy have become increasingly evident during the second decade of this century. For instance, a drill is used merely a few hours each year while it is kept in storage for the rest of the time. It’s a shame and extremely inefficient. Surely it would be much more practical if you shared your sunbed with a large group of people who could each use it when they wanted to?

Those at Skoon Energy think that the same thing is happening when it comes to large batteries. Festivals, construction sites or ships all need a lot of power, although not all year round. To be able to meet this peak demand for electricity, this Dutch team makes it possible to have a mobile battery delivered in a container as conveniently as a home-delivered meal. And when it is no longer needed? Then it can just go on to the next destination.

Felyx Sharing – Thousands of share-scooters

In an average big city when you look all around you, it’s impossible to imagine the streets without all those brightly colored shared cars and shared bikes. This trend has actually evolved quite fast, but what struck the founders of Felyx Sharing was that there was at the time hardly anything like this available for electric scooters.

Two years on, Felyx Sharing now has a fleet of 1250 flexible rental scooters in Amsterdam, Brussels, Rotterdam and The Hague. Simplicity is essential here. Users install an app and are able to drive off with just a few clicks of a button. The parking spaces are unattended and located at several locations. The team is ambitious and has big plans ahead for scaling up!

Not convinced? Try driving into the center of Amsterdam with a car.

Herbi Clean – Cleaning with acorns

Cleaning products typically contain a lot of harmful chemicals. It is not without reason that those orange warning labels appear on the packaging. Yet Mother Nature also has a cleaning lady hiding in her, as the Polish company Herbi Clean has proven. They came up with cleaning agents made of acorns which do not need any ominous orange warning labels.

How do they do that? This is due to tannin, a substance that plants produce in order to protect their seeds from greedy herbivores. Acorns contain 7% of this substance which has an anti-bacterial effect. And it is precisely these microorganisms that tannin fights that are the main cause of a lot of grime. There are already six products on the Polish supermarket shelves, including a toilet cleaner and a detergent.

It is actually quite odd that not more research is being done into cleaning products made from plant-based material. Why should we spray our homes with dangerous substances or artificial chemicals if there is a substance in nature that does exactly the same without the disadvantages? As there seems to be a lot more to be gained from this, Innovation Origins Herbi Clean has been awarded the title of Start-up of the Week.

Start-up of the day: Virtual playground supports recovery of children in hospitals

A virtual playground for children in hospitals. PlaygroundVR is putting the finishing touches to the technical development of their virtual reality environment. “Playing outdoors is invaluable for children’s development,” says Freek Teunen, co-founder of PlaygroundVR. “Outside, there is plenty of scope for their imagination. They do a thousand things with just a ball.” Former students of Eindhoven University of Technology also want to offer this experience to younger hospital patients.

Children can be adversely affected if they are in hospital for a longer period of time and are unable to play outside as a result. “By wearing the VR glasses, children can imagine themselves to be at a playground just like any other playground they normally have in their daily lives. There are also fantasy elements such as a large music box or flying hoops,” Jason van Eunen explains, co-founder of PlaygroundVR. The whole thing is very colorful. “The children can see each other through virtual avatars. So they are really able to play together.”

How did you come up with this idea?

We are very passionate about the technology. That’s why we wanted to see how we might make Virtual Reality (VR) as useful as possible. The advantage of VR is that the user is able to forget for a moment where he or she is and can completely immerse themselves in another world. Subsequently, we soon discovered that our technology would be very beneficial for hospital patients. They aren’t able to play like they do at home. As the children themselves have indicated, they miss playing outside the most. That’s why we have developed a virtual playground.”



Doesn’t that just mean that you are making children sit in front of a screen even more?

Teunen: “We certainly don’t want children to be wearing VR glasses all of the time. That’s why we also work with play sessions, a short period when children can play with the glasses. Although we do consider that our virtual playground is different from the shooting games that children often play on screens. Of course we would like to see children playing outdoors. But if that isn’t possible, VR glasses are a pretty good solution. We still connect children with each other and give them the experience of playing outside.” Van Eunen adds to that: “It would be better if children could learn new games in the virtual playground and eventually apply them in real life. We want to further stimulate imagination and creativity that way.”

How are you going to finance this?

“We have launched a crowdfunding project this week. Our goal is to raise 55 thousand euros. Then three hospitals will be able to use the virtual playground in their children’s ward. Approximately half of that money will be used for the latest developments in technology. We will use the rest to further roll out the concept,” Teunen states.

“The goal behind crowdfunding is to show hospitals the pedagogical benefits of virtual playgrounds. This is difficult with VR because it is a completely novel technology. We first have to prove that it works. That’s what we’re going to do with the money that we raise. We think that children feel better because of the playground and as a result are able to recover much faster.”

“Financing this product with donations doesn’t seem right to us. That’s why next year we want to see how we could use the virtual playground as part of a course of treatment. We are hoping that the playground will become a healthcare resource so that it can be paid for out of medical insurance costs.”

Why did you opt for crowdfunding?

“We think crowdfunding is very appropriate for our objective. PlaygroundVR is a start-up and definitely contributes to society. We provide children with moments of happiness at a time when they are extremely vulnerable. Crowdfunding is a way to build support,” says Van Eunen.

© PlaygroundVR

How can VR be part of a treatment?

Van Eunen: ” Distractions are very important to children when they are in hospital. That’s why there are pedagogical staff in the children’s ward and  CliniClowns and iPads to keep the children busy. PlaygroundVR is distracting in several ways because it ensures that the child feels like he or she is completely out of the room. Instead of watching a movie, the kids are really immersed in it.”

“This distraction caused by wearing VR glasses can be used during a treatment. For instance, when a child is holding an infusion needle in their hand. This is an unpleasant experience because it doesn’t feel right and it’s not nice to watch either. A child can play outside with the VR-glasses on and is less aware that a needle is being stuck into their hand. This helps to avoid pain and maybe even painkillers. This allows a child to get through a whole treatment feeling a lot less fearful.”

“We spoke to a girl who literally said that she had forgotten for a short while that her leg was broken. Because she could just walk in a playground. Those are the wonderful things about VR. You can’t do that with an iPad,” Teunen says proudly.

Was there a time when you were afraid that the concept was not going to work?

“Yes,” says Teunen. “As soon as we started work on this three years ago. We took part in a pitch competition at the time, but were turned down right away. Back then, the VR industry had not yet developed very far and the technology was still quite unknown. Now we can see that healthcare professionals and parents do appreciate the value of it. They really like PlaygroundVR better than, for example, a Playstation with Call of Duty.”

“The challenge for us is to find hospitals that dare to take this innovation aboard. Institutions often find these kinds of new technologies exciting to implement. We are already in talks with a number of hospitals that are interested as it is. We will be launching the virtual playground at the Wilhelmina Children’s Hospital towards the end of the year.”

It sounds like an important aspiration to help children. Do you have a concrete goal that you want to achieve?

“Of course, it’s cool if we can improve the hospital experience for a few children. But in the long term we want to be able to connect children with each other who are in different hospitals. Then children who have the same conditions will be able to share their experiences with each other. This way, we not only want to forge new contacts between children, but we also want to bring children together with their families and friends. As an example, we could connect children in the hospital to their classmates so that they would be able to play together in the playground,” Van Eunen explains.

Teunen continues: “We want to show that it is more fun to invent a game that has a real impact. Aside from that, we want to contribute to the VR revolution in healthcare. VR can really make a difference in healthcare. Not just for children, but also for people with dementia, for example. We are  hopeful that PlaygroundVR will inspire others to make great applications for healthcare as well.”

Tomorrow is Good: a digital health strategy for the hospitals in need

ziekenhuis failliet

Bart: Lucien, it’s your turn again for the column

Lucien: Thanks, that’s just in time. I’m breeding on something, here at Lisbon Airport

Bart: That’s a perfect breeding ground

Lucien: I’m tempted to write about the two hospitals that just went bankrupt. But I’m going crazy because of all those people with all those opinions already.

Bart: It’s a relevant topic for our audience. Just go ahead!

That’s how this conversation with Bart started on a subject you don’t really want to write about because a hospital that is forced to close its doors while it’s still full of patients is very very sad of course. For patients and their families, for employees, but also for those involved in home care, the general practitioner and insurance companies, etc. There are personal dramas taking place everywhere.

At times like this, everyone asks the questions about how this could have happened and how this should not happen and just has to be solved by money. You can hear that now (again) from those people who get angry over every single euro more for the care in times of quiet.

Now we have to look for an adequate solution, and behind the scenes, people will surely be working feverishly on that. I don’t know to what extent new models of care will emerge, but right now there is an opportunity. Now that a new plan has to be drawn up, we could agree on a number of starting points, such as the impact and use of digital care as a choice for patients, where it is logical and feasible.

I was in Lisbon today to present myself to the Portuguese Order of Doctors, and on several occasions after my lecture, I received the message that they are not doing anything about this at all in that country. Although I know that the government here in Portugal is indeed working on it, it still is a signal, also in terms of disconnect. We don’t ‘find’ each other. The patient who increasingly wants to use digital tools, just like in many other parts of his daily life. George Post (a professor from the US with whom I recently sat in a panel in Cambridge UK) called this ‘social recalibration’, a term I have lovingly adopted. The health care industry would do well to apply some of the most normal things in our daily lives as well, like video consultations (nowadays also properly reimbursed as a service), email traffic (there is a safe way available for it) or blood pressure and weight measurement from home.

Could this have averted this crisis in these hospitals? No, of course not. There is no quick fix or silver bullet for the current challenges that hospitals also face.

Still, for the people who want to take the initiative in Lelystad: please think seriously about a digital strategy as a part of your solution, it’s really 2018! I’d like to think along with you.

About this column:

In a weekly column, alternately written by Maarten Steinbuch, Mary Fiers, Carlo van de Weijer, Lucien Engelen, Tessie Hartjes and Auke Hoekstra, Innovation Origins tries to find out what the future will look like. The six columnists, occasionally supplemented with guest bloggers, are all working in their own way on solutions for the problems of our time. So that Tomorrow will be Better. Here are all previous episodes.