XR Experience at Volvo Cars
Timmy Ghiurau

— You all know that modern industry is changing. We’re moving more to become a mobility provider and working to become like a tech company in the future with more software & technology in cars. Also gaining understanding how we can build unique system like how our car can afford more services beyond mobility. Volvo’s vision is a freedom to move in a personal safety sustainable way and we always been having a safety as a core. Now our task is to look at how we can use these values & principles that we have beyond mobility, how we connect the different partners. This is the area that I lead immersing & emerging technologies. We do collaborate with a lot of startups, universities to engage in research and push boundaries beyond always seeking efficiency and what is beyond efficiency. That’s kind of how some of the tasks are to make sure that we have some long-termed strategy for specific areas and always trying to push boundaries within technology & creative segment.

— What is one of the biggest challenges? Automotive is the most traditional industries out there. It’s super slow for when the company is really big the decisions are made really slow. In that sense any efficiency, speed are kindly appreciated because it helps the whole flow. It’s really important for ambidextrous mindset in connection with XR for with this technology we can help to design, validate concepts and do research faster. We’re helping to produce faster, design faster and make it cost-efficient. We buy time technically focusing on what we want in the future in more experimental areas but at the same time we need to serve both purposes looking for emergence and using more ways for these technologies. The challenge is to understand the main vision, to understand why we do & what we do. By doing that we are always being prepared to tackle. Partly hard challenge is the transformation for it is kind of an identity crisis most car-companies are trying to understand. We are the premium brand, the sustainable brand, mobility brand or like a tech company we are always into this transformation. Sometimes it is hard to navigate and that’s why you need to be super grounded in area you work for it is a kind of stranded foundation and give it a sense of identity. For me it probably was even more challenging because it does come from the creative industry, from music & fashion, from gaming. Working with Unity in VR and then coming to a car-company where everything was different… It’s also super satisfying to see things you contribute into impact millions of lives, save lives, can offer their users experience. So you can see the value much faster. That’s why working with the future sometimes brings challenges that actually prove value immediately on a short term.

— Everything from artificial intelligence because we’re looking to maintaining being leader in safety and adapting to work with a lot of sensors, computer vision, light art technology for self-driving cars. Then we are having this collaboration with Google where we have the Android Automotive OS upgrading system in the car so we need to think about users’ experience in the car in the different way and still keep it safe & sustainable because you don’t want it to be too destructive. And also that stuff with Blockchain where we actually are tracing every material & component in the car like: how much cobalt, lithium, and carbon print we have. That’s always a challenge how not to be hypocritical as a car company talking about sustainability. We want to reach our goals, trace back and lead in the industry that raises a weariness that is important. We’re looking to using these technologies for being able to reach our sustainability goals. Another aspect is about XR we do use. Volvo’s been always using VR since 80s for most car companies have different cave systems like Chunky VR headsets with magnetic field. It makes sense since we work with car design, a lot of featurations, evaluations, testing & interaction. It does save a lot of time & money. We do sit with every new headset that we get. We have a super strong collaboration with Varjo, we invested in that company & for us that is the only headset we can actually use for proper reviews & researches since it has hi-resolution, accuracy in the materials, ability to read text. That’s important to simulate and do validation mark.

— We tried everything; we’ve always been leaders in this area. We had first VR launch with XC90 with the Google Cardboard. Then the first HoloLens launch & activation, we were part of it with Microsoft, Varjo launch for VR 1, XR 1, we were first to be able to drive with the headset on the road and we pushed our limits, Varjo limits & Unity limits at the same time because we were crazy enough to do so. We have everything working with Tesla suit, Google, Apple. We do want to be a leader and a part of that Metaverse in that sense. In a classic sense AR is really useful for retail car configurators. We did tests, launches. With VR it is nice to put products in context: like how XC90 looks in Shanghai, on Stockholm streets. What we love with Varjo & VR 1 is the first time you can switch between them. All that video pastries while I drive XC90, I can replace the car while I drive an auto car concept that doesn’t exist. I can just change colors, I can put barriers in front of the cars so we can test safety scenarios & scenarios that are too dangerous to test in real life so we wouldn’t put our users in danger for we can do it in VR. Both the user and the car sensors think that there is a real pedestrian in front, and then we measure response time, training our models. That is why for XR that part where we can shift between VR & AR is really powerful and it’s like a teleportation. When we launched XR 1 together with Varjo we had the virtual car next to the real one, same configuration and you could see a reflection in both, and people were confused in that sense because your brain adapts. And they were like – I thought there was real one & now it disappeared and that one appeared now. So it’s powerful with that context.

— It was successful especially in the beginning for when you launch a car model or a car concept you have limited physical cars. So we could reach more markets, journalists that way. That time it was 2014 and we did it. Of course with the resolution it was very hard to work with the both 360 degrees videos but in itself being able to convey the story and highlight special features, design changes that we made, all the functions, safety functions we have really helps to convey the story & highlight the features we wanted to highlight. Same time together with that launch we did projection map and launched it in the museum where journalists were invited and we projected on the XC90 the dynamic care flow & different sensors where we highlighted the design of cars and its statics. I would say it was successful. Google Cardboard is accessible and super cheap for people to have access on mobile phones so it was easy to spread it out. But that time it was nothing directable, resolution was not great but it did job.

— The most interesting thing that I worked on was driving in VR. That was a challenge for us, Varjo, Unity and we joined forces. To be honest, everyone thinks we spent years to do it but we did it actually in 3 days – me and 2 designers. We had a car model done already, we built an automated pipeline and made Volvo asset libraries so if we used a car in a simulator or in car configurator when not using it in this experience, the most was used in sensor verification when trying it in a retail experience so being able to repurpose it driving it in Unity just to test out. When Varjo approached us and they came to our office with tests, the resolution was amazing and our first question was: cool, but can we drive with it? They went back home, did some homework and 3 weeks later they came with VR headset which was the prototype before VR 1. The resolution, the video feed, the latency were high, I was driving the first time and I did say on the record before that maybe Volvo is not happy because driving about 160 per hour first time with the headset on the test track in safe environment. I didn’t get emotional sickness, but still at turns we overlaid 100 million polygon cars that a super realistic with a lot of computers and batteries in their trunk to run it. I think that most time we spent on finding the right tracking system so we started to collaborate with the ART (ARTTRACK) which made it professional art tracking system. Then Varjo came back again and it was even better. The latency that we got was a single digit photon to photon – from 4 to 8 milliseconds when you really don’t know anything. We still use it and the car is used every day to prepare for the launch of our new platforms to evaluate it with users. It has an eye-tracking. We measure an interaction, sensations, emotions, stress. It’s a really powerful platform. After that we were thinking about 360 degrees cameras. Why not embedding that into the headset then we make a car disappear completely when we drive. Suddenly you can replace the car with whatever car you want, you remove everything that is on the road. It is a really interesting sensation.

— It’s been a buzzword for a while the same as VR was a buzzword for a few years. We do see the seriousness of it and as we are strong players in it with our child-company together with Balenciaga… For this game where we launched a car concept and they launched a collection in the game. And we see Christian’s Ferrari launching a modelling Fortnite. It’s a big space especially we’re reaching for the new generation. If we want to be fresh and new for the generation that is upcoming – they don’t watch TV, have Adblockers & new tips that some of the commercials will never reach them. You need to find new ways to make it interactive, to work to make sense. We’re focusing also on a storytelling. We’re having collaboration with Alex McDowell who is the Creative Director of the Minority Report, Avatar, Men of Steel and so on. These are great world builders. We tried to learn his methodology for storytelling but also used as a platform for innovation. Putting ourselves in future scenarios in… let’s see… we’re in 2024 and the world will look like this. Does the Volvo make sense in this scenario or that scenario? Does the safety still make sense or the cars are self-driving and this makes sense? I think that type of Metaverse we actually are putting ourselves in various situations and we invite others to take part in that. This is the way we will see the world and participate in building together. That’s interesting. What Mark Zuckerberg announced comes back to remote meetings, tele presents and that was accelerated by Covid. I remember I was a bit upset when Covid started because all my conferences and VR conferences were cancelled, moved online. I was a bit sad as I missed the community. Some of the conferences moved in VR. And I said ‘Let’s meet in VR’. I was a bit skeptical for working with it is not the same. Seven years ago we all meet in a Metaverse and we will all had these virtual meetings. It was the time to practice what we were reaching and it was amazing. The brain adapts so quick to know all this combination as perception, hearing the right special sounds which is a super crucial components in VR. We had an avatar that is close enough even cartoonish. Your brain makes correlations and then it becomes really comfortable to collaborate in the indoors environment.

— Totally! That’s where we are shifting also. We’re testing business models; we created various companies for shared mobility cared by Volvo. We have different car ownership models – the traditional way «Would be my car always» we’re shifting to «My car sometimes» to «Sometimes when I need the car» with car sharing aspects. It does open and makes it more accessible and also the way we’re thinking about user experience and design it does become more appealing to the newer & young generations. Still in that sense our approach that we have sustainability, personalization, it makes people perceive it differently, also being more conscious. The choices, impact we have on transportation and definitely it reaches the young audiences in another way.

— In the opened innovations area now we always are looking for new emerging markets. How to create a new market, how to work in an ecosystem, how to partner up with different companies we know, how to seek a mobility working with policy makers, city planners? How we can play a role in the future and how to sustain it in a personal safe way? That’s why we created smaller companies who focus on future of mobility, electrification, entertainment, self-driving car technologies that also be used to collaborate with other brands, businesses, cities. This gives us a bit of more agility and ways to work. We also have a tech fund that is an investment fund we use to invest in Luminar, Varjo. We have a few incubators & accelerators that bring startups and we do collaborations and integrations that really allow us moving faster.

— In XR? That’s an interesting one. I’ve seen a lot of movies and read some books as a young kid, seen the Matrix and was always imagining building worlds. I’m a dreamer and been dreaming a lot since I was a kid. Even in classrooms my mind was somewhere else. I have really big imagination imagining different contexts. I heard about VR in early nineties. I thought it is so interesting you put or immerse yourself in different worlds. Today I became really passionate about the human brain, perception, cognition, and eye-tracking. As part of this I’m tracking company that was bought by Oculus called THEEYETRIBE in Copenhagen. I understood that when we’ll live in the future we’ll have these wearable glasses, lenses, different sensors. That time I was thinking if we will all have these what is the right industry to learn from? And then I looked at automotive because autonomous cars are all already equipped with cameras, computer vision sensors, light art to make sense of the world around us. I’m trying to merge & learn from other sciences and look in the future influence and the traditional industry to follow up. This is my goal with XR. I’m a big fan of ethics and for me there is a reason to push boundaries and get to a point where I can raise questions – what if, why not, what’s too much, where should we stop? That is an interesting challenge for me and that’s where I am really involved.

— I do think entertainment, music, filming industries. You already seen some productions, I guess. I remember playing a computer game called Mafia in early 2000. There was a scene there where you had to break into museum, there were 2 security guards in their 50s talking to each other: ‘Hey, have you heard about new color TVs?’. The answer is: ‘What crazy times we live in imagining the future where you can control the movies’. And then they were talking about wearing something being inside the movie. What if we are in the movie? That scene triggered me. I was maybe 14-15 playing that game. I thought that I will really be disrupted by this being able to be a part of it being in the next window. Justin Bieber wrote a song if the possibilities are endless. Then of course in fashion & retail we already see some initiatives. It goes more for sustainability aspect in it. I can wear the same patterns, maybe a green-screen dress but most will see that I have a unique clothing patterns & styles. I do think that a lot of industries, architecture & construction. I see how real-time 3D in VR opens collaboration in building the house, and you are able to see, visualize it on sides and configure it on the way in real time and that is really powerful.

— I was a part of MIT Hackaton: The Reality Hack and there was a judge. I partnered with them and some of those participants… That’s where I saw Teslasuit in action I borrowed the participants our VR headset (the Varjo’s) and I was able in XR to float the AMT hall with water. With this suit you feel the temperature and the pressure of the water as you were there. That was like: WOW! I mean the impact that it had because everyone talks about climate changes. And when you suddenly there you feel that impact. One of my friends did an installation for World Economic Forum about the Amazon Fire called «Be Earth». He showed that it in Davos last year and we actually hired the guy. He is the creative guy that kicked out from film industry & entertainment. We hired him at Volvo now. But the story goes again… I was there at Amazon Fire and so for myself… It’s not only the forest that you think is so far from us. And that had again an impact. And then Half-Life:Alyx. For me that is the ultimate VR experience. I make all of our students, interns or new employees play it for a while to understand what is good interaction in VR, what is good quality, how do you design certain levels.

— I do want to believe more in VR because with it you can build whatever world you want. AR has lower entry point because it is more accessible to mobile phones. I do see it can have already a lot of impact. Personally I like VR – that part you may change between real world and virtual world like teleportation aspect. Probably you saw the latest announcement from Varjo where they made a reality cloud that is literally a teleportation. Through the headset linear cameras you can stream live. You guys can be in this room with me without me having to model it or create it. I think that’s an interesting stuff having these sensors for capturing reality with no need to model everything in the world.