Virtual makeovers are better than ever. Beauty companies are trying to cash in

Agustina Sartori might as well be a chameleon. Her lips keep changing color, from matte red to glossy cherry to shimmering peach. Her eyes change, too: her eyelashes thicken, and her eyelids turn glittery silver, then plum, then orange. Suddenly, her eyebrows thicken. We are watching a live view of her transformation on her iPhone screen, where she looks ready for a fun night out. Sitting next to me in real life, however, her makeup is much more subtle. It’s not magic; Sartori is playing with a live virtual-makeover feature in cosmetics retailer Ulta Beauty’s iPhone app. It uses augmented reality and a phone’s front-facing camera to test thousands of shades and textures of lipstick, eyeshadow, and other cosmetics by applying them instantly to your face. Sartori is the founder of GlamST, a startup that Ulta Beauty bought late last year. Now the director of augmented reality innovation at Ulta, her technology helps power the company’s smartphone apps and is a striking example of how far augmented-reality makeup has come. A shakeup for the makeup world Augmented reality, or AR, a tech that overlays virtual images on live video of the real world, has been available in a handful of mobile apps for years — like kids games or an IKEA app that lets you rearrange virtual furniture in your real home. But with the exception of, say, Pokémon Go, it has largely failed to catch on with consumers. Virtual makeup, one of the first triumphs of the technology, is helping AR go mainstream and sell real-world products, too. AR has become eerily accurate enough that major beauty companies including Ulta Beauty, Sephora, and L’Oréal are using it to try to sell their latest products. And consumers are interested in trying it as a way to experiment quickly, without touching any icky tester products. The illusion of AR makeup is helped by improvements in underlying technology like facial feature and finger tracking. Smartphone operating systems from Google and Apple now offer advanced AR tools, and the latest phones have much more powerful front-facing cameras than they did just a few years back. The app makers have also done meticulous work to match real-life colors and textures of beauty products with what pops up on your smartphone screen. The latest killer feature, at least for the companies trying to make money, is the ability to try on, say, Orly nail polish in Seize the Clay or Urban Decay Moondust Eyeshadow in Intergalactic, and immediately buy those same products from the phone. For the companies making and using this technology, trying on makeup via AR makes perfect sense since the idea of trying before you buy is so entrenched in the makeup shopping experience. “If you have 100 lipstick shades, there’s no way to try them all on in real life, realistically,” said ModiFace founder and CEO Parham Aarabi, whose AR cosmetics startup L’Oréal bought last year. “But in AR you can try them all in about a minute.” Hoping AR eyeliner can help the bottom line Virtual lipsticks and smokey eye shadows are popular in apps, but are they translating into more makeup sales? Hard data isn’t easy to come by, but several companies say it does encourage people to play around with different colors and can also convince them to open up their wallets. L’Oréal chief digital officer Lubomira Rochet said people typically spend more time on an app or website that has AR makeup or skin-care features, and those who try it buy a product 10 percent more often than those who don’t. This, she said, “makes a huge impact on the bottom line.” At Ulta Beauty, Prama Bhatt, senior vice president of digital and e-commerce, said tens of millions of colors have been tried on, and people who use the virtual tools to test an item are more inclined to add that product to their digital shopping bag and purchase it. Similar to Sephora, the company is considering how AR may work beyond the mobile app. While Ulta Beauty only has AR makeup try-on stations in a handful of its 1,163 stores, Bhatt said it’s planning to test out a hair-color try-on app in some stores with AR technology from Perfect Corp. (That company already offers its own app, YouCam Makeup, which attempts the technically difficult task of letting you test bright shades of Punky Colour hair dye. It doesn’t look nearly as realistic as the app’s faux makeup.) “Like other industries, specifically in beauty, the ability to discover, sample, try on is a critical part of the journey,” Bhatt said. “So it makes a lot of sense from a business perspective as well as an experiential perspective.” It may soon become something that consumers expect to be able to do before they spend money on a makeup purchase, said Brendan Witcher, a Forrester Research analyst who covers digital business strategy. But while it could help you find something new or see how several products will look together, he’s not sure how helpful it will be once you find makeup you really like. “I’m not going to go to augmented reality to buy the same lipstick I’ve been buying for three years,” he said. Augmented history of virtual makeovers The technology took a long time to get here. When Aarabi started ModiFace in 2006, the iPhone hadn’t even been announced yet. There was no such thing as an app store, let alone an app that you could use to try on dozens of lipstick shades in minutes. Aarabi is a professor at the University of Toronto who studies artificial intelligence for face tracking. He and a colleague started out by first creating a website that let people upload a still image of themselves and swap their lips, eyes, and nose for someone else’s (you might switch your lips with, say, Angelina Jolie’s). It was “just for fun,” but then Allergan, a pharmaceutical company whose products include anti-aging products like Botox and Juvéderm, asked them to make a tool that let people upload a photo of their face and see how anti-aging products improved their features. It was a tool on a website? Another early partnership was a skin spot-correction tool for Garnier, a hair and skincare brand owned by L’Oréal. “The more we got into it, the more we realized there was a need for simulation in virtual makeovers and try-on in the beauty space,” Aarabi said. Last March, L’Oréal acquired ModiFace, which offers several of its own virtual makeover apps and licenses technology to other companies. Sephora now uses ModiFace to power virtual makeovers in its apps and on mounted tablet computers in stores. For years it was impossible to make AR makeup look realistic on a live video feed of a smartphone user’s face. ModiFace began performing makeup simulations on still photos in 2009 but didn’t roll out a live video feature in an app until 2013. And when Sartori started working on GlamST in her native Uruguay in 2014, she initially planned on using the technology for in-store touch-screen kiosks, rather than as a mobile app. Laggy lipstick and other room for improvement As impressive as it is to see AR winged eyeliner that stays on your eyelids when you turn your head, there are still notable differences between virtual makeup and the real thing. Showing virtual makeup on a real, live face — which can be twisting, winking, smiling, or frowning — involves a complicated hardware-and-software tango. In a nutshell, an app has to detect a person’s face in each frame of a video taken by the phone’s front-facing camera, overlay an image (say, eyeshadow and blush) atop just the right parts, and show it to the user, all within milliseconds. If the timing is off, digital lipstick will lag behind your lips as you turn your face, or eyeshadow will dance above your eyelids. The virtual products aren’t always a match for their real world counterparts either. To make sure the colors come as close to reality as possible, these companies must first photograph swatches of the cosmetics in different lighting and on different skin tones, paying attention to aspects like finish, coverage, and texture. “We’re really trying to say, ‘What you’re trying on here is super similar to what you can try in real life,’” Sartori said. “That piece for me is very important because if you’re shopping online and want to see different shades, when the product arrives you want it to be similar.” To get a sense for how well this can work, I invited Sartori to show me some makeup — real and virtual — in person. She tried on a host of virtual lip colors while sitting in our office, which mostly looked realistic and only occasionally seemed a bit askew from her real lips. Then, she compared them to swatches of the same makeup on her arm. The digitally rendered hues matched the real ones; a metallic-looking Kylie Cosmetics lip gloss glimmered just the way you’d expect it to, while a matte red MAC Cosmetics lipstick appeared muted and thick. Looking beyond blush to hair and shoes In the near future you may be able to virtually try on more than lipstick, blush and eyeshadow. Wannaby, a startup based in Minsk, Belarus, that is working on ways to try on three-dimensional products like shoes. Wannaby, which already offers an app for checking out different shades of nail polish, recently released an early version of an iPhone app called Wanna Kicks. It lets you use the iPhone’s rear-facing camera to see how sneakers from companies such as Nike, New Balance, and Adidas look on your feet. If you like what you see, there’s a link in the app to buy the shoes online. The tech has a long way to go. The foot-tracking is still finicky — it doesn’t work well when you move your foot — and there were fewer than a dozen 3-D shoe models (each in various patterns) in the app in mid-February. It does give a sense for what the future could look like as smartphones and the technology underpinning augmented reality continue to improve. Beauty companies are also hoping to use AR to make virtual makeovers even more realistic looking. They’re working on tricky tasks like figuring out how to render glitter correctly, and making extreme hair-color changes look normal. They’re also considering how to personalize product suggestions, in hopes of getting you to try virtually and then buy even more. But this may be even harder than showing you lipstick or sneakers that appear to be on your feet. When it comes to understanding what users care about, Aarabi said, “That remains a challenge.”