Augmented reality demos hint at the future of immersion

Twitter is awash with impressive demos of augmented reality using Apple’s ARKit or Google’s ARCore. I think it’s cool that there’s a palpable sense of excitement around AR—I’m pretty excited about it myself—but I think that there’s perhaps a little too much early hype, and that what the demos don’t show is perhaps more suggestive of the genuinely exciting future of AR.

Below is an example of the demos I’m talking about — a mockup of an AR menu that shows each of the individual dishes as a rendered 3D model, digitally placed into the environment (and I want to make clear I’m genuinely not picking on this, just using it as an illustration):

This raises a few questions, not least around delivery. As a customer of this restaurant, how do I access these models? Do I have to download an app for the restaurant? Is it a WebAR experience that I see by following  a URL?

There’s so much still to be defined about future AR platforms. Ben Evans’ post, The First Decade of Augmented Reality, grapples with a lot of the issues of how AR content will be delivered and accessed:

Do I stand outside a restaurant and say ‘Hey Foursquare, is this any good?’ or does the device’s OS do that automatically? How is this brokered – by the OS, the services that you’ve added or by a single ‘Google Brain’ in the cloud?

The demo also raises important questions about utility; for example, why is seeing a 3D model of your food on a table better than seeing a 3D model in the web page you visit, or the app you download? Or, why is it better even than seeing a regular photo, or just reading the description on the menu? Do you get more information from seeing a model in AR than from any other medium?

Matt Miesniks’ essay, the product design challenges of AR on smartphones, details what’s necessary to make AR truly useful, and it proceeds from a very fundamental basis:

The simple question “Why do this in AR, wouldn’t a regular app be better for the user?” is often enough to cause a rethink of the entire premise.

And a series of tweets by Steven Johnson nails the issue with a lot of the demos we’re seeing:

Again, I’m not setting out to criticise the demos; I think experimentation is critical to the development of a new technology—even if, as Miesnieks points out in a separate essay, a lot of this experimentation has already happened before

I’m seeing lots of ARKit demos that I saw 4 years ago built on Vuforia and 4 years before that on Layar. Developers are re-learning the same lessons, but at much greater scale.

But placing 3D objects into physical scenes is just one narrow facet of the greater potential of AR. When we can extract spacial data and information from an image, and also manipulate that image digitally, augmented reality becomes something much more interesting.

In Matthew Panzarino’s review of the new iPhones he talks about the Portrait Lighting feature—which uses machine learning smarts to create studio-style photography—as augmented reality. And it is.

AR isn’t just putting a virtual bird on it or dropping an Ikea couch into your living room. It’s altering the fabric of reality to enhance, remove or augment it.

The AR demos we’re seeing now are fun and sometimes impressive, but my intuition is that they’re not really representative of what AR will eventually be, and there are going to be a few interesting years until we start to see that revealed.

Trends in digital media for 2017

Alright, stand back everyone: I’m about to have some opinions about technology in 2017. Because obviously there’s been a shortage of those.

As part of my Technologist role at +rehabstudio I put together internal briefings about digital media, consumer technology, where the digital marketing industry could go in the near future, and what we should be communicating to our clients. Not trying to make predictions, but to follow trends.

This article is based on my latest briefing. It’s somewhat informed, purposely skimpy on detail, and very incomplete: I have some thoughts on advertising and publishing that I can’t quite distil yet, and machine learning is a vast surface that I can barely scratch.


If for nothing more than press coverage, 2016 was the year of messaging, and the explosion of the messaging bot. The biggest player in the game, Facebook’s Messenger, launched their bot platform in April, and by November some 33,000 bots had been released. Recent tools added to the platform include embedded webviews, HTML5 games, and in-app payments.

The first six months of bots were largely the ‘fart app’ stage, but there are signs that brands and services are finally starting to see the real opportunities in messaging: removing friction from their users’ interactions with them. Friction in app management and UI complexity, for example.

The same removal of friction is also a key driver behind the growth of home assistants and voice interaction, like Alexa. Removing the UI abstraction between users and tasks is a clear trend. As an illustration, compare two user flows for watching Stranger Things on Netflix on your TV; first using a smartphone:

  1. Unlock phone.
  2. Find and open Netflix app.
  3. Press the ‘cast’ button.
  4. Find ‘Stranger Things’.
  5. Play.

Now using Google Home:

  1. “OK Google, play Stranger Things from Netflix on My TV.”

Home assistants make the smart home easier to manage. No more separate apps for Wemo, Hue, Nest, etc; a single voice interface (perhaps glued together with a cloud service like IFTT) controls all the different devices in your home.

Messaging and voice are visible aspects of the trend towards the interface on demand:

The app only appears in a particular context when necessary and in the format which is most convenient for the user.

While native mobile apps are still a growth area, it’s becoming much harder to get users to download and engage with apps outside of a small popular core. This is especially true for retail, where consumers are more omnivorous and like to browse widely.

Improvements in the capabilities of web apps (especially on Chrome for Android) suggest an alternative to native apps in some cases. This has been demonstrated by the success of new web apps from major retail brands like Flipkart and Ali Baba in developing economies where an official app store may not be available, or network costs may make app downloads undesirable.

Web apps require no installation, avoiding the app store problem. They’re starting to get important features like push notifications and payment APIs. And messaging platforms, with their large installed user base, provide the web with a social and distribution layer that the browser never did:

Messaging apps and social networks [are] wrappers for the mobile web. They’re actually browsers… [and] give us the social context and connections we crave, something traditional browsers do not.

So it may be that for some brands, a website optimised for performance, engagement, and sharing, along with a decent messaging and social strategy, will offer a better investment than native apps and app store marketing. Patagonia already closed their native app. Gartner predict that some 20% of brands will follow by 2019:

Many brands are finding that their mobile apps are not paying off.

The most important app on your phone could be the camera, which will be increasingly important this year. First, by revealing the ‘dark matter’ of the internet: images, video and sound. So much of this data is uploaded every day, but without the semantic value of text, it’s meaning is lost to non-humans — like search engines, for example. But machine learning is becoming very good at understanding the content of this opaque data, meaning the role of the camera changes:

It’s not really a camera, taking pictures; it’s an eye, that can see.

It can see faces, landmarks, logos, objects; hear background chat and music. That’s understanding context, location, purchase history, and behaviour, without being explicitly told anything. This is why Facebook, through Messenger and Instagram, are furiously copying Snapchat’s best features: they want their young audience and the data they bring.

Will it be intrusive? Yes. Will it happen? Yes. I’ve tried to avoid making hard predictions in this piece, but I am as confident as I can be that our image and video history will be used for marketing data.

Cameras will also be important in altering the images that are shown to the users. Augmented reality is an exciting technology, although good-enough dedicated hardware is still a while away. But there’s a definite market drift in that direction, and leading it is Snapchat: they’re stealthily introducing AR through modifying the base layer of reality—first, by altering faces using their lenses. This isn’t frivolous; it’s expanding the range of digital communication, like emoji do for text.

If people are talking in pictures, they need those pictures to be capable of expressing the whole range of human emotion.

Recent Snapchat lenses have started altering voices, and your environment. They’ve recently bought a company that specialises in adding 3D objects into real environments. With Spectacles they’re not only removing friction from the process of taking a photo, they’re prototyping hardware at scale. This is the road to AR. Snap Inc. want to be the camera company — not in the way that Nikon was, but in the way that Facebook is the social company.

The companion to an augmented reality is a virtual one, but I don’t believe we’ll see VR going mainstream in 2017—and I say that as a proponent. It’s static, isolating, and it requires people to form a new behaviour. It’s interesting to see creators experiment with the form, and I’ve no doubt that we’ll see some very interesting experiences launched this year. But domestic sales aren’t huge, and high-end units are too expensive, and low-end not quite up to scratch yet. Still think it will be big for gamers, though.


I have more. A lot more. But I think it will all be better explained in a series of subsequent blog posts, so I’ll aim to do that. In the meantime, would love to hear your thoughts, arguments, objections, and conclusions.