For the past few years I’ve got into the habit of start­ing the new year with an arti­cle con­sol­i­dat­ing my thoughts on where we’re at with con­sumer dig­i­tal tech­nol­o­gy; look­ing at the land­scape, and at what the biggest play­ers are doing—my focus is most­ly on Ama­zon, Apple, Face­book, and Google, but it’s not exclu­sive­ly on them. I want to tease out a few trends to help ori­ent myself in my role for the year ahead. I try not to make pre­dic­tions, but per­haps play out some pos­si­bil­i­ties.

There are two big declines at the core of this year’s trends, which I think set the tone for where con­sumer tech might head in 2019. They are the smart­phone decline, and the Face­book decline.

The Smartphone Decline

The smart­phone mar­ket is in reces­sion. To be clear, hun­dreds of mil­lions of new smart­phones are still sold every year, but that num­ber seems to have peaked (it’s 6% down from last year in the last report­ed finan­cial quar­ter, accord­ing to IDC). There are many rea­sons for this, but the fun­da­men­tal truth is that most of the ‘easy’ growth is gone; it’s hard­er to sell to new cus­tomers, and peo­ple who already own smart­phones are upgrad­ing less fre­quent­ly.

Source: IDC, Jake Swearingen/Intelligencer

Despite what Apple say in each of their annu­al new iPhone pre­sen­ta­tions, hard­ware has pret­ty much reached a plateau; there’s very lit­tle dif­fer­en­ti­a­tion year-on-year, so peo­ple are hang­ing on to their phones for longer. Also, Chi­nese brands like Oppo, Huawei and Xiao­mi are mak­ing mid-range phones with excel­lent tech­ni­cal spec­i­fi­ca­tions that are tak­ing sales from flag­ship phones. It’s only the first week of 2019 and already Apple and Sam­sung have report­ed low­er than expect­ed sales.

There are a num­ber of notice­able effects of this: first­ly and most obvi­ous­ly, that brand-loy­al fans are being charged more for new flag­ship smart­phones; for exam­ple, the ‘bud­get’ iPhone XR is $50/£50 more expen­sive than last year’s iPhone 8, while the iPhone X and this year’s XS start at $999/£999. Sec­ond­ly, phone mak­ers are ramp­ing up their ser­vice offer­ings; from stor­age sub­scrip­tions such as iCloud, Google One, and Sam­sung Cloud, to media stores such as iTunes, Apple Music, and YouTube Music, the com­pa­nies want to make more mon­ey from their cus­tomers through soft­ware sales after the hard­ware pur­chase.

The third notice­able effect is that there’s a lot of exper­i­men­ta­tion with smart­phone form and fea­tures as man­u­fac­tur­ers look for ways to stand out from the com­pe­ti­tion. From sub-screen fin­ger­print scan­ners to in-screen speak­ers to fold­able screens, there are many new innovations/gimmicks. But the cur­rent are­na for com­pet­ing to dif­fer­en­ti­ate is the cam­era.

Computational Photography

While some man­u­fac­tur­ers com­pete on cam­era hardware—like Huawei’s Mate Pro with its three rear-fac­ing Leica cameras—most are mak­ing a show of how good com­pu­ta­tion­al pho­tog­ra­phy is becom­ing. It was notable that Apple spent a long chunk of its last iPhone pre­sen­ta­tion high­light­ing new com­pu­ta­tion­al cam­era fea­tures, and Google’s Pix­el 3 was sold almost entire­ly on its pho­tog­ra­phy soft­ware (and it’s prob­a­bly the best sin­gle-lens smart­phone cam­era avail­able).

As well as improv­ing pho­tographs through fea­tures like Smart HDR and Night Sight, there are also ear­ly steps in mak­ing the cam­era a key input to the device, with ser­vices like Google Lens, Bix­by Vision, and Pin­ter­est Lens. Google’s cam­era recog­nis­es plants, fab­rics, text, phone num­bers and web­site address­es, and a claimed one bil­lion prod­ucts, four times more than at launch. More cat­e­gories are set to be added in 2019 and I expect that Google will allow brands and ser­vices to add cus­tom train­ing to Lens to make their own prod­ucts and pro­mo­tion­al mate­ri­als search­able and, even­tu­al­ly, shop­pable.

Com­pu­ta­tion­al pho­tog­ra­phy also dri­ves the adop­tion of on-device neur­al engine chips, for the com­plex cal­cu­la­tions required for image pro­cess­ing. Apple’s A12 Bion­ic, Google’s Visu­al Core and Huawei’s Kirin 980 are all exam­ples of ded­i­cat­ed mobile chipsets for this task. These chips, with pre-trained on-device machine learn­ing mod­els, also enable oth­er machine-learn­ing pow­ered func­tions and will be increas­ing­ly at the heart of future mobile and wear­able devices.

Wearables

The fourth effect of the smart­phone plateau is that phone mak­ers are diver­si­fy­ing their wear­able hard­ware ranges. It’s inter­est­ing to note that at this year’s Con­sumer Elec­tron­ics Show (CES) some phone man­u­fac­tur­ers chose to dis­play wear­ables direct­ly along­side phones, not as a sep­a­rate cat­e­go­ry. The first wave of wear­ables is on the wrist (such as Apple’s Watch and Google’s renewed sup­port for WearOS), and in the ears—with head­phones, ear­phones and ear­buds fea­tur­ing built-in sup­port for smart assis­tants, such as Apple’s Air­Pods with Siri, Google’s Pix­el Buds with Assis­tant.

These hear­ables (to use the neol­o­gism) are an inter­est­ing cat­e­go­ry. Last year chip-mak­er Qual­comm released a new sys­tem-on-a-chip (SoC) that’s specif­i­cal­ly designed for wire­less ear­buds with third-par­ty smart assis­tant inte­gra­tion, mak­ing it much eas­i­er for elec­tron­ics man­u­fac­tur­ers to make their own smart ear­phones. Lat­er in the year they teamed up with Ama­zon to pro­vide a ‘ref­er­ence design’ for Alexa-enabled hear­ables, giv­ing man­u­fac­tur­ers fur­ther incen­tive to choose Amazon’s plat­form.

An emerg­ing cat­e­go­ry to watch out for is glasses/headwear. Some head-mount­ed devices have already launched, such as Microsoft’s HoloLens, the Mag­ic Leap One, North Focals, and Vuz­ix Blade. But there are still some fun­da­men­tal issues to over­come before they hit the con­sumer main­stream: they need good bat­tery life and high qual­i­ty sen­sors, both with­out mak­ing the device too heavy to be com­fort­able; the cost needs to drop sig­nif­i­cant­ly; they need to look like some­thing you’d be hap­py to wear in pub­lic (unlike Mag­ic Leap or HoloLens One); and, cru­cial­ly, there are issues with the lens­es still to be worked out (more about this lat­er in this arti­cle)

A cou­ple of oth­er emerg­ing tech­nolo­gies will boost wear­ables in the future. eSIM puts SIM card func­tion­al­i­ty onto hard­ware chips, so devices can direct­ly send and receive data over mobile net­works. This opens the door to autonomous wear­ables that don’t require your phone as a data hub, such as Apple Watch 3 & 4 and a smat­ter­ing of Wear OS devices. eSIM tech­nol­o­gy could also work for hear­ables, mak­ing them into inde­pen­dent­ly-oper­at­ing portable smart speak­ers. And the roll-out of 5G net­works will make it pos­si­ble to stream high-qual­i­ty data to future smart glass­es, pro­vid­ing the 3D assets that make rich aug­ment­ed real­i­ty expe­ri­ences pos­si­ble.

Smarter wear­ables indi­cate a move away from the phone screen and towards an atom­ic com­put­er, where actions are invoked from dif­fer­ent devices as appro­pri­ate; for exam­ple, see­ing mes­sag­ing noti­fi­ca­tions on your wrist, or trans­lat­ing a phrase through your ear.

Source: Luke Wrob­lews­ki

When the com­put­er is frag­ment­ed across many dif­fer­ent sur­faces you’ll need a sys­tem to man­age your mem­o­ry, iden­ti­ty and his­to­ry across them. And for sur­faces where there’s no screen avail­able, like ear­phones, the pri­ma­ry access to inter­net ser­vices will be through voice com­mands. The solu­tion to both of these is the smart assis­tant.

Smart Assistants

One of the biggest con­tests in con­sumer tech in 2018 was between Ama­zon and Google. More specif­i­cal­ly, the con­test is to make their respec­tive smart assis­tants, Alexa and Google Assis­tant, dom­i­nate as a new com­merce plat­form; and the first piece of the prize is your home.

An esti­mat­ed 23%-32% of US house­holds (and some 18% in the UK) now own a smart speak­er; Alexa pow­ers most of them, with Google Assis­tant in sec­ond place. While Google may have sold few­er devices in the home mar­ket, they still have greater reach because Assis­tant is also avail­able on most new Android phones (out­side China)—while Ama­zon are obvi­ous­ly delight­ed with 100 mil­lion Alexa-pow­ered devices sold, Google say that by the end of this month Assis­tant will be avail­able on one bil­lion devices (although it must be point­ed out that one bil­lion devices doesn’t equal one bil­lion active users).

In the home Ama­zon have more at stake, because unlike Google’s Android they don’t have a plat­form of their own, which Alexa pro­vides them with. Cre­at­ing this mar­ket is so impor­tant to Ama­zon that they have a report­ed 10,000 staff work­ing on Alexa.

Some­times mar­kets need to be made, not sim­ply seized.

Ben Thomp­son

Google’s mis­sion it to pre­vent Ama­zon from run­ning away with the mar­ket, because a con­se­quence of Alexa becom­ing the pri­ma­ry inter­face in your home is that it then extends Alexa’s use­ful­ness into your phone, and any­thing that pre­vents Google from being your first port of call for find­ing things out is an exis­ten­tial threat to that com­pa­ny. (This is a sim­i­lar threat to the launch of the iPhone and Apple’s con­trol over the nascent mobile web—to which Google’s response was Android.)

Just four years after Ama­zon announced the first Echo as a stand­alone home device, an increas­ing num­ber of house­hold objects have become con­trol­lable by, or enhanced by, smart speak­ers. Bret Kin­sel­la of Voice­bot describes this as the sec­ond phase of voice assis­tants:

Voice is quick­ly becom­ing an expec­ta­tion. For sev­er­al prod­uct cat­e­gories it is now a min­i­mum mar­ket require­ment

Bret Kin­sel­la

Using Blue­tooth and WiFi, smart speak­ers are able to direct­ly con­trol new devices in your home even if they’re not ‘smart’ devices. Google’s fresh­ly-announced Assis­tant Con­nect pro­gram is for exact­ly this pur­pose, while Ama­zon are a step ahead with their pre­view late last year of their Alexa-enhanced microwave oven and wall clock, which gain extra pow­ers when con­nect­ed to an Echo speak­er by Blue­tooth.

Not every device incor­po­rates a micro­phone and voice-pro­cess­ing func­tions: greater sup­port for pro­gram­ming inter­faces and the abil­i­ty to relay com­mands from speech-acti­vat­ed devices such as smart speak­ers brings voice con­trol to oth­er­wise “deaf” prod­ucts.

CCS Insights

The prob­lem with voice-only devices—especially smart speakers—is that, while they’re great for giv­ing sim­ple answers, for con­trol­ling media play­back, or for cer­tain heav­i­ly-pre­scribed func­tions such as set­ting alarms and timers, a lot of their capa­bil­i­ty is hid­den. A usabil­i­ty study by the Nielsen Nor­man Group found users were very unaware of what their smart assis­tants could do for them.

Con­sid­er­ing that 62% of the needs could be ful­ly or par­tial­ly solved by today’s intel­li­gent assis­tants, users employed their cur­rent assis­tants only in one of the 9 times when they could have used them with some suc­cess.

Ralu­ca Budiu

This con­tin­ues to be a prob­lem for Ama­zon and Google as they want to get more brands onto their plat­forms in order to make those plat­forms more valuable—but there’s still no way for users to find out about new brand­ed appli­ca­tions with­out some oth­er form of pro­mo­tion (both Alexa and Assis­tant have cat­a­logues of third-par­ty actions and reg­u­lar emails telling about new fea­tures). I talked about this in a lit­tle more detail in my pre­vi­ous arti­cle, Think­ing Out Loud: Under­stand­ing Voice UI, and How To Build for It.

One approach to solv­ing this dis­cov­er­abil­i­ty prob­lem is with intent-based dis­cov­ery; rather than the user hav­ing to know how to say “Hey Google, talk to Nike Coach”, they might ask “Hey Google, how can I start run­ning?”, to which Assis­tant might sug­gest Nike Coach as an option. This means third-par­ty assis­tant appli­ca­tions, espe­cial­ly from brands, should very clear­ly be designed to meet a user’s intent; and, fur­ther, to meet the very dif­fer­ent con­texts of (for exam­ple) using an assis­tant on a speak­er in your home, and using an assis­tant in your ear while out of home.

Anoth­er poten­tial solu­tion lies in the move to add screens to voice devices, whether through smart home hubs (Ama­zon Echo Show, Google Home Hub), the TV (Ama­zon FireTV Stick with Alexa, Google Home + Chrome­cast), phones, tablets, and lap­tops, or by turn­ing those phones and tablets into smart home hubs using dock­ing stands (Amazon’s Fire HD Show Mode Dock, Google’s Pix­el Stand). A smart assis­tant with a screen makes it eas­i­er to present more options to a user want­i­ng to find third-par­ty appli­ca­tions, and opens up the pos­si­bil­i­ty of sug­gest­ing more brand­ed appli­ca­tions ambi­ent­ly.

While Ama­zon and Google dom­i­nate the smart speak­er mar­ket, that’s not the end of the sto­ry. Apple’s Home­Pod is unlike­ly to take much mar­ket share (although will make them lots of mon­ey), but Siri is doing very well on phones, home media (Apple TV), and Air­Pods and Beats ear­phones. It was also notable to see many new devices at CES come with sup­port for Apple’s Air­play 2 pro­to­col.

Charts showing the % of people who have used voice assistants in the past month; Google Assistant wins on smartphones, with 62% of users, while Alexa wins in the home with 74%.
Source: Glob­al­We­bIndex

As I’ve said fre­quent­ly through­out 2018: Amazon’s end goal isn’t an Echo in every home, it’s Alexa in every thing. And the same goes for Google with Assis­tant, Apple with Siri, and every oth­er play­er in the mar­ket. The ulti­mate prize to be won by any com­pa­ny is to have their smart assis­tant be the meta-oper­at­ing-sys­tem across all the sur­faces of your inter­ac­tions with the internet—from your home to your car to the wear­able tech on your body. Ecosys­tem lock-in will be much stronger when an assis­tant is inte­grat­ed into every­thing you own. I cov­ered this in more detail in a pre­vi­ous arti­cle, Why Is Every Com­pa­ny Mak­ing a Dig­i­tal Assis­tant?.

As for oth­er play­ers, Samsung’s Bix­by hasn’t made much of an impact this year but a recent­ly-released Bix­by 2.0 is appar­ent­ly much-improved and Sam­sung have already declared their inten­tion to make their 500m annu­al home appli­ances Bix­by-com­pat­i­ble by 2020. Chi­nese mak­ers Aliba­ba, Baidu and Xiao­mi are impact­ing on the glob­al mar­ket, and could launch their smart speak­ers and assis­tants out­side of Asia soon.

Microsoft’s Cor­tana is… well, I’ve no idea what their strat­e­gy is as their part­ner­ship with Ama­zon puts Alexa on Xbox and Win­dows and so Cor­tana doesn’t seem to have much of a role any­more. Final­ly there’s Face­book, who launched their video-call­ing device / smart speak­er, Por­tal, late in 2018; I’ve no idea how sales have gone but they’re going to have an uphill strug­gle after their very bad year.

The Facebook Decline

Face­book was rarely out of the news in 2018 for a series of scan­dals, from pri­va­cy to fake news to bias to being an “enabling envi­ron­ment” for the geno­cide of Rohingya in Myan­mar. They have a fur­ther, per­haps exis­ten­tial, prob­lem as can be seen from two charts: the first shows how growth of time spent on Facebook’s feed is in a slight decline, as dig­i­tal video con­sump­tion increas­es.

Chart showing growth in average time Americans spend on media activities; television growth is negative, digital and mobile are in decline, Facebook is plateaued, digital video is growing
Source: Axios/eMarketer

The sec­ond shows how, while glob­al­ly Facebook’s growth remains strong, it’s less impres­sive when con­sid­er­ing users in Europe and North America—the most valu­able users in terms of mon­eti­sa­tion.

Source: businessinsider.com

Togeth­er these charts indi­cate that Facebook’s not acquir­ing new high-val­ue users—as with smart­phones, the ‘easy’ growth, which saw it bloom to over two bil­lion users in around ten years, is over—and the users they have are spend­ing less time engag­ing with the news feed (which is, more pre­cise­ly, where the decline is hap­pen­ing, not in the com­pa­ny as a whole). A decline in atten­tion to the news feed means a decline in adver­tis­ing rev­enue.

A lot of the pub­lic, per­ma­nent shar­ing on the news feed is mov­ing instead to pri­vate, ephemer­al shar­ing through mes­sag­ing and Sto­ries. The good news for Face­book is that they’re well-placed to ben­e­fit from that shift with their own Mes­sen­ger and their acqui­si­tions of Insta­gram and What­sApp.

This is the future. Peo­ple want to share in ways that don’t stick around per­ma­nent­ly, and I want to be sure that we ful­ly embrace this.

Mark Zucker­berg

The Sto­ries for­mat is the great break­out hit of the lat­ter days of the smart­phone age. From its begin­ning in Snapchat it was—ahem—adopt­ed by Insta­gram, What­sApp, and Mes­sen­ger and usage on those plat­forms now eclipses those of Snapchat. Sub­se­quent­ly we’ve seen the for­mat tak­en up (with vary­ing degrees of suc­cess) by a great num­ber of oth­er apps—most notably, Chi­nese behe­moth WeChat’s lat­est major update was entire­ly Sto­ries-focused (albeit called Time Cap­sules), and Google are get­ting behind the web-based AMP Sto­ries for­mat in search results pages.

Chart showing millions of daily active users of ‘Stories’ format on Facebook & Messenger (300), Instagram (400), and WhatsApp (450) versus Snapchat (188)

Mes­sag­ing in gen­er­al, after the bot gold rush a cou­ple of years ago, seems to be coa­lesc­ing around cus­tomer sup­port. There should be an oppor­tu­ni­ty there for brands to turn cus­tomer sup­port into fur­ther engage­ment, espe­cial­ly post-sales—starting from util­i­ty rather than cam­paign (which should always have been the approach).

A sec­ond wave of mes­sag­ing appli­ca­tions could come through WhatsApp’s Busi­ness API, Google’s local busi­ness mes­sag­ing, and the Google-backed Rich Com­mu­ni­ca­tion Ser­vice (RCS)—although the suc­cess of RCS relies on every mobile net­work car­ri­er get­ting on board, and Apple sup­port­ing the pro­to­col in iMes­sage, which cur­rent­ly seems unlike­ly.

But both Sto­ries and mes­sag­ing are hard­er to mon­e­tise than the news feed, and the ques­tion is whether these prop­er­ties can increase mon­eti­sa­tion with­out alien­at­ing users. Face­book are try­ing to diver­si­fy their rev­enue streams with tools for small busi­ness­es (espe­cial­ly fash­ion) to become more shop­pable on Insta­gram, while the What­sApp Busi­ness API aims to charge busi­ness­es for access to cus­tomers, as they try to reduce their depen­den­cy on adver­tis­ing.

The Advertising Shitshow

The ad tech ecosys­tem needs to be burned to the ground. Until that hap­pens every­thing that’s wrong with the inter­net will con­tin­ue to just get worse, because ad tech cre­ates the incen­tive to make it worse.

Aram Zuck­er-Scharff

Bil­lions of pounds are being spent on dig­i­tal adver­tis­ing, and bil­lions are being wast­ed. Ad com­pa­nies them­selves are doing very well, but pub­lish­ing busi­ness­es that rely on ads for rev­enue are strug­gling. One of Nie­man­Labs pre­dic­tions for jour­nal­ism in 2019 is “good­bye atten­tion econ­o­my”. I real­ly hope so.

The atten­tion econ­o­my is tox­ic. It’s respon­si­ble for garbage con­tent, fake news, and the exces­sive pow­er of the giant social-media plat­forms. Com­pet­ing for mon­ey forces media to think about how to give their users long-term val­ue instead of short-term grat­i­fi­ca­tion

Gideon Lich­field

The con­sumer back­lash isn’t against adver­tis­ing per se—just that ads are too pro­lif­ic, annoy­ing, irrel­e­vant, intru­sive, and cum­brous. Annoy­ing over­lays, auto­play­ing video, Bit­coin scams… lit­tle won­der that a sur­vey ear­ly last year found that over 40% of users said that they’d used an ad block­er in the past month.

Ad-frus­tra­tion, whether from annoy­ance with ads or a feel­ing that they’re exces­sive, is the most pop­u­lar moti­va­tion to block ads in all age groups.

Glob­al­We­bIndex report

Peo­ple hate dig­i­tal adver­tis­ing, and dig­i­tal adver­tis­ing hates peo­ple. I don’t know what’s going to hap­pen in this field in 2019, but it feels like its reach­ing cri­sis point.

The Web

I wor­ry about the long-term rel­e­vance of the web because brows­ing, espe­cial­ly on phones, involves run­ning a gaunt­let of pop­ups and over­lays and but­tons for grant­i­ng or refus­ing per­mis­sion to be tracked. It’s a com­bi­na­tion of the adver­tis­ing shit­show and some well-inten­tioned but mis­guid­ed GDPR and cook­ie reg­u­la­tions from the EU, and it’s a ter­ri­ble expe­ri­ence and poten­tial­ly coun­ter­pro­duc­tive to pre­serv­ing pri­va­cy.

Web­sites throw up pop-ups and over­lays that no one reads, or ban entire con­ti­nents, not because their users care but because a reg­u­la­tor said so.

Ben Thomp­son

Unlike many of my peers, I hope the AMP project is suc­cess­ful, because it impos­es strict rules about page load­ing speed and adver­tis­ing which essen­tial­ly ben­e­fit users. I ful­ly under­stand con­cerns about being a ‘land grab’ for the web from Google, but its new gov­er­nance mod­el, includ­ing inde­pen­dent advi­so­ry and lead­er­ship boards, hope­ful­ly indi­cate that this should no longer be a con­cern.

XR: the Post-mobile Platform

XR is a spec­trum of the inter­ac­tion between our sens­es and the dig­i­tal world. At one of the spec­trum is cold, hard, real­i­ty; at the oth­er, our sens­es are ful­ly cap­tured by a dig­i­tal layer—this is vir­tu­al real­i­ty (VR). In between we have aug­ment­ed real­i­ty (AR), where a dig­i­tal lay­er is over­laid on (our sense of) the phys­i­cal world but with­out any aware­ness of its space; and mixed real­i­ty (MR), where the dig­i­tal lay­er inter­acts with the phys­i­cal space of real­i­ty. XR is gen­er­al­ly under­stood as an acronym for eXtend­ed Real­i­ty, or some­times the X is under­stood to be a vari­able, like the 𝑥 in alge­bra; as a place­hold­er for ‘your real­i­ty here’. The XR spec­trum is some­times called the immer­sive spec­trum.

Source: Rob Scott

I still don’t believe that VR is going to break into the home con­sumer main­stream. It has a small but ded­i­cat­ed mar­ket in gam­ing, in busi­ness appli­ca­tions, and per­haps expe­ri­en­tial des­ti­na­tions (although IMAX clos­ing their VR cen­tres isn’t a good sign), but it’s just too iso­lat­ing to be used regularly—at least with cur­rent hard­ware devices.

The vir­tu­al real­i­ty mar­ket is fun­da­men­tal­ly con­strained by its very nature: because it is about the tem­po­rary exit from real life, not the addi­tion to it, there sim­ply isn’t near­ly as much room for vir­tu­al real­i­ty as there is for any num­ber of oth­er tech prod­ucts.

Ben Thomp­son

I think VR might stand a chance on notion­al future hard­ware which is capa­ble of dis­play­ing the full XR spec­trum, from light­ly-enhanced AR to full VR, where VR is sim­ply a mode that’s tog­gled on and off with­out requir­ing ded­i­cat­ed spe­cial­ist hard­ware.

I believe that XR is going to be the post-mobile plat­form (or, at least, a post-mobile plat­form), but there’s no exist­ing hard­ware solu­tion that’s suf­fi­cient to take us there yet. Smart­phone-based AR is great for aug­ment­ed self­ies, as enhanced ‘mir­rors’ (espe­cial­ly for cos­met­ics brands), for AR stick­ers in mes­sages, and per­haps for shar­ing (pass­ing the phone to a friend) to, for exam­ple, show a friend an object such as fur­ni­ture; but it’s not com­fort­able or con­ve­nient enough for extend­ed use, which elim­i­nates a lot of poten­tial use cas­es.

XR will real­ly hit its stride when some­one makes a break­through head-mount­ed device—probably glass­es. I men­tioned ear­li­er some of the issues that still need to be worked out, includ­ing sen­sors, bat­tery life, and appear­ance. But many of these are already being worked out through phones and wear­ables, and I don’t see them as insur­mount­able—a new head­set by start­up Nre­al looks to have made some promis­ing steps for­ward this year already, espe­cial­ly in appear­ance.

The biggest block­ing prob­lem right now is the lens. Some ear­ly AR head­sets, such as those from North and Vusix, use a heads-up-dis­play; infor­ma­tion pro­ject­ed onto the lens, in a sim­i­lar way that Google Glass did a few years ago. What you see is a flat image over­laid on your vision, like a smart watch screen being dan­gled in front of your eyes. It adds infor­ma­tion to your vision, but there’s no sense of it inter­act­ing with the phys­i­cal world. This inter­ac­tion, or mixed real­i­ty, requires the illu­sion of dimen­sion­al­i­ty which is pro­vid­ed by sen­sors build­ing a pic­ture of the world around you and feed­ing it to a visu­al dis­play. There are cur­rent­ly two approach­es to this dis­play.

With video see-through (VST) lens­es, a cam­era records what’s in front of you and pass­es that to LED screens, so your eyes are see­ing a cap­tured vision of the world, then adds dig­i­tal infor­ma­tion to the sig­nal; you’re nev­er see­ing the world direct­ly, only a medi­at­ed image. VST lens­es have issues of social aware­ness, as they block your eyes from peo­ple look­ing at you, and of focus as every­thing is on a flat plane a few cen­time­tres from your pupils, which will tire your eyes.

With opti­cal see-through (OST) lens­es, light con­tain­ing dig­i­tal infor­ma­tion is pro­ject­ed onto dark­ened glass lens while you look through it, so dig­i­tal objects appear to ren­der direct­ly in the scene. The major draw­back with OST is that it can only add light to a scene and not sub­tract it, mean­ing they don’t work well in bright light. The mixed real­i­ty head­sets HoloLens and Mag­ic Leap cur­rent­ly use OST, but it will be inter­est­ing to see if that’s where the mar­ket even­tu­al­ly lands.

Illustration of the differences between optical see-through and video see-through augmented reality lenses.
Source: Prototypr.io

Regard­less of the tech­nol­o­gy approach, I believe that XR offers new oppor­tu­ni­ties in visu­al user inter­faces; it could free com­put­ing from the con­fines of a 15cm rec­tan­gu­lar screen, and could be the dom­i­nant plat­form of the next ten years—if the prob­lems can be over­come. I don’t fore­see the break­through device com­ing this year, but I hope to be proved wrong.

One inter­est­ing cul­tur­al trend to watch will be the devel­op­ment of dig­i­tal iden­ti­ty in XR. Apple’s ani­mat­ed Emo­ji, Snapchat’s Bit­mo­ji, Samsung’s AR Emo­ji, even (to a degree) Google’s Gboard Min­is, are all devel­op­ing our online visu­al iden­ti­ties from sta­t­ic avatars to ani­mat­ed char­ac­ters based on our phys­i­cal attrib­ut­es. Machine learn­ing can cre­ate a mod­el of our speech pat­terns, as in Gmail’s Smart Com­pose and Smart Replies. Extrap­o­lat­ing from this, we could have dig­i­tal selves that cre­ate entire­ly vir­tu­al iden­ti­ties to aug­ment our actu­al selves. I don’t know where this is going yet, but it’s inter­est­ing.

2019 vs 2018

Look­ing back at last year’s equiv­a­lent trends piece, I’m struck by the sim­i­lar­i­ties to this year’s. For all that it some­times feels like we’re mov­ing at a whirl­wind pace, tech­no­log­i­cal progress is most­ly quite sta­ble, with true break­throughs like the smart­phone hap­pen­ing more rarely than we might think. Rather than con­stant rev­o­lu­tions, we have more of a punc­tu­at­ed equi­lib­ri­um mod­el; occa­sion­al bursts of rapid trans­for­ma­tion fol­lowed by peri­ods of rel­a­tive sta­bil­i­ty.


Fea­tured image: Soft­wear by Kiki van Eijk.


Also pub­lished on Medi­um.