This week we talk about the HoloLens, the Apple Vision Pro, and the Meta Ray-Ban Smart Glasses.

We also discuss augmented reality, virtual reality, and Orion.

Recommended Book: The Mountain in the Sea by Ray Nayler

Transcript

Originally released as a development device in 2016—so aimed at folks who make software, primarily, not at the general public—the HoloLens, made by Microsoft, was a fairly innovative device that looked like virtual reality headgear, but which allowed folks to interact with graphical elements overlayed on a transparent surface so that they seemed to be positioned within the real world; so-called augmented reality.

This functionality relied upon some of the tech Microsoft had developed for its earlier Kinect accessory, which allowed Xbox owners to play games using their bodies instead of more conventional controllers—it used a camera to figure out where people, and their arms, legs, and so on, were in space, and that helped this new team figure out how to map a person’s living room, for instance, in order to place graphical elements throughout that room when viewed through the HoloLens’ lenses; so stuff could appear behind your couch, pop out of a wall, or seem to be perched atop a table.

The HoloLens was not the only option in this space, as several other companies, including other tech titans, but also startups like Magic Leap, were making similar devices, but it was arguably the most successful in the sense that it both developed this augmented reality technology fairly rapidly, and in the sense that it was able to negotiate collaborations and business relationships with entities like NASA, the US Military, and Autodesk—in some cases ensuring their hardware and software would play well with the hardware and software most commonly used in offices around the world, and in some cases showcasing the device’s capabilities for potential scientific, defense, and next-step exploratory purposes.

Like many new devices, Microsoft positioned the HoloLens, early on, as a potential hub for entertainment, launching it with a bunch of games and movie-like experiences that took advantage of its ability to adapt those entertainments to the spaces in which the end-user would consumer them: having enemies pop out of a wall in the user’s kitchen, for instance, or projecting a movie screen on their ceiling.

It was also pitched as a training tool, though, giving would-be astronauts the ability to practice working with tools in space, or helping doctors-in-training go through digital surgeries with realistic-looking patients before they ever got their hands dirty in real life. And the company leaned into that market with the second edition of the headset, which was announced and made available for pre-order in early-2019, optimizing it even further for enterprise purposes with a slew of upgrades, and pricing it accordingly, at $3,500.

Among those upgrades was better overall hardware with higher-end specs, but it also did away with controllers and instead reoriented entirely toward eye- and hand-tracking options, combined with voice controls, allowing the user to speak their commands and use hand-gestures to interact with the digital things projected over the real-world spaces they inhabited.

The original model also had basic hand-tracking functionality, but the new model expanded those capabilities substantially, while also expanding upon the first edition’s fairly meager 30 degrees of augmented view: a relatively small portion of the user’s line of sight could be filled with graphics, in other words, and the new version upgraded that to 52 degrees; so still not wall to wall interact-with-able graphics, but a significant upgrade.

Unfortunately for fans of the HoloLens, Microsoft recently confirmed that they have ended production of their second generation device, and that while they will continue to issue security updates and support for their existing customers, like the US Department of Defense, they haven’t announced a replacement for it—which could mean they’re getting out of this space entirely.

Which is interesting in the sense that this is a space, the world of augmented reality, which some newer entrants are rebranding as mixed reality, that seems to be blowing up right now: two of Microsoft’s main competitors are throwing a lot of money and credibility into their own offerings, and pitching this type of hardware as the next-step in personal devices.

Some analysts have posited, though, that Microsoft maybe just got into this now-burgeoning arena just a little too early, investing in some truly compelling innovations, but doing so at a moment in which the cost was too high to justify the eventual output, and now they might be ceding the space to their competition rather than doubling-down on something they don’t think will pay off for them, or they may be approaching it from another angle entirely, going back to the drawing board and focusing on new innovations that will bypass the HoloLens brand entirely.

What I’d like to talk about today are the offerings we’re seeing from those other brands, and what seems to be happening, and may happen in the near-future, in this augmented-reality, mixed-reality segment of the tech world.

—

I did an episode on spacial computing and the Apple Vision Pro back when the device was made available for purchase in the US, in February of 2024.

This device was considered to be a pretty big deal because of who was making it, Apple, which has a fairly solid record of making new devices with unfamiliar interfaces popular and even common, and because the approach they were taking: basically throwing a lot of money at this thing, and charging accordingly, around $3,500, which is the same price the second HoloLens was being sold for, as I noted in the intro.

But because of that high price point, they were able to load this thing up with all sorts of bells and whistles, some of which were fundamental to its functionality—like super-high-density lenses that helped prevent nausea and other sorts of discord in their users—and some that were maybe just interesting experiments, like projecting a live video of the user’s eyes, which are concealed by the headset, on the front of the headset, which to me is a somewhat spooky and silly effect, but which is nonetheless technically impressive, and is something that seems aimed at making these things less anti-social, because you can wear the Vision Pro and still see people, and this projection of their eyes allows them to see you and your facial expression at the same time.

I’ve actually had the chance to use this device since that episode went live, and while there are a lot of weird little limitations and hindrances to this device going mainstream at the moment, the technology works surprisingly well right out of the box, with the eye- and hand-tracking elements working shockingly, almost magically well for relatively early-edition tech; Apple is pretty good at making novel user-interfaces intuitive, and that component of this device, at least, seemed like a slam dunk to me—for casual use-cases, at least.

That said, the company has been criticized for that high price point and their seeming fixation on things like putting the users’ eyes on the outside of the headset, rather than, for instance, investing in more content and figuring out how to make the thing more comfortable for long periods of time—a common complaint with basically every virtual reality or mixed-reality headset ever developed, because of the sheer amount of hardware that has to be crammed into a finite, head-and-face-mounted space, that space also needing to be properly balanced, and it can’t get too hot, for perhaps obvious reasons.

Those criticisms related to price are the result not of comparison to HoloLens, as again, the pricing is basically the same between these two devices, but instead the result of what Meta has done with their mixed-reality offerings, which are based on products and technology they acquired when they bought Oculus Labs; they’ve leaned into providing virtual reality devices for the low- and mid-market consumer, and their newest model, the Meta Quest 3S is a stand-alone device that costs between about $300 and $400, and it has mixed-reality functionality, similar to the Vision Pro and HoloLens.

While Meta’s Quest line doesn’t have anywhere near the specs and polish of the Vision Pro, then, and while it didn’t arrive as early as the HoloLens, only hitting shelves quite recently, it does provide enough functionality and serves enough peoples’ purposes, and at a far lower price point, that it, along with its other Quest-line kin, has managed to gobble up a lot of market share, especially in the consumer mixed-reality arena, because far more people are willing to take a bet on a newer technology with questionable utility that costs $300 compared to one that costs them more than ten-times as much.

Interestingly, though, while Meta’s Reality Labs sub-brand seems to be doing decently well with their Quest line of headsets, a product that they made in collaboration with glasses and sunglasses company EssilorLuxottica, which owns a huge chunk of the total glasses and sunglasses global market, via their many sub-brands, may end up being the more popular and widely used device, at least for the foreseeable future.

The Ray-Ban Meta Smartglasses looks almost exactly like traditional, Ray-Ban sunglasses, but with slightly bulkier arms and with camera lenses built into the frames near where the arms connect to them.

If you’re not looking carefully, then, these things can be easily mistaken for just normal old Ray-Bans, but they are smartglasses in that they contain those two cameras on the front, alongside open-air speakers, a microphone, and a touchpad, all of which allow the wearer to interact with and use them in various ways, including listening to music and talking on the phone, but also taking photos of what they’re looking at, recording video of the same, and asking an AI chatbot questions like, what type of flower is this, and getting an audible answer.

These things cost around what you would pay for a Quest headset: something like $300-400, but their functionality is very different: they don’t project graphics to overlay the user’s view, in that regard they function like normal sunglasses or prescription glasses, but if you want to snap a photo, livestream whatever it is you’re seeing, or ask a question, you can do that using a combination of vocal commands and interacting with the built-in touchpad.

And while this isn’t the mixed-reality that many of us might think of when we hear that term, it’s still the same general concept, as it allows the user to engage with technology in real-life, in the real-world, overlaying the real world with digital, easily accessed, internet-derived information and other utilities. And it manages to do so without looking super obtrusive, like earlier versions of the same concept—Google’s Google Glass smartglasses come to mind, which were earlier versions of basically the same idea, but with some limited graphical overlay options, and in a form factor that made the wearer look like an awkward, somewhat creepy cyborg.

Snap, the parent company of Snapchat, has a similar offering which originally leaned into the same “these look just like glasses, but have little camera lenses in them” strategy, though with their newest iteration, their Spectacles smartglasses product has reoriented toward a look that’s more akin to a larger, clunkier version of the free 3d glasses you might use at the movie theater—not exactly inconspicuous, though offering much of the same functionality as Meta’s Raybans, alongside some basic graphical overlay functions: a lightweight version of what the Vision Pro and Quest offer, basically, and in a much small package.

These new Spectacles are only available for folks who sign up for the company’s developer program at the moment, however, and are purchased not as a one-off, but for $99/month, with a minimum commitment of 12 months—so the price tag is quite a bit higher than those Quests and Raybans, as well.

Interestingly, Meta’s Reality Labs recently held an event in which they showed off an arguably more advanced version of Snap’s Spectacles, called Orion.

These things are being pitched as the be-all, end-all mixed-reality solution that every company is trying to develop, but which they can’t develop yet, at least not at scale. They look like giant, cartoony glasses—they’re shaped like glasses, but comically oversized ones—and they provide many of the same benefits as today’s Quest headset, but without the large, heavy headset component; so these could theoretically be used in the real-world, not just in one’s living room or office.

The company announced this product along with the caveat that they cannot make it on scale, yet, because cramming that much functionality into such a small device is really stressing the capacity of current manufacturing technologies, and while they can build one of these glasses, with its accompanying wristband and a little controller, both of which help the glasses do what they do, in terms of compute and the user interface, for about $10,000 per unit, they could not, today, build enough of them to make it a real, sellable product, much less do so at a profit.

So this was a look at what they hope to be doing within the next decade, and basically gives them credibility as the company that’s already building what’s next—now it’s just a matter of bringing down costs, scaling up production, and making all the components smaller and more energy efficient; which is a lot of work that will take years, but is also something they should theoretically at least be able to do.

To be clear, most other big tech companies should be capable of build really snazzy, futuristic one-offs like the Orion, as well, especially if they, like Meta, offload some of the device’s functionality into accessory hardware—the Vision Pro has offloaded its battery into a somewhat clunky, pocketable appendage, for instance, and most of these devices make use of some kind of external controller, to make the user interface snappier and more accurate.

But Meta is attempting to show that this is the direction they see wearable technology going, and maybe our engagement with the digital world more holistically, as well. It’s easy to imagine a world in which we all have these sorts of capabilities built into our glasses and wristbands and other wearables, rather than having to work with flat, not-mixed-reality screens all the time, especially once you see the tech in action, even if only as a not-for-sale example.

One aspect of this potential future that Meta is forecasting is already leading to some soul-searching, though.

Some students at Harvard modified a pair of Meta Ray-Bans to use facial recognition and reverse-image search technology so they could basically look at a stranger, then learn a bunch of stuff about them really quickly, to the point that these students were able to do this, then pretend to know the that stranger, talk about their work, find their spouse’s phone number—a bunch of details that made it seem like they knew this person they’d only just met.

All of which is pretty wild and interesting, but also potentially frightening, considering that this is basically doxing someone on demand, in public, and it could be used—like many other tech innovations, granted—to enable and augment stalking or kidnapping or other such crimes.

None of which is destiny, of course. Nor is the success of this product type.

But there does seem to be a lot of interest in what these gadgets seem like they might offer, especially as the prices drop, and as more entrants carve out space in that relatively lower-cost space—which is a space Apple is reportedly planning to enter soon, too, with a new edition of their Vision Pro that would cost maybe something like half as much as the first one, and possibly smart glasses and maybe even Airpods with cameras meant for release over the next couple of years.

So it may be that the early divulgence of these next-step devices, showing us where these things might go with these higher-priced, smaller audience initial editions, could allow us to predict and prepare for some of their negative externalities before they go completely mainstream, so that when they finally arrive in their finished form, we’re a bit more prepared to enjoy the benefits while suffering fewer (though almost certainly not zero) of their potential downsides.

Show Notes

https://en.wikipedia.org/wiki/Spatial_computing

https://en.wikipedia.org/wiki/Apple_Vision_Pro

https://en.wikipedia.org/wiki/Meta_Quest_3S

https://en.wikipedia.org/wiki/Meta_Platforms

https://www.reddit.com/r/RayBanStories/comments/1e3frhc/my_honest_review_of_the_rayban_metas_as_everyday/

https://en.wikipedia.org/wiki/Ray-Ban_Meta

https://www.spectacles.com/spectacles-24?lang=en-US

https://en.wikipedia.org/wiki/Spectacles_(product)

https://forums.macrumors.com/threads/students-add-facial-recognition-to-meta-smart-glasses-to-identify-strangers-in-real-time.2438942/

https://archive.ph/6TqgF

https://www.theverge.com/24253908/meta-orion-ar-glasses-demo-mark-zuckerberg-interview

https://about.fb.com/news/2024/09/introducing-orion-our-first-true-augmented-reality-glasses/

https://www.reddit.com/r/augmentedreality/comments/1frdjt2/meta_orion_ar_glasses_the_first_deep_dive_into/

https://appleinsider.com/articles/24/10/13/cheaper-apple-vision-headset-rumored-to-cost-2000-arriving-in-2026

https://www.uploadvr.com/microsoft-discontinuing-hololens-2/

https://www.theverge.com/2024/10/1/24259369/microsoft-hololens-2-discontinuation-support

https://www.theverge.com/2022/6/7/23159049/microsoft-hololens-boss-alex-kipman-leaves-resigns-misconduct-allegations

https://en.wikipedia.org/wiki/Microsoft_HoloLens

This is a public episode. If you’d like to discuss this with other subscribers or get access to bonus episodes, visit letsknowthings.substack.com/subscribe

Get the future of podcasts.