Category: Augmented Reality

This includes all posts that are about anything to do with AR.

Categories
Augmented Reality Policy, Legal, Regulatory

Spimes and the Law

Until a technology (or suite of technologies) reaches a critical mass market "mind share" (defined as there's high awareness among those outside the field but not necessarily mass market adoption), there's little attention given to legal matters. Well, the big exception I'm aware of in the domains I monitor is the attention that mobile network operators give to what goes over their infrastructure. Their sensitivity is due to the fact that operators are, under national regulatory policies, responsible for how their services are used and abused.

Although spimes are too diverse to be recognized as a trend by the mass market, Augmented Reality is definitely approaching the public consciousness in some countries. One of those is the United States. AR is on several 2012 top technology trend lists (post on Mashable, BBC ran a four minute video piece on AR in New York City during a prime time TV news magazine, hat tip to Brain Wassom for this one).

With over 50% of the US cellular users on smartphones (according to Flurry report quoted on Tech Crunch on December 27, 2011), and a very litigious society, Americans and American firms are likely to be among the first group to test and put in place laws about what can and can't be done when combining digital data and the physical world. In June 2011, during the third AR Standards Community meeting, an attorney practicing geospatial law in the US, Kevin Pomfret, spoke about the potential legal and regulatory issues associated with AR.  He identified the potential liability associated with accidents or errors during AR use as one of the issues. Pornography and content for "mature audiences" was another area that he highlighted.

This week, Brian Wassom blogged about legal topics that he believes will be important in 2012. Below are my responses and thoughts about the points made in Wassom's post.

1. The first Licensing Model for AR Content. I'm a stickler when it comes to vocabulary. Although often used, the term "AR content" is a misnomer. What people are referring to is content that is consumed in context and synchronized with the real world in some way.  In this light, there are really two parts to the "AR content" equation: the reality to which an augmentation is anchored, and the augmentation itself.

Wassom's point is that, in 2012, a business model around premium AR content will be proven. I hope this will happen but I don't see a business model as a legal matter. There is a legal issue, but it is not unique to AR. There certainly are greedy people who cross the line when it comes to licensing rights. Those who own the information that becomes the augmentation have a right to control its use (and if it is used commercially with their approval, to share in the proceeds). There's never been any doubt that content creators, owners, curators and aggregators all have a role to play and should be compensated for their contributions to the success of AR. However, when information is in digital format and broken down into the smallest possible unit, what is the appropriate  licensing model for an individual data field or point? How those who provide AR experiences will manage to attribute and to license each individual augmentation is unclear to me. Perhaps the key is to treat augmentations from third parties the way we treat digital images on the web. If the use is non-commercial, it can be attributed, but there is no revenue to share. When the use is commercial, in "premium AR experiences," the entity charging a premium fee must have permission from the owner of the original data and distribute the revenues equitably.

2. The First Negligence Law Suit. Wassom believes that people whose awareness of the physical world is impaired by augmentations so completely that they are injured would feel that they can put the blame on the provider of the AR experience. He's got a point, but this should be nipped in the bud. I think those who provide platforms for AR (applications or content), should begin their experiences with a terms and conditions/disclaimer type of agreement. It may take one negligence or liability suit to drive the point home, but over time we will all be required to agree to a clause which releases the provider from responsibility for the actions of users. Why don't people publishing AR experiences preempt the whole problem and introduce the clause today?

3. First Patent Fight will be over Eyewear. I'm following the evolution of hands-free AR hardware and agree that it is not going to be much longer before these are commercially available. Nevertheless, it will require that more than 100,000 of these to be sold before it will be worth anyone's time and money to go after the provider of the first generation of eyewear for consumer use. In my opinion, it will take several generations more (we are already in the second or third generation) before the technology is sufficiently mature to be financially viable. So by that time, a completely different cast of characters will be involved. Why would anyone with patents in this area want to stifle the innovation in hands-free AR in 2012 with a patent fight?

4. Trademarks. I'm looking forward to the day when the term "Augmented Reality" completely disappears. If there begins to be trademark claims around the term in 2012, so much the better! Then people will begin to drop the term to refer to what will be commonplace anyway and we can accelerate the time necessary to just accept that this will be a convenient and compelling way to live.

5. AR frowned upon by family values and fundamentalist religious groups due to explicit content. It is clear that adult content drives the growth of many technologies and AR is unlikely to be different so I agree with Wassom that the adoption of AR in this market is inevitable. It is likely to provide a very clear business case for premium content so, from a financial point of view, this is all good. Having pedophiles using AR would not be surprising and certainly not desirable, but I don't think the technology making AR possible will be treated any differently from other technologies. It's not the technology to blame. It's human nature.      

Categories
Augmented Reality Business Strategy

The Top Mobile AR Regions in 2012

Due to a variety of factors, including the fact that the two biggest manufacturers are not publishing these figures, no one knows precisely how many smartphones shipped in 2011. Smartphone shipments are just part of the data which would be useful to guide where mobile AR efforts could have the highest impact in 2012.

Let's assume that only iOS and Android users could have mobile AR experiences. Those who have smartphones need to know how to use them and see some benefits as regular users. Running apps at least once a month is a strong qualification. Take a look at the figure released by Flurry on December 23, 2011 (below) at the current total addressable market (TAM) for iOS and Android apps.

The largest chunk of the world’s active installed base is in the US, with 109 million out of 264 million, or 41%. But, the US is not the geography in which we have the greatest number of AR experiences available (experiences are more heavily concentrated in Western Europe). And, considering the conditions put on providers of geospatial information and the nascent state of that market, smartphone users in the next largest market, China, are unlikely to have a proliferation of AR experiences in 2012.

Now, look at the other Top 10 addressable markets in the second figure (below).

In addition to boosting the opportunities to engage users in the US, developers of mobile AR should be seriously expanding their AR publishing reach into Japan, Germany, France, UK and Italy.

Categories
Augmented Reality

Three Predictions for mobile AR in 2012

In 2011, mobile Augmented Reality continued to mature and major new players entered (e.g., Autonomy with Aurasma, Nokia with Live View). Several important deals with handset manufacturers (e.g., Wikitude with RIM and Huawei, metaio with Texas Instruments and other chip houses) indicate that more users will have the AR capability pre-loaded in their 2012 devices, but it was also a tough year for the segment's "legacy players," if we can use this label.

For example, some of the leading developers in 2009 (e.g., Acrossair, Hoppala) morphed and grew very quiet. On the technology provider side, there have been a few important developments (that failed to make headlines) in the last months of the year: Layar quietly shared to its developer ecosystem that it was changing direction and there were significant lay-offs in late November; in October, Total Immersion released its latest D'Fusion Studio at no cost for non-commercial use (presumably if the platform was selling well and lots of new developers were signing up, they would not have needed to release the no cost program). Two of the other leading AR developer tool providers are comparatively quiet as well: Wikitude's new platform ARchitect Developer Kit is (as of mid-December) still in Beta; AR Toolworks has not announced any new features to its commercial product line in ten months.

What will 2012 bring? My predictions for the year ahead include:

1. metaio undergoes a dramatic change in company management or structure through either merger with another technology provider or acquisition.

2. Increased fragmentation of market (more difficulty achieving a critical mass of loyal users on one system or platform) due to several significant new platforms for AR being introduced in 2012 and few companies publishing open APIs to bridge between the technology silos.

3. Social AR capabilities (self publishing features or connections to social networks) advance significantly compared with the state of the art on the last day of 2011.

Let's see in the coming weeks what others predict and, at the end of 2012, compare these with the developments of the year.

December 13, 2011 update: Remco Vroom and Johannes La Poutre over at TAB World Media seem to like the idea that 2012 will be the year of point-to-know. More about this as a consumer trend on TrendWatching for 2012.

December 30, 2011 update: I watched a 1-minute animated slideshow containing no details on the predictions of Kiran Voleti

January 3, 2012 update: Brian Wassom has posted his five predictions for AR and the law in 2012. I've written a new post about my reactions to his predictions.

January 9, 2012 update: Catchfire Media Predicts that AR will go mainstream in 2012!

I'll append the predictions of other industry watchers to this post when I learn of them.

Categories
Augmented Reality Social and Societal Standards

Virtual Public Art Project

Some believe that experiencing art in digital forms while interacting with or set in real world settings will be a widely adopted use case for Augmented Reality. People will be able to experience more examples of artistic expression, in different places and to contribute by expressing themselves through their software and mobile devices. Projects to explore the interaction of digital and physical objects are quite popular at the annual SIGGRAPH event.

One of the earliest projects using the Layar AR browser for artistic expression in public (and private) spaces is the Virtual Public Art Project begun in March 2010 by Christopher Manzione, a New York City artist and sculptor. Manzione created the VPAP by publishing his creations in a dedicated layer. The site says:

VPAP is the first mobile AR outdoor art experience ever, and maximizes public reception of AR art through compatibility with both iPhone 3GS and Android phones using the free Layar application.

Artists around the world have invested in putting their digital work into the VPAP layer. Projects like this one certainly have the potential to dramatically change how people interact with one another and with art, especially if they are also able to leave their comments or opinions about the artist's work.

One of the limitations of the current VPAP, and perhaps a reason it has not received attention since the fall/winter of 2010-2011, is that it is only viewable on one browser. If there were standards for AR formatting, as there are today for formatting content viewed in a Web browser, then any viewer application, capable of detecting the user's context such as the AR browsers from wikitude and metaio (junaio) would also provide access to the artists' work. In an ideal world one source of content could offer all users the same or similar experiences, using their software of choice.

In the face of multiple proprietary technology silos (and client applications with projects requiring wide, browser-neutral audiences), some AR experience developers offer services based on a single back end with interfaces to each of the publishing platforms. Examples include the Hoppala Augmentation by Hoppala Agency, BuildAR by MOB Labs and MARways by mCRUMBS. In each case, these platforms streamline the publishing process for the content creator to have the widest possible reach.

Will they also need to write interfaces to the next AR browsers? What will these platforms be capable of when Web browsers also support AR?

I guess these are not questions on which artists should be spending their time.

Categories
Augmented Reality Events Research & Development Social and Societal

Algorithmic City and Pattern Language

The Mobile City Web portal and the blog of Martijn de Waal are inspirational to me. He introduces many concepts that, although they do not use precisely the same words/vocabulary, they mirror what I've been thinking and seeing. One of the posts on The Mobile City from March 2011 is a review by guest contributor Michiel de Lange of a compendium of articles about technology in cities edited by Alessandro Aurigi and Fiorella De Cindio.

The Augmented Urban Spaces (2008) would be the basis for a great conference on "smarter cities" and Internet of Things.

Another post that I found stiulting is Martijn's report of the Cognitive Cities Salon he attended in Amsterdam, Martijn highlighted a talk given by Edwin Gardner entitled "The Algorithmic City" which I am sorry to have missed and, unfortunately, I have not found the slides on-line (yet). From what Martijn writes, the subject of Algorithmic Cities is currently theoretical but one can imagine a day when it will become common place.

The Algorithmic City is the result of urban planners using algorithms as part of their process(es). Quoting from the Mobile City blog post published on July 3 2011:

"So far algorhithms have shown up in ‘parametric design’ where all kinds of parameters can be tweeked that the computer will then turn into a design for a building or even a complete city. Gardner is not so much interested in this approach. The problem is that there is no relation between the paramaters, the shapes generated and the society that is going to make use of these shapes. Social or economic data are hardly used as parameters and the result is ‘a fetishism of easthetics’, at best beautiful to look at, but completely meaningless.

Instead, Gardner takes inspiration from Christophers Alexander‘s book A Pattern Lanugage."

I'm not sure if there is a connection between the theoretical work of Gardner and a video of the next version of a software solution provided by Procedural (a startup based in Zurich purchased by ESRI on July 11, 2011), called CityEngine but somehow the two come together in my mind. Using CityEngine, design algorithms are manipulated using the simplest of gestures, such as dragging. It's not a software platform I'm likely to have need for in the near future, but I hope that urban planners will soon have opportunities to experiment with it, to explore the Algorithmic City concept, and that I will witness the results. Maybe someone will build an AR experience to "see" the Algorithmic City using smartphones.

Categories
Augmented Reality Research & Development

ElipseAR-Cloud Image Recognition

There's long been a debate among computer vision experts between those who envisage feature extraction and matching in the network and those who implement it on the device. There are many factors one must consider and trade offs that must be made but in the end everything boils down to cost: what can you gain by where you put the different tasks if the tasks must be done in real time. For many applications, feature extraction is a bottle neck due to lack of computational power. Qualcomm's AR SDK is an example of device-based recognition.

ElipseAR, a startup based in Madrid, Spain, is heavily on the network side of the debate. They are planning to release a set of tools for markerless (feature-based) Augmented Reality and Computer Vision development that will make image matching and tracking, 3D animation rendering, geolocation using the camera view, face recognition easier to integrate into AR applications. Existing AR apps? Future AR apps?

The company's web site clearly makes a distinction between image recognition, image tracking and matching. What's not clear at the moment, because their position differs depending on which page you are reading, is how much of the ElipseAR processing is to happen within the device and how much will be in the network. They also may be confusing "image" with real time video recognition.

At the moment the company says it will offer its tools for commercial use at no charge. The beta program started in early July and is expected to run until the end of 2011.

Tests must be conducted in real world circumstances to measure the merits of the new algorithm and its architecture. It will be compared against not only other network-based image recognizers such as kooaba, but also other SDKs that have been out for much longer such as Qualcomm AR SDK, Qconcept and others. It's difficult to imagine, for example, ElipseAR getting out ahead of String Labs which released their code June 16, 2011.

Even if the reliability of the ElipseAR algorithms and architecture prove to be up to industry benchmarks, there will continue to be latency out of the control of the developer or user in the cellular networks. There have been rumors that the network "effect" can be overcome, but this will never be a universally reliable solution because coverage of mobile networks is and never will be 100%.

Categories
Augmented Reality Research & Development

Reduced Reality

In the physical world, visually noisy environments are common. Some cultures enjoy or at least live in a stimulating visual landscape, be it on their screens or in/on the real world. I recall there being more visual noise in Asian urban landscapes than I am accustomed to. I prefer the work of designers that hide or disguise the clutter in every day life. Take, for example, power and telephone lines. For a variety of reasons these are above ground in some parts of the world and below in others.

I prefer a visually "simple" world. Blocks of uniform or lightly textured surfaces: the sky, the water of lake Geneva, even skyscrapers.

Why would the same algorithms and systems used to attach additional information to the real world not also be useful to reduce information? Power and telephone lines could "disappear" from view, as would graffiti and trash.

There was a poster at ISMAR2010 that demonstrated the "reduction" of reality using a mobile device to cover/camouflage a QR code in a photograph. By sampling from the background in the immediate proximity and tiling the same pixel colors and textures over the marker, there was a sense of continuity, the marker disappeared. Unfortunately, the specific project and references made to it are difficult to find but I hope to see more of this in the next ISMAR event in Basel.

Categories
3D Information Augmented Reality Research & Development

AR for Blacksburg

The AR-4-Basel project is a pilot for what could become a widespread trend: a municipality or any size area can make data sets it owns and maintains for its citizens available to AR developers who then can prepare AR experiences for visitors and inhabitants.

Ever since starting the AR-4-Basel project in May, I have been planning how to expand and apply the lessons learned to other cities. The first to follow is definitely Barcelona, Spain. The AR-4-Barcelona project is already ramping up. Then, Berlin is my next target. I’d like to explore the possibility of getting something started in Beijing as well, if there is going to be an AR in China conference in 2012.

Another “B” city which has all the earmarks of a future haven for AR experiences is Blacksburg, Virginia!

“The 3D Blacksburg Collaborative is a consortium of multi-disciplinary researchers, experts and students from various universities and governments, who are creating a data and delivery infrastructure for an interactive virtual 3D city model.”

Which “B” city would you nominate for a future AR project?

Categories
Augmented Reality News Research & Development

Pittsburgh Pattern Recognition

On July 22 2011, Google acquired PittPatt, the Pittsburgh Pattern Recognition Team, a privately-held spin out of CMU Robotics.

Three questions jumped out when I learned of this acquisition.

  • Why? Doesn't Google already have face recognition technology?
    Unfortunately, based on the publicly available information, it's not clear what is new or different about PittPatt's technology. Okay, so they have an SDK. There are several possible explanations for this acquisition. Maybe the previous facial recognition technology Google had acquired with Neven Vision in August 2006 then released as part of Picasa in 3rd quarter 2008 (it appeared in Picasa as early as May 2007) was insufficient. Insufficient could mean inaccurate too often, too difficult to implement in mobile, not scalable. That doesn't seem likely.
    Maybe the difference is that the PittPatt technology was working on video as well as still images. YouTube already has a face recognition algorithm, but it is not real time. For AR it would be valuable if the face recognition and tracking performs reliably in real time.
    Another possible explanation has to do with IP. Given the people who founded PittPatt, perhaps there are some intellectual properties that Google wants for itself or to which it wants to prevent a competitor to have access.
     
  • What are the hot "nearby" properties that will get a boost in their valuation as a result of Google's purchase?
    Faces are the most important attribute we have as individuals and the human brain is hard wired to search for and identify faces. Simulating what our brains do with and for faces is a fundamental computer vision challenge. Since this is not trivial and so many applications could be powered by face recognition (and when algorithms can recognize faces, other 3D objects will not be far behind), there's always a lot of resources going into developing robust, accurate algorithms.

     

     

    Many–perhaps dozens–of commercial and academic groups continually work on facial recognition and tracking technology. Someone has certainly done the landscape analysis on this topic. One of the face recognition research groups with which I've had contact is at Idiap in Martigny, Switzerland. Led by Sebastien Marcel, this research team is focusing on the use of such highly accurate facial recognition that it can be the basis for granting access. KeyLemon is an Idiap spin off using the Idiap technology for biometric authentication to personal computers. And, there is (almost certainly) a sizable group already in Google dedicated to this topic. 
     

  • What value added services or features can emerge that are not in conflict with Google's privacy policy and haven't been thought of already/implemented by Google and others?
    This is an important question that probably has a very long and complex, multi-part answer. I suspect it has a lot to do with 3D objects. What's great about studying faces is that there are so many different ones to work with and they are plastic (distort easily). When the algorithms for detecting, recognizing and tracking faces in video are available on mobile devices, we can imagine that other naturally occurring and plastic objects would not be too far behind.

I hope Eric Schmidt is proven wrong about there not being facial recognition in the future of Google Goggles and similar applications and we see what is behind the curtain in the PittPatt acquisition!