A barrier to this trend is the difficulty of getting a customer to download a company-specific app and to remember it is there when they need it. I suggested in my earlier comments that this would be eventually remedied by a “general-purpose” personal assistant, like Apple’s Siri or Google’s voice search knowing about such apps and calling them up when requested, much like we often find customer service web sites through a search engine.

However, there are a number of practical obstacles to this happening quickly. In particular, there is a dearth of such company-specific personal assistants. Until there are enough to make searching for a particular one likely to find a match, there isn’t much motivation for a general assistant or search engine to include this functionality.

The company [24]7 announced today an alternate approach that has the potential to jump-start the process. Rather than requiring downloading an app to use the visual and touch features of the mobile phone with speech, the company has a solution that converts a conventional customer service call from a mobile phone into a dual-channel experience, using both the voice and data channel simultaneously. One way it works is for the automated IVR system (or a human agent) to send a text message to the phone with a web link. Clicking on the link opens the phone’s web browser to a web site that is integrated with the IVR or agent’s software. The speech or text interaction can continue, but the customer can also view options visually and make choices by touching, much as if an app had been downloaded. This approach leads to a multimodal experience resulting from a conventional call to a toll-free customer service number.

When companies do have a personal assistant app, this process can end with the IVR or agent asking if they can provide a more convenient personal assistant app that will avoid the need to make a call (and potentially wade through an IVR menu). A click on an option on the web site could download the app (and register it with the general personal assistant, perhaps).

In the meantime, [24]7 has a compelling solution which they report has been highly successful in initial deployments. I understand they will even develop the application for qualified customers and charge only for completed transactions. Perhaps company personal assistant functionality will arrive faster than expected.

The smartphone provides an example of this always-there connection, and personal assistants and natural-language search technology as examples of the language connection. Examples such as Bluetooth earphones provide hands-free interaction options, although the limitation to speech interaction can be a possible impediment to their generality. Wearable displays such as Google’s experimental eyeglasses allow a more multimodal interaction at the cost of being perhaps a bit too obvious.

One option is a watch interface to a smartphone or other mobile device—a “smartwatch,” a new device worn on the wrist, with rumored flexible curved screen “iWatches” from Apple and Samsung possibilities reported in the press. A watch-like device is an interesting compromise between an earpiece and eyeglass display. The most compelling feature, from the view of an always-available connection to computers, is that it is essentially invisible until we use it, and many of us always wear a watch. It is less noticable than even an earpiece with microphone, yet can have at least a small display in addition to a microphone and speaker. If there is some way of entering text, it can also be used quietly (if somewhat inefficiently) when speech interaction is not an option. The mobile phone it connects with can remain in a pocket, purse, or briefcase. Some speculation has suggested a curved display, which might increase screen space. The always-there feature becomes very powerful for features such as reminding us of appointments or quietly informing us of incoming calls or messages (through vibration). It is at least more discreet to check the source of an incoming call or text message in a social or business situation on a watch than a smartphone.

The connection need not be with a mobile device. The watch could sense, for example, a compatible home automation system and allow the owner to, for example, change the thermostat. Or even act as a remote control for a smart TV.

Battery life could be an issue. Removable battery cartridges might be an option, with extra cartridges recharged outside the watch.

A smartwatch might be another area in which Apple or another company introduces us to an indispensible device we didn’t know we had to have. The market for replacing existing wristwatches or creating a new motivation to wear one is huge, particularly if one “upgrades” every couple of years. And the varieties could range from basic to gold-plated, a huge market opportunity if the device is done right (not a small challenge).

As a convenient, non-intrusive option tied to one or more mobile devices, the watch form factor makes perfect sense as an eventual winning product, given our desire to always have the Web, personal communications, and computer intelligence in general always ready to serve our needs. The small screen will many times simply motivate us to pull out our smartphone or look at our tablet computer, but could sufficiently augment those devices to fill a gap in the market.

The rapid adoption of smartphones and tablet computers reflects a basic societal trend—our increasing dependence on the Web and computer intelligence and a desire to always have that power available to us. My just-published book, The Software Society: Cultural and Economic Impact, discusses this trend and related trends in detail.

The growth of mobile devices is driven by a fundamental desire to always have computer intelligence available to us. This desire has driven the growth of smartphones and tablet computers. Before smartphones, “feature phones” allowed a more constant connection to other people through voice or texting, as well as, in some devices, access to email on the go. What smartphones largely added, beyond access to other people, was access to computers (e.g., the Web and navigation services). Arguably, this is a fundamental societal trend that makes computer intelligence something we increasingly depend upon.

The implications are subtle and deep. Perhaps one way of thinking about the long-term impact is to think about the experience of today’s young children. Eventually, almost all will grow up with tablet computers and smartphones, using apps for children even before they can read or write. Increasingly, even young children are given mobile phones. Most of us can type today in part because we grew up with personal computers. (Those of us old enough to have written reports by hand and given them to “secretaries” to type may reflect on how deep the change in habits has become, but younger people assume they should learn to type.) Today’s children will expect to always be connected to computers (as well as to other people not in the immediate vicinity). We will also all fairly quickly become dependent on the devices to keep track of much of our personal contacts and data (and to perhaps become our wallets).

As always-available becomes a requirement, issues will arise. One is a potential shortage of spectrum to support an increasing number of mobile users and data-intensive services such as video downloads. Another is battery life; an indication of this issue are reviews of mobile devices that discuss at length how long a device can last without being recharged. Another is convenience; we may want Bluetooth earpieces or Google-style eyeglass displays so that we don’t have to take the devices out of our pockets and so that our hands are free. “Availability” is an important driving need for future mobile devices (if not already a buying criterion).

As always-available becomes a necessity, one can foresee developments such as the following:

• Backup services for mobile devices, so that upgrading to a new device or replacing one that was damaged becomes painless and doesn’t result in loss of information. (Manufacturers are likely to offer this service to tie users to their brand.)
• Mobile devices that can download identities, becoming equivalent to your primary device. (You might even borrow a friend’s phone to do this if you forgot yours or its battery ran down.)
• Ubiquitous availability of the same personal information or preferences across devices, including some not as portable (e.g., PCs).

Availability features will become a key decision that drives device choice in the long term. Those features may also be part of cooperating “technology communities” (discussed in my book). A buyer may find a primary choice they must make is to which community they wish to belong. Companies not part of a community may find themselves struggling.

Chevrolet announced at the Los Angeles International Auto Show  on November 27 that, early next year, the Chevrolet Spark (Chevrolet’s first mini car for the U.S. and Canada) and Sonic LTZ and RS (five-door hatchbacks) will integrate Apple’s Siri. (The Chevy Impala is to have “natural” speech recognition in the MyLink system; see p. 7.) Apple announced earlier this year it was working with car manufacturers to build Siri technology into voice control systems in automobiles.

Google’s search app for Apple’s iPhone and iPad has been updated. The voice search feature now accepts inquiries in natural language and presents results more directly when possible (as opposed to solely a list of candidate web sites). The improvements make it an alternative to Siri, although the Google app must be launched before voice search is available, and Siri is always available, for example, by simply bringing the phone to one’s ear. Further, it is a web search application; it doesn’t have personal assistant functionality that requires integration with other apps on the phone, such as setting calendar reminders.

To use the speech recognition, one taps on the microphone icon in the search box and speaks an inquiry. Google gave an example of the natural language flexibility in a blog posting; one can say, for example, “Is United Airlines flight 318 on time?” In this case, since the answer is short, it is spoken aloud.

Google’s Knowledge Graph, which attempts to give more direct answers alongside the usual list of web sites, is also used to present results (SSN, September 2012, p. 1). Knowledge Graph uses semantics, going beyond word matching to try to understand the answer the user wants. Amit Singhal, SVP, Engineering, Google, gave an example of the search terms “taj mahal” in a blog. The inquiry might refer to the monument in India, a Grammy Award-winning musician, a casino in Atlantic City, NJ, or perhaps a local Indian restaurant. The “graph” understands real-world entities and their relationships to one another: “things, not strings,” as he put it. The figure at the end of this article shows how the Knowledge Graph puts up some of the information in the web sites directly in response to an inquiry on “Marie Curie,” even suggesting related searches that might interest you. Some further examples of inquiries and results are:

• “What does Yankee Stadium look like?” Hundreds of pictures appear instantly.
• “Play me a trailer of the upcoming James Bond movie.” The trailer starts playing immediately within Google Search.
• “When does daylight savings time end?” The answer will appear above the search results, so you can set your clock without having to click on a link.
•  “Who’s in the cast of The Office?” A complete cast list appears.

The same enhanced voice search seems to be included in the new version of Google TV.

Google’s Knowledge Graph for a search for Marie Curie

In the user interfaces of these devices, there is a tension between consistency of experience across devices and user interface models that are specific to the device characteristics. For example, a smartphone screen is much smaller than a PC screen and features touch rather than a mouse. It is thus difficult to simply replicate a personal computer user interface on a smartphone, despite the major applications being similar. As an example of the difference that the size makes, voice assistants like Siri and voice search are mostly used on smartphones; there doesn’t seem to be a big demand for voice interaction on personal computers. Yet, we would ideally like to be able to have a new device be familiar from our experience with other devices—a result of consistency.

The issue of specificity versus consistency is defining companies’ approaches to the marketplace. Apple has largely adopted the specificity model, with an attempt to make the interface for each device uniquely tuned to that device. They seek consistency mostly through applications and services such as iTunes. The Siri personal assistant could unify the experience further if they made it available on all their devices and accepted text input as well as voice.

Microsoft seems to have adopted the consistency model with Windows 8. They’ve attempted to make one interface suit a personal computer, a tablet, and a smartphone. Not surprisingly, some compromises were required, and some reviews have been critical, for example, about some features being hidden that were displayed on PCs. Microsoft faces the challenge that no one is familiar with Windows 8 on any device. The company has a good base of enterprise business and lots of cash, so they are positioned to support a long-term transition to the “consistency” model. It is a bit of a gamble perhaps, but the alternative could mean abandoning the consumer space.

On the other hand, Microsoft does seem to be focusing on the hardware aspects of devices to achieve specific advantages. The Surface tablet, for example, is an interesting attempt to include the option of a standard keyboard in a tablet format, creating an alternative to a laptop if the user wants to use Microsoft productivity software such as Word, PowerPoint, and Excel.

Google has the advantage of a basic consistent interface for its core service—the search box. Today, that box has evolved. The box allows voice or text requests in natural language and presents results that attempt a semantic understanding. Yet, customers still think of the search box as one specific “thing” with a consistent interface that is largely web-based and available on any device with a web browser.

But the broader picture for Google is their Android operating system, which has a full user interface. The open Android software provides consistency for hardware vendors using that operating system, but Google will apparently become their competitor as well with the Nexus phone and the Motorola Mobility acquisition. The consistency in this case would make it easy for a customer using any Android phone or tablet to move to a Google Android device. To have an advantage over competitors, however, Google would need some feature outside Android and search, since competitors will have those basics.

Amazon has created consistency in the online buying experience, expanding from books to almost anything one wants. It is also selling hardware now, with e-readers that are evolving into general tablet computers. The company has a core competency, like Google, in selling products, particularly books. It’s Kindle reading application is available as exclusive hardware, but is also available as an application for personal computers and tablet computers made by anyone. The application is highly consistent across platforms, and something done in the app on one platform (e.g., highlighting a paragraph in a book) is immediately available to the app on another platform, increasing consistency.

There is an alternative that balances consistency and specificity—adaptive interfaces. Limited user choices have always been available on all platforms through “preferences.” You can change some default parts of the user interface to alternatives, e.g., constantly showing a task bar or hiding it until invoked. You might choose one alternative on a larger screen and another on a smaller screen.

Adaptive solutions also can work without user intervention by changing software behavior by “personalization,” e.g., by learning what your interests are. Amazon’s listing of other products you might like, based on previous purchases, is one example. Often, the adaption isn’t visible, but behind the scenes, such as when you add a product name to the vocabulary of a voice dictation system.

Consistency creates a community of devices. Specificity can create unique advantages. Adaptation may ease the tension between these extremes.

(Originally published in Speech Strategy News, August 2012)

AT&T Research (formerly Bell Labs) has been involved in speech technology research for many decades, for example, developing a continuous digit recognizer in the 50s. (See the interview with Mazin Gilbert, AVP of the Intelligent Systems Organization, AT&T Research, SSN, May 2012, p. 15.) The company’s speech recognition technology has found a home in some deployed applications, including the Vlingo voice assistant that is part of the new Samsung Galaxy III, with the assistant called S-Voice (SSN, July 2012, p. 1). (Vlingo is now part of Nuance, so the technology used in the Samsung phone lines may evolve eventually to Nuance technology.) Among other applications, AT&T speech technology has been used within AT&T for IVR customers for over 20 years.

The AT&T Watson speech technology has now been made available to developers as a network-based service accessed through an Application Programming Interface (API) that AT&T recently released. Gilbert summarized in a note to Speech Strategy News: “By exposing the speech APIs, we are lowering the barrier to entry for developers to empower their applications with speech. The responses we have received so far have been overwhelming. Our plans don’t stop here. We will continue to expose additional APIs and innovations to enable developers to create more advanced and personalized mobile applications ranging from virtual assistants to interactive gaming. Stay tuned!”

There is a registration charge of $99 for developers, Gilbert indicated, which will allow developers to use all AT&T APIs, including speech, as they become available, without a per transaction charge through 2012. Gilbert said that AT&T is working on pricing beyond 2012, and current projections have pricing at about one cent for most “small transcriptions.” He said AT&T will review pricing as we get closer to 2013, but he does not anticipate pricing “going anywhere but down.” (More detailed pricing information is available online. AT&T also sent out an eblast with a discount code that allows getting the API with the $99 fee waived through August.)

AT&T Watson is a network-based engine that integrates a variety of speech capabilities, including speaker-independent speech recognition, AT&T Natural Voices text-to-speech, speaker verification, natural language understanding, LLAMA-based machine learning, search, translation, and dialog management. AT&T says that the Watson speech engine continuously improves accuracy by learning different accents and speech patterns. WATSON can combine speech with other modalities, such as a touch-screen tap (“show me the closest coffee shop to here”) or other gesture (see figure). AT&T said in advertising material that AT&T has accumulated more than 600 patents on the AT&T Watson technology.

Watson uses a plugin architecture where each subtask is contained in its own plugin. Depending on the task to be performed, Watson selects the right plugins at run time, assembles them into a working engine, and coordinates the information exchange between the plugins. It also handles communication with the end device.

However, only speech recognition (speech-to-text transcription with Statistical Language Models that are tuned for specific “contexts”) is available initially with the current API. The API allows sending audio and receiving back text. AT&T indicated that native and HTML5-based Software Development Kits (SDKs) would be available “soon.”

The contexts make the speech recognition more accurate and also support specialized vocabularies, including:

• Generic speech-to-text (general dictation, automatically detects English or Spanish, and returns the appropriate text transcription);
• Web Search speech-to-text;
• Local Business Search speech-to-text;
• Voicemail-to-text;
• SMS (text message) speech-to-text;
• Question Transcription (converts questions to text); and
• TV Speech to Text (AT&T’s U-verse video programming guide).

The contexts are language models built, maintained, and tuned by AT&T.

AT&T is also offering the AT&T Application Resource Optimizer (ARO) as open source code. ARO is a free diagnostic tool that helps to optimize a mobile app’s performance, speed, network impact, and battery utilization.

AT&T Watson takes input, analyzes it, performs one or more services, and returns a result in real time. Input can be audio files, speech, gestures, face recognition, and text. (Source: AT&T)

Direct-to-content approach goes beyond classical search

Dragon Go! was Nuance Communications’ personal assistant application for smartphones, requiring somewhat more structured requests than Apple’s Siri. Nuance announced on October 24 that it had renamed and upgraded the app, calling it a “Beta” release of the Dragon Mobile Assistant for Android. (See Editor’s Notes, p. 5, re naming of personal-assistant-like apps.) The assistant app is available free in English on Google Play in the US, supporting Android Ice Cream Sandwich (4.0) and later versions.

Dragon has expanded the natural language understanding capabilities of the assistant app.  The Dragon Mobile Assistant sends texts, makes calls, and sets appointments. It delivers direct access to open content on the mobile web from more than 200 content providers, bypassing web search in cases when it can determine a web site that matches a user request versus “having to scroll through blue links,” as a Nuance spokesperson put it.  Content is presented via Nuance’s “carousel,” where people can see like content from additional content providers, as well as what is available via Google or Bing!, YouTube, Twitter, and Wikipedia. Content providers include AccuWeather, OpenTable, Yelp, ESPN, Expedia, Fandango, Ask.com, and others.

Michael Thompson, executive vice president and general manager, Nuance Mobile, summarized, “Consumers loved the original Dragon Go! and wanted even more. So we totally redesigned the experience in a conversational way and added the most popular personal assistant features.”

Nuance provided examples of the conversational mode accepted by Dragon Mobile Assistant:

• “When is Finding Nemo 3D playing today?”
• “Set up a meeting at noon tomorrow titled Lunch with Jake Barnes”
• “Tell Natalie, ‘I’m on the way to the office and will grab you a Starbucks’”
• “Get directions to the New England Aquarium”
• “How’s the weather look this week?”
• “Send out a new Tweet, I can’t believe that finish to the game!”

The Mobile Assistant can also guide someone through an interaction:

• User: “I need to create an appointment”
• Dragon: “When should it start?”
• User: “7pm tomorrow”
• Dragon: “Here is your appointment, would you like to save, edit, or cancel it?”
• User: “Change the subject to “Dinner with John”
• Dragon: “Here are the best matches for John, please select.” (if more than one contact named “John”)

For a hands-free experience, the application can be awakened by saying, “Hi Dragon.” Inbound text messages can be spoken using Driver Mode.

The company indicated that expanded availability and new features are planned before the end of 2012. Vlad Sejnoha, chief technology officer, Nuance, said that Nuance is “investing deeply in Natural Language Understanding and conversational dialog technologies.” He stated, “We will continue to add new innovations to the Dragon Mobile Assistant to explore what’s possible, and to push the limits of intelligent voice interfaces across all mobile form factors, including phones, tablets, PCs, cars, and televisions.”

These initiatives are a major refocusing for Microsoft, but one that the company has been signaling with its actions. For example, Windows 8 is a courageous (and perhaps dangerous) leap from previous versions, with a number of innovative features that make it suitable for touch devices, yet retaining compatibility with past applications, a tough balancing act. The Kinect voice and gesture controller for Xbox was innovative, and the company has continued to expand its use.

The surprise to me was the “focus areas” Ballmer listed in the letter. The first two out of the five he listed were, as written:

• Developing new form factors that have increasingly natural ways to use them including touch, gestures and speech.
• Making technology more intuitive and able to act on our behalf instead of at our command with machine learning.

My background includes having edited two books on voice user interface design, founding a speech recognition company, and having written a technical book on Computer Pattern Recognition, but I must admit that I never expected to see mentions of natural user interfaces and machine learning in a short message to shareholders by the CEO of one of the largest companies in the U.S. Their mention as focus areas suggests that areas once viewed as leading-edge technology have achieved mainstream importance, to the degree that their successful deployment can impact the future of a major company.

My interpretation is that Microsoft recognizes that one can only stretch the Graphical User Interface so far, whether we use touch or a mouse. As we add more features and applications – however clever we are about making complexity more accessible by pointing or sliding – finding a specific capability or information can be difficult and annoying, particularly on smaller-screen devices. The only solution that will allow us to continue adding utility to our devices is a model that lets us simply ask for what we want (in text or voice) and gives it to us as directly as possible (or even anticipates what we might want before we ask). In a January blog entry, I outlined some of the technology required to meet this objective: What does it take for mobile personal assistants to “understand” us? Earlier this month, I noted that we will want all this understanding to be consistently available on whatever device we are using, a further challenge: see Ubiquitous Personal Assistants.

The “machine learning” reference can be interpreted globally: learning from user behavior what people in general want when they are in a certain situation (e.g., leaving for work in the morning) or when they ask for one thing that will lead to another (asking for a Chinese restaurant nearby is likely to lead to asking for a review and/or a reservation). It can be interpreted personally: learning from your activities and requests what, when, and where you want specific information or actions. Machine learning technology is also involved in speech recognition that adapts to your voice as you use it and gets more accurate over time. In any case, the results can be impressive when done well. Machine learning benefits from the huge amount of data available for analysis from Web and mobile interactions and the computing power available today to analyze that data.

Apple and Google are also aiming at this type of natural interaction. All three companies are developing integrated ecosystems that can tightly couple our human intelligence with computer intelligence across a range of products. And they have the budgets to make it happen.