Speech Recognition and Natural Language Processing as a Highly Effective Means of Human-Computer Interaction

By Mike Machowski

Abstract

     As computers have become more pervasive in many parts of society it has become clear that most people have great difficulty understanding and communicating with computers. Often users are unable to simply state what they want done and must learn archaic commands or non-intuitive procedures in order to get anything done. Furthermore, such communication is often accomplished via slow, difficult to use devices such as mice or keyboards. It is becoming clear that a easier, faster, and more intuitive method of communicating with computers is needed. One proposed method is the combination of Speech Recognition and Natural Language Processing software. Speech Recognition (SR) software is software that has the ability to audibly detect human speech and parse that speech in order to generate a string of words, sounds or phonemes to represent what the person said. Natural Language Processing (NLP) software has the ability to process the output from Speech Recognition software and understand what the user meant. The NLP software could then translate what it believes to be the user's command into an actual machine command and execute it.

How Speech Recognition and Natural Language Processing Work

    Speech recognition and Natural Language Processing systems are tremendously complex pieces of software. While their are a variety of algorithms used in the implementation of such systems, there seem to be something of a standard understanding of the fundamental methods used. Speech Recognition works by disassembling sound into atomic units and then piecing back together, while Natural Language Processing attempts to translate words into ideas by examining context, patterns, phrases, etc.

    Speech recognition works by breaking down sounds the hardware "hears" into smaller, non-divisible, sounds called phonemes. Phonemes are distinct, atomic units of sound. For example, the word "those" is made up of three phonemes; the first is the "th" sound, the second the hard "o" sound, and the final phoneme the "s" sound. A series of phonemes make up syllables, syllables make up words, and words make up sentences, which in turn represent ideas and commands. Generally, phonemes can be thought of as the sound made by one or more letters in sequence with other letters. When the SR software has broken sounds into phonemes and syllables, a "best guess" algorithm is used to map the phonemes and syllables to actual words.

    Once the Speech Recognition software translates sound into words, Natural Language Processing software takes over. NLP software parses strings of words into logical units based on context, speech patterns, and more "best guess" algorithms. These logical units of speech are then parsed and analyzed , and finally translated into actual commands the computer can understand based on the same principles used to generate logical units.

    Optimally Speech Recognition and Natural Language Processing software can work with each other in a non-linear fashion in order to facilitate better comprehension of what the user says and means. For example, a SR package could ask a NLP package if it thinks the "tue" sound means "to", "two", "too", or if it is part of a larger word such as "tutelage". The NLP system could make a suggestion to the SR system by analyzing what seems to make the most sense given the context of what the user has previously said. It could work the other way around as well. For example, a NLP system could query a SR system to see if a user seemed to emphasize a certain word or phrase in a given sentence. If the NLP realizes when the user emphasizes certain words, it may be able to more accurately determine what the user wants. ( e.g. the sentence "I don't like that !" differs subtly, yet importantly from the sentence " I don't like that!") SR systems may be able to determine which sounds or words were emphasized by analyzing the volume, tone, and speed of the phonemes spoken by the user and report that information back to the NLP system.

Problems with Speech Recognition and Natural Language Processing

     So why isn't Speech Recognition and Natural Language Processing use more widespread? Thus far, it has proven too difficult and too impractical to provide these services on most systems. SR has been plagued by problems stemming from the difficulties of understanding different types of voices ( e.g. male vs. female voices), parsing sounds when people have different dialects ( e.g. different accents), distinguishing between background noise and commands issued to the computer, etc. Moreover, if SR is to work in real time, the software must have access to a large, fast database of known words and the ability to add more words. Natural Language Processing software's problems are even more difficult to overcome the Speech Recognition's. NLP must be able to understand sentences peppered with verbal artifacts, slang, synonyms, ambiguities, and colloquialisms.

    Speech recognition packages have begun to deal with some of these problems and the results have been promising. Historically, SR software has been plagued with problems stemming from differences in pronunciation, enunciation, and speech patterns. For example, the way a child with a high-pitched voice and a southern-drawl pronounces "gravel" may differ significantly from how a deep-voiced man from the northeast pronounces the same word; yet adept SR software should be able to determine that both people are speaking the same word. This can be accomplished by allowing variable patterns of phonemes make up a given word. Of course, doing so will increase the size of the database needed to map phonemes to words. However, this issue is becoming less problematic as computers become faster and cheaper. Indeed, this problem has become so trivial that some computerized telephone services have begun using speech recognition software to gather information from users (Admittedly, the vocabulary of these systems is extremely limited; e.g. A computer will ask a user some simple questions with "yes" or "no" answers). The problem of distinguishing speech directed at the computer from background noise has not been dealt with as successfully. Currently, users of SR packages often must either work in an environment with minimal background noise, or must wear a head set with a sampling microphone inches from his or her mouth. Furthermore, users often have to "inform" the computer when it is being spoken to. This often implies pressing some button ( e.g. a foot pedal) or similar action. Certainly, this is not the most desirable user interface. It is inefficient, taxing, and is not "user-centered."

    Verbal artifacts must be dealt with. Verbal artifacts are words or phrases that are spoken, but add little, if any, content to a sentence. For example, the sentence. "Golly, I sure love pudding!" contains two verbal artifacts: "Golly" and "sure". These two words add little to the meaning the speaker of the sentence was attempting to convey. NLP software must be able to identify these types of words for what they are and react appropriately. NLP also needs to be able to recognize that human beings are capable of conveying a single idea in synonymous ways. This is not as simple a task as it might initially seem. For instance, NLP systems must be able to understand that a user saying "Take a memo." is conveying virtually the same idea as a user saying "Why don't you record this for me?" The first sentence is direct and rather unambiguous, but the second sentence comes in the form of a question even though it is really a request. Surely, one can understand where some confusion could arise in such a situation.

    Unfortunately researchers exploring NLP systems have not yet been able to develop systems that are robust enough to handle these dilemmas. Currently these problems are dealt with by simply "hard-coding" certain phrases and words that are synonymous. Ultimately, NLP will need to be able to recognize and react to such synonyms by the context it comes in, the user's habits ( e.g. does the user normally make requests in the form of a question, or is the user actually asking a question), etc. Moreover, NLPs must have an extraordinary understanding of grammatical rules, practices, and structures. Furthermore, truly adept NLPs would need to be able to identify and react accordingly to sarcasm, humor, rhetorical questions, etc.

Benefits of Speech Recognition and Natural Language Processing

    Why then should we try to implement Speech Recognition and Natural Language Processing if it is so hard to do? Simply put, SR and NLP could revolutionize the entire field of Human-Computer interaction like nothing before. SR and NLP can completely abstract Human-Computer Interaction, eliminating the need to understand anything about the computer's internal workings or how to accomplish certain tasks.

"Indeed, one should be able to say to a computer 'Do what I meant, not what I said.' "  

      What will be some specific benefits of SR and NLP interfaces? SR and NLP will allow real time language translation. If a computer can figure out what words one utters and understand what one actually means, it is a trivial task to translate an idea from one language to another. That's the key: computers with capable Natural Language Processing abilities will begin to act on the ideas that their users have, not the commands explicitly given to them. Indeed, one should be able to say to a computer "Do what I meant, not what I said."