Recognizing Multiple Keywords Using Pocketsphinx

Recognizing multiple keywords using PocketSphinx

you can use addKeywordSearch which uses to file with keyphrases. One phrase per line with threshold for each phrase in //, for example

up /1.0/
down /1.0/
left /1.0/
right /1.0/
forwards /1e-1/

Threshold must be selected to avoid false alarms.

how to use pocketsphinx in android for multiple keyword activation without going to menu

You have asked two question so I will answer them both in points.

• I want to call some methods directly when it recognizes the three keywords
  
  If you look at the demo app's source code, you will notice that there are two functions that are getting control and performing computation on the hypothesis (recognized keywords) - onPartialResult() and onResult. The former is used to get quick updates about the words being spoken while the latter is used to get the predictions after the recognizer has been stopped, i.e. after all the keywords have been spoken and the user has stopped the recognizer.
  
  So, first you need to decide whether you want to use onPartialResult() or onResult(). After that, notice that both methods perform some operations on the recognized text. onPartialResult() calls switchSearch() while onResult() calls makeText(). In your case too, you want to call some function when it recognizes your three keywords. Create a function in the same class and call it as you'd normally call the function anywhere else in your program!
• I also don't know whether I need to use switchSearch() and KWS_SEARCH like in the demo app.
  
  In that case, you should look at the answer suggested by Nikolay Shmyrev in the comments above. It explicitly details how to recognize several keywords in Android.

Listen to more than one KeyphraseSearch

You can create the file in assets with keyphrase list of the following form (one per-line with the following threshold):

 oh mighty computer /1e-40/
 hello my name is /1e-38/

and create a search with addKeywordSearch which loads a file instead of addKeyphraseSearch.

pocketsphinx - how to switch from keyword spotting to grammar mode

A quote from tutorial:

Developer can configure several “search” objects with different grammars and language models and switch them in runtime to provide interactive experience for the user.

There are different possible search modes:

• keyword - efficiently looks for keyphrase and ignores other speech.
  allows to configure detection threshold
• grammar - recognizes speech
  according to JSGF grammar. Unlike keyphrase grammar search doesn't
  ignore words which are not in grammar but tries to recognize them.
• ngram/lm - recognizes natural speech with a language model.
• allphone - recognizes phonemes with a phonetic language model.

Each search has a name and can be referenced by a name, names are application-specific. The function ps_set_search allows to activate the search previously added by a name.

To add the search one needs to point to the grammar/language model describing the search. The location of the grammar is specific to the application. If only a simple recognition is required it is sufficient to add a single search or just configure the required mode with configuration options.

The exact design of a searches depends on your application. For example, you might want to listen for activation keyword first and once keyword is recognized switch to ngram search to recognize actual command. Once you recognized the command you can switch to grammar search to recognize the confirmation and then switch back to keyword listening mode to wait for another command.

The code to switch searches in Python looks like this:

# Init decoder
config = Decoder.default_config()
config.set_string('-hmm', path.join(MODELDIR, 'en-us/en-us'))
config.set_string('-dict', path.join(MODELDIR, 'en-us/cmudict-en-us.dict'))
decoder = Decoder(config)

# Add searches
decoder.set_kws('keyword', 'keyword.list')
decoder.set_lm_file('lm', 'query.lm')
decoder.set_search('keyword')

import pyaudio
p = pyaudio.PyAudio()
stream = p.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True, frames_per_buffer=1024)
stream.start_stream()

in_speech_bf = False
decoder.start_utt()
while True:
    buf = stream.read(1024)
    if buf:
        decoder.process_raw(buf, False, False)
        if decoder.get_in_speech() != in_speech_bf:
            in_speech_bf = decoder.get_in_speech()
            if not in_speech_bf:
                decoder.end_utt()

                # Print hypothesis and switch search to another mode
                print 'Result:', decoder.hyp().hypstr

                if decoder.get_search() == 'keyword':
                     decoder.set_search('lm')
                else:
                     decoder.set_search('keyword')

                decoder.start_utt()
    else:
        break
decoder.end_utt()

How does pocketsphinx output keywords in spotting mode

If you suppress stderr with 2> /dev/null or with -logfn /dev/null, you will see the following in stdout in case word is detected:

talk

In case you want to see the times you can also add -time yes to command line, then you will see on stdout:

talk talk
talk 7.210 7.300 0.947237
talk 6.820 6.920 0.955132

Pocketsphinx in python returns random words in keyword search

You need to remove this line

  config.set_string('-lm', lmdir )

Keyphrase search and lm search are mutually exclusive.

PocketSphinx differences between key phrase and grammar

It is currently not possible to create robust grammar, this feature is not supported. For that reason it is recommended to use keyword spotting mode if you want to listen continuously.

Keyphrase can be multiple words and you can also use multiple keyphrases, the sample for that is covered in

Recognizing multiple keywords using PocketSphinx

---

Related Topics