--- source_path: "models/tpl/tpl-spot-vad-lvcsr.md" canonical_url: "https://doc.sensory.com/tnl/7.8/models/tpl/tpl-spot-vad-lvcsr/" --- # tpl-spot-vad-lvcsr _(TrulyNatural only)_ This [template](https://doc.sensory.com/tnl/7.8/models/tpl/index.md#template-type) runs the [wake word](https://doc.sensory.com/tnl/7.8/models/types/wake-word.md#wake-word-type) in slot [0](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#0) until it detects, segments the audio following the wake word with a [VAD](https://doc.sensory.com/tnl/7.8/models/types/vad.md#vad-type), and sends the segmented audio to the [LVCSR](https://doc.sensory.com/tnl/7.8/models/types/lvcsr.md#lvcsr-type) or [STT](https://doc.sensory.com/tnl/7.8/models/types/stt.md#stt-type) recognizer in slot [1](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#1). This behavior is also available in the [tpl-opt-spot-vad-lvcsr](https://doc.sensory.com/tnl/7.8/models/tpl/tpl-opt-spot-vad-lvcsr.md#tpl-opt-spot-vad-lvcsr-type) template, which adds an option to skip the wake word. `tpl-spot-vad-lvcsr` has [task-type](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#task-type)` == `[phrasespot](https://doc.sensory.com/tnl/7.8/api/setting-keys/values.md#phrasespot). Expected [task types](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#task-type): * **Slot 0:** [phrasespot](https://doc.sensory.com/tnl/7.8/api/setting-keys/values.md#phrasespot) * **Slot 1:** [lvcsr](https://doc.sensory.com/tnl/7.8/api/setting-keys/values.md#lvcsr) **Also see these related items:** [tpl-spot-vad-lvcsr-3.23.0.snsr](https://doc.sensory.com/tnl/7.8/models/index.md#tpl-spot-vad-lvcsr), [tpl-opt-spot-vad-lvcsr](https://doc.sensory.com/tnl/7.8/models/tpl/tpl-opt-spot-vad-lvcsr.md#tpl-opt-spot-vad-lvcsr-type) ## Operation ```mermaid flowchart TD start((start)) start --> startWW subgraph slot0[slot 0 (phrasespot)] startWW((start)) fetchWW[/samples from ->audio-pcm/] audioWW(^sample-count) processWW[process] result(0.^result) stopWW((stop)) startWW --> fetchWW fetchWW --> audioWW audioWW --> processWW processWW --> fetchWW processWW -->|recognize| result result --> stopWW end subgraph slot1[slot 1 (lvcsr)] startSTT((start)) startSTTfinal((start)) stopSTT((stop)) stopSTTpartial((stop)) processSTT[process] partialSTT(^result-partial) intentSTT(^nlu-intent) slotSTT(^nlu-slot) resultSTT(^result) nluSTT{NLU
match?} slmSTT{SLM
included?} generateSTT[generate] slmstartSTT(^slm-start) slmresultpartialSTT(^slm-result-partial) slmresultSTT(^slm-result) startSTT --> processSTT processSTT ---->|hypothesis| partialSTT partialSTT --> stopSTTpartial startSTTfinal --> nluSTT nluSTT -->|yes| intentSTT nluSTT -->|no| resultSTT intentSTT --> slotSTT slotSTT --> resultSTT slotSTT -->|more| intentSTT resultSTT --> slmSTT slmSTT -->|yes| slmstartSTT slmSTT -->|no| stopSTT slmstartSTT -->|OK| generateSTT slmstartSTT -->|STOP| stopSTT generateSTT -->|response| slmresultpartialSTT slmresultpartialSTT --> generateSTT generateSTT -->|done| slmresultSTT slmresultSTT --> stopSTT end listenBegin(^listen-begin) listenEnd(^listen-end) stopWW --> listenBegin listenBegin --> fetch0 fetch0[/samples from ->audio-pcm/] fetch1[/samples from ->audio-pcm/] audio0(^sample-count) audio1(^sample-count) silence(^silence) begin(^begin) END(^end) limit(^limit) process0[VAD process] process1[VAD process] final@{ shape: f-circ } fetch0 --> audio0 audio0 --> process0 process0 --> fetch0 process0 -->|speech start| begin process0 -->|timeout| silence silence ~~~ final silence --> listenEnd begin --> fetch1 fetch1 --> audio1 audio1 --> process1 process1 --> startSTT stopSTTpartial --> fetch1 process1 -->|speech end| END process1 -->|speech limit| limit END --> final limit --> final final --> startSTTfinal stopSTT --> listenEnd listenEnd --> startWW ``` Operation flow. 1. Read audio data from [->audio-pcm](https://doc.sensory.com/tnl/7.8/api/setting-keys/runtime.md#-audio-pcm). 2. Invoke [^sample-count](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#sample-count-event). 3. If processing does not detect a wake word, continue at step 1. 4. Invoke [0.^result](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#result) for the wake word. 5. Invoke [^listen-begin](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#listen-begin) and start VAD processing. 6. Read audio data from [->audio-pcm](https://doc.sensory.com/tnl/7.8/api/setting-keys/runtime.md#-audio-pcm). 7. Invoke [^sample-count](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#sample-count-event). 8. If VAD processing does not detect the start of speech within the [leading-silence](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#leading-silence) timeout, invoke [^silence](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#silence) and continue at step 15. 9. Invoke [^begin](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#begin) if processing detects the start of speech, else continue at step 6. 10. Read audio date from [->audio-pcm](https://doc.sensory.com/tnl/7.8/api/setting-keys/runtime.md#-audio-pcm). 11. Invoke [^sample-count](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#sample-count-event). 12. If VAD processing detects an endpoint invoke either [^limit](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#limit) or [^end](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#end) and continue at step 14. 13. Process VAD segmented audio in the LVCSR or STT recognizer * Invoke [^result-partial](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#result-partial) with interim recognition result hypothesis. * Continue at step 10. 14. Produce a final LVCSR or STT recognition hypothesis. * Invoke [^nlu-intent](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#nlu-intent) and [^nlu-slot](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#nlu-slot) for each NLU intent found. * Invoke [^result](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#result) with the final recognition hypothesis. * If there's no SLM, continue at step 15. * Invoke [^slm-start](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#slm-start), if the callback returns [STOP](https://doc.sensory.com/tnl/7.8/api/inference.md#rc_stop), continue at step 15. * Generate SLM result, invoking [^slm-result-partial](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#slm-result-partial) on each generated token. * Invoke [^slm-result](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#slm-result) with complete SLM result. 15. Invoke [^listen-end](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#listen-end) and start listening for the wake word again at step 1. Register callback handlers with [setHandler](https://doc.sensory.com/tnl/7.8/api/inference.md#sethandler) only for those events you're interested in. ## Settings **Available events:** [^begin](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#begin), [^end](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#end), [^limit](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#limit), [^listen-begin](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#listen-begin), [^listen-end](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#listen-end), [^nlu-intent](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#nlu-intent), [^nlu-slot](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#nlu-slot), [^result](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#result), [^result-partial](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#result-partial), [^sample-count](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#sample-count-event), [^silence](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#silence), [^slm-result](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#slm-result), [^slm-result-partial](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#slm-result-partial), [^slm-start](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#slm-start) **Available iterators:** [operating-point-iterator](https://doc.sensory.com/tnl/7.8/api/setting-keys/iterators.md#operating-point-iterator), [vocab-iterator](https://doc.sensory.com/tnl/7.8/api/setting-keys/iterators.md#vocab-iterator) **Available results:** [audio-stream](https://doc.sensory.com/tnl/7.8/api/setting-keys/results.md#audio-stream), [audio-stream-first](https://doc.sensory.com/tnl/7.8/api/setting-keys/results.md#audio-stream-first), [audio-stream-last](https://doc.sensory.com/tnl/7.8/api/setting-keys/results.md#audio-stream-last) **Available runtime settings:** [->audio-pcm](https://doc.sensory.com/tnl/7.8/api/setting-keys/runtime.md#-audio-pcm), [audio-stream-from](https://doc.sensory.com/tnl/7.8/api/setting-keys/runtime.md#audio-stream-from), [audio-stream-to](https://doc.sensory.com/tnl/7.8/api/setting-keys/runtime.md#audio-stream-to) **Available configuration settings:** [audio-stream-size](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#audio-stream-size), [audio-stream-size](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#audio-stream-size), [backlog-interval](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#backlog-interval), [backoff](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#backoff), [custom-vocab](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#custom-vocab), [delay](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#delay), [duration-ms](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#duration-ms), [hold-over](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#hold-over), [include-leading-silence](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#include-leading-silence), [include-wake-word-audio](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#include-wake-word-audio), [leading-silence](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#leading-silence), [low-fr-operating-point](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#low-fr-operating-point), [max-recording](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#max-recording), [operating-point](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#operating-point), [partial-result-interval](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#partial-result-interval), [samples-per-second](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#samples-per-second), [stt-profile](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#stt-profile), [sv-threshold](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#sv-threshold), [wake-word-at-end](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#wake-word-at-end) **Available values:** [lvcsr](https://doc.sensory.com/tnl/7.8/api/setting-keys/values.md#lvcsr), [phrasespot](https://doc.sensory.com/tnl/7.8/api/setting-keys/values.md#phrasespot) **Also see these related items:** [live-spot.c](https://doc.sensory.com/tnl/7.8/api/sample/c/live-spot.md#live-spot-code), [snsr-eval.c](https://doc.sensory.com/tnl/7.8/api/sample/c/snsr-eval.md#snsr-eval-code), [PhraseSpot.java](https://doc.sensory.com/tnl/7.8/api/sample/android/enroll-trigger.md#et-code) ## Notes Use this template for command and control type applications where commands are initiated with a wake word. The [^result-partial](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#result-partial) and [^result](https://doc.sensory.com/tnl/7.8/api/setting-keys/events.md#result) events are for the LVCSR or STT recognizer in slot 1. If you need direct access to the wake word result, prefix the event with the slot path: `0.^result` Use the slot prefix to read values in the `0.^result` event handler too, for example call [getString](https://doc.sensory.com/tnl/7.8/api/inference.md#getters) with key [0.text](https://doc.sensory.com/tnl/7.8/api/setting-keys/results.md#text) to read the wake word transcription. Set [include-wake-word-audio](https://doc.sensory.com/tnl/7.8/api/setting-keys/configuration.md#include-wake-word-audio)` = 1` to include the wake word audio in the samples passed to the LVCSR or STT recognizer. STT hypotheses do not include the wake word text unless Sensory specifically configured the model to do so. ## Examples ```console % cd $HOME/Sensory/TrulyNaturalSDK/7.9.0-pre.0 % bin/snsr-edit -o vg-stt.snsr\ -t model/tpl-spot-vad-lvcsr-3.23.0.snsr\ -f 0 model/spot-voicegenie-enUS-6.5.1-m.snsr\ -f 1 model/stt-enUS-automotive-medium-2.3.15-pnc.snsr\ -s include-wake-word-audio=1 # Say "Voice genie, open the sunroof." % snsr-eval -vt vg-stt.snsr Using live audio from default capture device. ^C to stop. P 2770 3250 (0.4166) Open the sun P 2810 3650 (0.7161) Open the sunroof 1815 3990 [^end] VAD speech region. NLU intent: open_window (0.9956) = open the sunroof NLU entity: roof (0.9595) = sunroof 2810 3690 (0.4394) Open the sunroof. ^C ``` *[API]: Application Programming Interface *[FR]: False Reject: the recognizer did not trigger when the target phrase was spoken *[LVCSR]: Large Vocabulary Continuous Speech Recognition model, feed-forward neural net acoustic model with FST decoder *[NLU]: Natural Language Understanding model *[SLM]: Generative Small Language Model *[STT]: Speech To Text: transformers with language model and CTC decoding *[TNL]: TrulyNatural, Sensory's large-vocabulary speech recognition technology *[VAD]: Voice Activity Detector