Recognize a new spoken word from a few examples
A new voice command should need examples, not a new training run. PLiX learns an acoustic embedding across languages, then recognizes unseen words by comparing a query with prototypes built from a few labeled support clips.
Multilingual episodic training learns to compare words across 20 languages.
Support-derived prototypes let a frozen encoder recognize a new vocabulary without retraining.
Transfer and hardware evaluation examine unseen languages, language identification and encoder latency/energy.
As technology advances and digital devices become prevalent, seamless human-machine communication is increasingly gaining significance. The growing adoption of mobile, wearable, and other Internet of Things (IoT) devices has changed how we interact with these smart devices, making accurate spoken words recognition a crucial component for effective interaction. However, building robust spoken words detection system that can handle novel keywords remains challenging, especially for low-resource languages with limited training data. Here, we propose PLiX, a multilingual and plug-and-play keyword spotting system that leverages few-shot learning to harness massive real-world data and enable the recognition of unseen spoken words at test-time. Our few-shot deep models are learned with millions of one-second audio clips across 20 languages, achieving state-of-the-art performance while being highly efficient. Extensive evaluations show that PLiX can generalize to novel spoken words given as few as just one support example and performs well on unseen languages out of the box. We release models and inference code to serve as a foundation for future research and voice-enabled user interface development for emerging devices.
Read more ↗The multilingual base uses EfficientNet-v2 Medium, trained in 10-way, 5-shot episodes across 20 languages. Training and test vocabularies contain disjoint words.
§2.2–3.3. The TinyNet-E alternatives are separately trained language-specific models, not a compressed copy of the multilingual base.
For a requested word c, encode each labeled support clip and average its embedding. The query is excluded from this computation.
§2.1.2, p. 3; Figure 1 explicitly separates the query path from mean pooling.
Choose the prototype closest to the query embedding in squared Euclidean distance. Changing support clips changes the classifier while leaving encoder weights fixed.
§2.1.2, expressed equivalently to the paper’s argmax of negative squared distance. Most word tests use one-second clips.
Recognition tests vary vocabulary size, support count and language. Hardware profiling measures encoder inference only on Raspberry Pi 4, Jetson Nano and Pixel 7.
Table 1 excludes log-mel extraction and prototype computation. Released weights and inference instructions are linked through aaqibsaeed/plixkws on Hugging Face.
Results from arXiv:2305.03058v1 ↗
The same encoder can recognize a new vocabulary from support clips. These are qualitative observations from separate recognition experiments.
| Evaluation | Observed result | Evidence |
|---|---|---|
| Multilingual base · held-out words | Recognizes new words with one support clip; accuracy varies by language and improves with more support. | Figure 6 · p. 9 |
| English base · larger vocabulary | More competing words reduce one-shot accuracy; more support clips improve recognition. | Figure 7 · pp. 9–10 |
| Multilingual base · unseen languages | Transfers without retraining to five languages; Tamil is the most difficult in the reported one-shot test. | Figure 10 · pp. 11–12 |
| Multilingual base · language identification | Transfers to FLEURS without retraining; the ten-language episodes are harder than the smaller tasks. | Figure 11 · p. 12 |
Qualitative reading of the recognition figures, not values estimated from bar heights. Held-out-word tests use closed-set MSWC episodes; transfer tests use five unseen MSWC languages and FLEURS. Accuracy is averaged over 1,000 episodes without plotted uncertainty intervals.
These are qualitative observations from separate experiments, not a pooled performance comparison. English-only and multilingual encoders have different training data. FLEURS contains 102 languages, but each reported episode distinguishes only 3, 5 or 10 languages.
The small encoder is faster in every measured configuration. These timings exclude preprocessing and prototype computation.
TinyNet-E (“Small”) and EfficientNet-v2 Medium (“Base”); average over 200 inference runs. Latency in milliseconds, lower is better; no dispersion reported.
Encoder-only timings exclude log-mel computation and the prototype classifier. Pixel 7 was tested only with quantized models; accuracy after quantization is not reported here.
| Device and model format | Small (ms) | Base (ms) |
|---|---|---|
| Raspberry Pi 4 · standard | 240.17 | 4480 |
| Jetson Nano · standard | 35.12 | 1954 |
| Pixel 7 · quantized | 5.12 | 78.07 |
| Raspberry Pi 4 · quantized | 6.92 | 107.13 |
| Jetson Nano · quantized | 7.32 | 101.37 |
The small encoder uses less energy in each measured configuration; quantization also lowers energy on the two directly comparable platforms.
Same encoders and device configurations as the latency panel; average over 200 inference runs measured with an external power monitor. Energy in millijoules, lower is better; no dispersion reported.
These are encoder costs under the reported measurement setup, not whole-application battery consumption or continuous listening cost.
| Device and model format | Small (mJ) | Base (mJ) |
|---|---|---|
| Raspberry Pi 4 · standard | 137.82 | 2011.44 |
| Jetson Nano · standard | 133.53 | 1227.24 |
| Pixel 7 · quantized | 3.55 | 39.71 |
| Raspberry Pi 4 · quantized | 4.15 | 71.75 |
| Jetson Nano · quantized | 4.92 | 64.37 |
arXiv:2305.03058v1, 3 May 2023; 18-page public preprint, no appendix. ↗
HF@misc{saeed2023plix,
author = {Saeed, Aaqib and Tsouvalas, Vasileios},
title = {Plug-and-Play Multilingual Few-shot Spoken Words Recognition},
year = {2023},
eprint = {2305.03058},
archivePrefix = {arXiv},
primaryClass = {eess.AS},
url = {https://arxiv.org/abs/2305.03058}
}