Recognize a new spoken word from a few examples
A new voice command should need examples, not a new training run. PLiX learns an acoustic embedding across languages, then recognizes unseen words by comparing a query with prototypes built from a few labeled support clips.
Multilingual episodic training learns to compare words across 20 languages.
Support-derived prototypes let a frozen encoder recognize a new vocabulary without retraining.
Transfer and hardware evaluation examine unseen languages, language identification and encoder latency/energy.
As technology advances and digital devices become prevalent, seamless human-machine communication is increasingly gaining significance. The growing adoption of mobile, wearable, and other Internet of Things (IoT) devices has changed how we interact with these smart devices, making accurate spoken words recognition a crucial component for effective interaction. However, building robust spoken words detection system that can handle novel keywords remains challenging, especially for low-resource languages with limited training data. Here, we propose PLiX, a multilingual and plug-and-play keyword spotting system that leverages few-shot learning to harness massive real-world data and enable the recognition of unseen spoken words at test-time. Our few-shot deep models are learned with millions of one-second audio clips across 20 languages, achieving state-of-the-art performance while being highly efficient. Extensive evaluations show that PLiX can generalize to novel spoken words given as few as just one support example and performs well on unseen languages out of the box. We release models and inference code to serve as a foundation for future research and voice-enabled user interface development for emerging devices.
Read more ↗The multilingual base uses EfficientNet-v2 Medium, trained in 10-way, 5-shot episodes across 20 languages. Training and test vocabularies contain disjoint words.
§2.2–3.3. The TinyNet-E alternatives are separately trained language-specific models, not a compressed copy of the multilingual base.
For a requested word c, encode each labeled support clip and average its embedding. The query is excluded from this computation.
§2.1.2, p. 3; Figure 1 explicitly separates the query path from mean pooling.
Choose the prototype closest to the query embedding in squared Euclidean distance. Changing support clips changes the classifier while leaving encoder weights fixed.
§2.1.2, expressed equivalently to the paper’s argmax of negative squared distance. Most word tests use one-second clips.
Recognition tests vary vocabulary size, support count and language. Hardware profiling measures encoder inference only on Raspberry Pi 4, Jetson Nano and Pixel 7.
Table 1 excludes log-mel extraction and prototype computation. Released weights and inference instructions are linked through aaqibsaeed/plixkws on Hugging Face.
Encoder only · 200 inference runs · excludes preprocessing and prototype classification
Encoder energy · 200 inference runs · external power monitor
| Device · quantized | Small (mJ) | Base (mJ) |
|---|---|---|
| Pixel 7 · quantized | 3.55 | 39.71 |
| Raspberry Pi 4 · quantized | 4.15 | 71.75 |
| Jetson Nano · quantized | 4.92 | 64.37 |
HF@misc{saeed2023plix,
author = {Saeed, Aaqib and Tsouvalas, Vasileios},
title = {Plug-and-Play Multilingual Few-shot Spoken Words Recognition},
year = {2023},
eprint = {2305.03058},
archivePrefix = {arXiv},
primaryClass = {eess.AS},
url = {https://arxiv.org/abs/2305.03058}
}