Whisper Diarization leverages OpenAI's Whisper for Automatic Speech Recognition (ASR) and incorporates Voice Activity Detection (VAD) and speaker embeddings to identify individual speakers within a speech recording. The pipeline begins by separating audio vocals for enhanced speaker embedding accuracy. Whisper is then used to generate a transcription, which is refined using ctc-forced-aligner to correct timestamps. Subsequent processing with MarbleNet and TitaNet facilitates VAD and speaker embedding, respectively. Finally, speaker embeddings are aligned with the corrected timestamps to pinpoint speaker identity for each word, further refined with punctuation models to address minor timing discrepancies.
This project offers a practical solution for speaker identification within speech data, building upon the powerful Whisper ASR model. It distinguishes itself through its integration of VAD and speaker embedding techniques, resulting in more accurate diarization compared to relying solely on ASR output. The use of ctc-forced-aligner for timestamp correction enhances accuracy, and the experimental parallel processing option allows for faster processing on systems with sufficient VRAM. The solution is flexible with command line options for model selection, device specification, and language handling.
- Core Functionality: Performs speaker diarization on audio files using Whisper, VAD, and speaker embeddings.
- Supported Platforms: Primarily designed for use on Linux systems, with instructions for installation on Ubuntu/Debian, Arch Linux, MacOS, and Windows.
- Configuration/Extensibility: Offers command-line arguments for customizing Whisper model, device, language, and batch size.
- Performance/Scalability: Includes an experimental parallel processing option for faster execution on systems with 10GB+ VRAM.
- Developer Experience: Provides clear installation instructions and usage examples with command-line options for flexibility.
- Dependencies: Requires FFmpeg and Cython.
- Output: Produces a diarized transcription with speaker labels associated with each segment.
The project is actively developed, showing recent commits and ongoing issue responses. Documentation is present, and the command-line interface is functional. However, the project acknowledges limitations regarding overlapping speakers and potential errors, indicating that further refinement and bug fixing are ongoing. The project relies heavily on external libraries and is relatively new (created in 2023), it presents a great opportunity for community collaboration and future enhancements.
This project benefits researchers and developers working with audio data who need to identify who is speaking in a recording. It addresses real-world use cases such as meeting transcriptions, call center analysis, and lecture recording processing. Whisper Diarization offers a valuable tool for automatically segmenting audio based on speaker, providing a significant improvement over manual annotation or simpler ASR-based approaches.