Skip to content
Natnael Masresha Zerihun

Speech-Corpus Collection Pipeline

10 Academy · 2022 · Academic project · Team project

A distributed platform for collecting Amharic and Swahili speech data: Kafka streaming, Spark processing, Airflow orchestration, and S3 storage behind a React recording front-end.

  • Kafka
  • Spark
  • Airflow
  • S3
  • FastAPI

Overview

Speech-to-text models need audio paired with text, and low-resource languages lack such corpora. This team project built the collection machinery: prompts and recorded audio flowing through Kafka topics, processed with Spark, orchestrated by Airflow, stored in S3, with FastAPI services and a React front-end for contributors.

Media

Event streaming architecture diagram
Streaming architecture: producers publish prompts and recordings to Kafka, Spark processes the streams, storage feeds dashboards and front-ends.
Recording web interface with microphone button
React front-end where contributors read a prompt and record audio.
Airflow DAG run view of the data pipeline
Airflow grid view of the Kafka pipeline DAG: produce, consume, transform, and load steps.

Resources