IBERSPEECH 2024 EVALUATION CHALLENGES

Aveiro, November, 2024

IberSPEECH’2024 will be held in Aveiro (Portugal), from 11 to 13 November 2024.
The IberSPEECH event –the sixth of its kind using this name– brings together the XIII Jornadas en Tecnologías del Habla and the IX Iberian SLTech Workshop events.
We are glad to announce that IberSPEECH'2024 will hold the traditional Albayzín evaluation organized by the Spanish Thematic Network on Speech Technologies.

ALBAYZIN EVALUATION CHALLENGE

The Albayzín evaluation challenge focuses on evaluating different speech technologies.

Five evaluations are proposed:

  • Speech to Text Challenge (S2TC), organized by RTVE and Universidad de Zaragoza, consists of automatically transcribe different types of TV shows. This year will be an optional subset with bilingual content of Spanish and any of the co-official languages (Catalan, Valencian, Galician and Basque). More information and evaluation plans ...

  • Speaker Diarization and Identity Assignment Challenge (SDIAC), organized by RTVE and Universidad de Zaragoza, consists of segmenting broadcast audio documents according to different speakers, linking those segments which originate from the same speaker and, optionally, identify a closed set of speakers. More information and evaluation plans ...

  • Search on Speech Challenge (SoSC), organized by Universidad San Pablo-CEU and Universidad Autónoma de Madrid, consists of finding a list of terms/queries in Spanish audio archives and is divided into two different tasks: Spoken Term Detection (STD) and Query-by-Example Spoken Term Detection (QbE-STD). Data will cover conference/workshop domain, TV shows and a novel domain that consists of a corpus of interviews with people from rural areas covering different dialectal varieties in Spain. More information and evaluation plans ...

  • Bilingual Basque-Spanish Speech to Text Challenge (BBS-S2TC), organized by the University of the Basque Country (UPV/EHU), the proposed task consists of automatically transcribing short segments of speech (ranging from 3 to 10 seconds) extracted from Basque Parliament sessions. Segments may be monolingual (Basque or Spanish) or bilingual (Basque and Spanish, including a code switching event). More information and evaluation plans ...

  • Wake-Up Word Detection Challenge (WUWDC), organized by Telefónica Innovación Digital, this challenge aims to assess the performance of State-of-The-Art Keyword Spotting systems in addressing various industrial needs such as accuracy, inference delay, computational load, and energy efficiency. More information and evaluation plans ...

Participation Channels in the Albayzin Evaluations

There are two ways to participate in the Albayzín evaluations according with the submission type:

  • The first way relies on editing the system description paper following the IberSpeech 2024 paper submission template so that the submitted paper (describing the system/s and the results) will appear in the IberSpeech 2024 proceedings. Moreover, participants will also have the chance to submit an extended version of this paper to a journal. This submission way implies sending one or more representatives to the evaluation workshop, to be held in Aveiro, Portugal as part of IberSpeech 2024 (November 2024).
  • The second way demands a free-format document in which participants describe the submitted system/s along with the results, but this will not appear in the IberSpeech 2024 proceedings. In this case, participants are allowed to present on-line their system/s without physically attending the conference, or send a video to the evaluation organizers explaining their submitted system/s, which will be shown during the evaluation workshop.

DATASETS

In order to participate in the Albayzín evaluations, the participants must download the databases and sign the corresponding licence agreement, if necessary.

RTVE Databases License

This dataset is necessary for the S2TC, SDIAC and SoSC challenges.
The data is available subject to the terms of a licence agreement with the RTVE.
To download the RTVE databases,  a representative of your research group, company,..., must sign the license agreement (Digital signature is valid)

OK Aura Database License

This dataset is necessary for the WUWDC challenge.
The data is available subject to the terms of a licence agreement with TELEFONICA I+D.
To download the OK Aura database,  a representative of your research group, company,..., must sign the license agreement
Please for the WUWDC challenge, read the Personal Data Protection Clause before registering.

ONLINE REGISTRATION:

RESULTS SUBMISSION:

The calendar for the Albayzín evaluations is:
  • May 20th, 2024: Registration opens
  • June 3rd, 2024: Release of training and development data
  • July 31st, 2024: Registration deadline.
  • September 2nd, 2024 Release of evaluation data
  • October 18th, 2024: Submission deadline
  • October 31st, 2024: System results distributed to participants
  • November 12th, 2024: Official results presented publicly and published
  • November 12th, 2024:Iberspeech 2024 Albayzin Evaluations special session in Aveiro

For any additional information, please contact the organizers of the calls.

The ALBAYZIN 2024 Evaluations Organizing Committee

Eduardo Lleida Solano, lleida at unizar.es, Universidad de Zaragoza, Spain
Alfonso Ortega Giménez, ortega at unizar.es, Universidad de Zaragoza, Spain
Javier Tejedor Noguerales, javier.tejedornoguerales at ceu.es, Universidad San Pablo CEU, Spain
Luis Javier Rodríguez Fuentes, luisjavier.rodriguez at ehu.es, Universidad del País Vasco, Spain
Doroteo Torre Toledano, doroteo.torre at uam.es, Universidad Autónoma de Madrid, Spain