IBERSPEECH 2020 EVALUATION CHALLENGES
Valladolid, November 18-20, 2020, march 2021
IberSPEECH’2020 will be held in Valladolid (Spain), from 18 to 20 November 2020. Due to the evolution of the COVID-19 pandemic, the organization committee of the IberSPEECH'2020 jointly with the Spanish Thematic Network on Speech Technologies have decided to move the IberSPEECH event to MARCH, 2021. The exact dates will be announced in brief.
The IberSPEECH event –the fourth of its kind using this name– brings together the X Jornadas en Tecnologías del Habla and the V Iberian SLTech Workshop events.
We are glad to announce that IberSPEECH'2020 will hold two evaluation challeges, the traditional Albayzín evaluation organized by the Spanish Thematic Network on Speech Technologies and the ALLIES evaluation organized by the ALLIES european project.
As the IberSPEECH'2020 event has been moved to MARCH'2021, the organizers of the Albayzin Evaluations and the RTVE sponsor have decided to extend the registration deadline until September 7th. The evaluation data will be released by September 7th. Please see the new dates at the bottom of this page.
ALBAYZIN EVALUATION CHALLENGE
The Albayzín evaluation challenge focuses on evaluating different speech technologies over TV and radio broadcast, and Basque Parliament content.
Five evaluations are proposed:
- Speech to Text Challenge (S2TC), organized by RTVE and Universidad de Zaragoza, consists of automatically transcribe different types of TV shows. More information and evaluation plans ...
- Speaker Diarization and Identity Asignement (SDIAC), organized by RTVE and Universidad de Zaragoza, consists of segmenting broadcast audio documents according to different speakers, linking those segments which originate from the same speaker and identify a closed set of speakers. More information and evaluation plans ...
- Multimodal Diarization and Scene Description Challenge (MDSDC), organized by RTVE and Universidad de Zaragoza, consists of segmenting broadcast audio-visual documents according to a closed set of speakers, faces and scene descriptors and linking those segments which originate from the same speaker, face and scene descriptor. More information and evaluation plans ...
- Search on Speech Challenge (SoSC), organized by Universidad San Pablo-CEU and AuDIaS from Universidad Autónoma de Madrid, consists of searching in audio content a list of terms/queries. More information and evaluation plans ...
-
Text-to-Speech Alignment Evaluation (S2TAC), this evaluation has been cancelled.
The organization will release a corpus consisting of more than 600 hours of TV shows from Radio Televisión Española (RTVE), more than 80 hours of speech from 3/24 TV channel (Corporació Catalana de Mitjans Audiovisuals, CCMA) and more than 20 hours of speech from Aragón Radio (Corporación Aragonesa de Radio y Televisión, CARTV).
RTVE Databases License
The data is available subject to the terms of a licence agreement with the RTVE.
To download the RTVE databases, a representative of your research group, company,..., must sign the license agreement (Digital signature is valid)
Online registration:
The calendar for the Albayzín S2TC, SDC, MDC and SoSC evaluations is:
- March 23th, 2020: Registration opens
- March 23th, 2020: Release of the training and development data
June 1st, 2020 September 7th, 2020: Registration deadline. Release of the evaluation dataJune 30th, 2020 October 9th, 2020: Submission deadlineJuly 15th, 2020 October 30th, 2020: Results distribution to the participantsSeptember 13, 2020 December 23th, 2020: Paper submission deadline
November 18-20, 2020 March, 2021: Iberspeech 2020 conference in Valladolid
The calendar for the Albayzín S2TAC evaluation is:
- May 4th, 2020 Registration open.
May 14th, 2020 June 15th, 2020: Evaluation plan released.
May 18th, 2020 June 15th, 2020: Development data released.
Scoring website enabled for dev data.July 13th, 2020 September 7th, 2020: Test data released.
Registration closed.July 24th, 2020 October 9th, 2020: Deadline for submitting results.July 27th, 2020 October 14th, 2020: Scoring website enabled for test data.September 13, 2020 December 23th, 2020: Paper submission deadline
November 18-20, 2020 March, 2021: Iberspeech 2020 conference in Valladolid
The ALLIES evaluation is coordinated by the LNE (Laboratoire national de métrologie et d’essais, France), LIUM (Le Mans Université, France), IDIAP (Switzerland) and UPC (Spain) inside the ALLIES project.
The ALLIES project aims at catalysing the development of autonomous lifelong intelligent systems by providing the community with scenarios, evaluation plans and metrics to evaluate those systems.
The ALLIES corpus consists of a new corpus of audio-visual documents (news, debates, talk show.) including more than 200 hours of TV shows from the French channel LCP.
The ALLIES evaluation focuses on freeing speaker diarization systems from the need of machine learning expert interventions upon two tasks:
- Diarization Across time: automatic systems use the stream of incoming data to update their knowledge and adapt to new data in order to sustain performance across time
- Lifelong learning speaker diarization systems are evaluated while processing a sequence of incoming audio documents with or without human-assisted learning including active learning (system initiative) or interactive learning (user initiative)
Details are provided in the evaluation plan
https://lium.univ-lemans.fr/allies-evaluation/
Registration:
Send an email to
lifelong-speaker-evaluation@univ-lemans.fr and specify
- the task(s) you're willing to participate
- the name of your team, that can be the name of your organization or any anonymous identity
The schedule for the ALLIES evaluation is:
- From now until 31st of March: Registration for participants is open
- 1st of March: Release of Development data
- 1st of March: Beat platform is open for development
- 1st of June: Evaluation data is released
- 30th of June: Final submission
- End of July: Paper submission deadline
- November: Iberspeech 2020 conference in Valladolid
For any additional information, please contact the organizers of the calls.
The ALBAYZIN 2020 Evaluations Organizing Committee
| Eduardo Lleida Solano, lleida@unizar.es, Universidad de Zaragoza, Spain |
| Alfonso Ortega Giménez, ortega@unizar.es, Universidad de Zaragoza, Spain |
| Luis Javier Rodríguez Fuentes, luisjavier.rodriguez@ehu.es, Universidad del País Vasco, Spain |
