 |
Project description |
Here is the description of the project:
 |
Project: Main results |
 |
- State of the art
- Overview: Mutimodal resource construction
- CID: presentation, numerical description, availability, diffusion
- Enriched Orthographic Transcription: description, comparison to other conventions, why E? (justify: transcription vs annotation), evaluations
- Manual Annotations (how to annotate+guide+annotation description, validation): gestures, backchannels, detachments, discourse, disfluences, prosody, laughters, narrative sequences, reported speech
- Automatic Annotations: phonetic, syntax
- Tools: anonymisation, LPL-Phon-Align, SPPAS, annotations-validation-evaluation, TOE-Editor, repetitions, POS-Tagger, morpho-syntax post-editor
- Construction of references corpus
- Formal models
- Overview: TFS
- Each domain is detailed (TFS+description): phonetic, prosody, syntax, discourse, disfluencies, gestures
- Results, exploitation: phonetic, backchannels,
- Knowledge representation and querying of multimodal annotations: pivot language,..., architecture, examples
- Interoperability
 |
Initial work packages |
WP1: Specification of the Multimodal Coding Scheme (MCS)
- Task 1.1. Survey of existing schemes and needs analysis
- Report: State of the art: the coding schemes in multimodality studies
- Task 1.2. Multimodal Coding Scheme specification
- Resource: Specifications of MCS (Multimodal Coding Scheme)
WP2: Annotation environment
- Task 2.1. The phonetic-prosodic domain
- Conventions: Transcription conventions in TOE format
- Tools: IPU segmenter, Converters TOE -> phonological form,
TOE -> pseudo-phonetic form, Help to the automatic
transcription Phoneme-signal aligner, INTSINT annotator
- Task 2.2. The morphosyntactic and syntactic domain
- Resource: Phonetized lexicon, Grammar adapted to the spoken language
- Tools: tagger for speech data (in MCS format), parser for speech data, in MCS format
- Task 2.3. Multimodal level
- Conventions: gesture annotations, edition of technical recommendations
- Task 2.4. Pragmatic level
- Conventions: Annotations
- Tools: adaptation of RST-Tool
- Task 2.5 Conversion, edition and validation tools
- Tools: XML converter ANVIL_to_MCS, XML converter MCS_to_ANVIL
WP3: Queries and manipulation
- Task 3.1 Needs Identification
- Report: Definition of the different kinds of interaction
- Resource: Test set of queries (expressed in natural language)
- Task 3.2 Definition of a query tool
- Resources: A set of operators extending XQuery
- Tools: query language, data description and manipulation, data storage system.
- Task 3.3 Exploitation of the query tools
- Reports: Study of backchannels in the test corpus, report of the
validation of the query language tool
WP4: Validation
- Task 4.1. Additional annotations of existing multimodal corpora
- Resources: Full annotation of the CID, the MapTask, the RFC corpora
- Task 4.2. Collection and annotation of new multimodal corpora
- Resources: Annotated multimodal corpus (narration, phone conversations,
pathological communication)
