Loading presentation...

Present Remotely

Send the link below via email or IM

Copy

Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in our knowledge base article

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.

DeleteCancel

Make your likes visible on Facebook?

Connect your Facebook account to Prezi and let your likes appear on your timeline.
You can change this under Settings & Account at any time.

No, thanks

BRAMS14Fev2014

Presentation, International Laboratory for Brain, Music and Sound Research, 14 Feb 2014
by

Jean ROUAT

on 11 April 2016

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of BRAMS14Fev2014

?
?
?
Vision -> Audition
Audition -> Vision
Sensorial Substitution/Completion
Research Platform

Neurosciences
Engineering
We want to use one sense to carry the information to the other sense
It can not be reduced to sounds or images recognition systems
We do not want to overload visual or auditory systems
Perception and cognition have to be taken into account
Brain knowledge is required for a better design and use
It should not be computing intensive and should work without telecommunication access.
Large range of applications: assistive, art and creation technologies, etc.
Objectives
Sensorial Substitution/Completion Research Platform
Jean Rouat
http://gel.usherbrooke.ca/rouat
Neurocomputational and Intelligent Signal Processing Group,
Département de génie électrique et génie informatique
Univ. de Sherbrooke, Québec, Canada
Laboratoire de neurosciences,
Département de Sciences biologiquesUniv. de Montréal, Québec, Canada
http://gel.usherbrooke.ca/necotis
http://smneuroscience.weebly.com
McGill Schulich School of Music
527 Sherbrooke St. West
Montreal, Quebec
http://www.cirmmt.org
Presentation at BRAMS, 14 Feb. 2014
Background and context
Coupling between audition and vision
Coupling between touch and vision
Coupling between touch and audition
The Multisensory Brain
The new handbook of multisensory processes, edited by Barry E. Stein, MIT Press 2012
The McGurk effect, 1976: Vision altering Audition
Hollich & al., Child Development, 2005: Use of Auditory streams to learn a language


Plasticity
Sensory substitution and the human--machine interface, Paul Bach-y-Rita & Stephen W. Kercel, Trends in Cognitive Sciences, 2003
Sensorial Substitution by Bach-y-Rita
ETC.
(large litterature in the field)
Find features and auditory objects to be mapped with visual stimulation
CRSNG, FRQ-NT, Université de Sherbrooke, Université de Montréal & CIRMMT
Acknowledgements
Thanks to the fantastic team in Sherbrooke and Montréal
•Mohammad Adeli, Ph.D. : Substitution from sound to vision, co-dir. : J. Rouat & S. Molotchnikoff
•Vishal Bharmauria, Ph.D. : Étude et modélisation de micro-circuits du système visuel, co-dir.: S. Molotchnikoff & J. Rouat
•Lyes Bachatène, Ph.D. : Étude de la plasticité dans le système visuel, dir.: S. Molotchnikoff
•Simon Brodeur, Msc.A. : Reconnaissance automatique de parole par techniques bio-inspirées, dir: J. Rouat
•Sarah Cattan, Ph.D. : Imagerie Intrinsèque: Changements des cartes corticales par l'adaptation, dir.: S. Molotchnikoff
•Vincent de Ladurantaye, Ph.D. : Reconnaissance automatique d’objets visuels dans les vidéos, dir: J. Rouat
•Yousof Erfani, Ph.D. : Tatouage de signaux sonores à l’aide de techniques parcimonieuses d’analyse, co-dir.: R. Pichevar & J. Rouat
•Sean Wood, Ph.D.: Traitement de l’effet cocktail party à l’aide d’analyses parcimonieuses et bio-inspirées, co-dir: J. Rouat

•Sébastien Gagnon, Msc. A. : Reconnaissance automatique de parole par fenêtre d’analyse à taux variable, dir: J. Rouat
•Guillaume Séguin-Godin, Msc. A. : Implémentation sur FPGA d’un réseau de neurones à décharges implémentant l’appariement temporel, co-dir: F. Mailhot & J. Rouat
•Shadi Siahpoush, Msc.A. : Décodage de décharges neuronales provenant du système auditif, co-dir: E. Plourde & J. Rouat
•Vincent Tremblay-Boucher, Msc. A. : Système de composition automatique de musique et de vidéos à partir de représentations parcimonieuses des signaux, dir: J. Rouat
•Jacques Vanden Abeele, prof. associé, départ. GEGI, Univ. de Sherbrooke, Neurosciences
•Jean Rouat, prof. titulaire, départ. GEGI, Univ. de Sherbrooke & prof. associé, départ. Sc. biologiques, Univ. de Montréal, Traitement des Signaux et Neurosciences
•Frédéric Mailhot, prof. agrégé, départ. GEGI, Univ. de Sherbrooke, Micro-électronique et FPGA
•Eric Plourde, prof. adjoint, départ. GEGI, Univ. de Sherbrooke, Traitement des Signaux et Neurosciences
•Stéphane Molotchnikoff, prof. titulaire, départ. Sc. biologiques, Univ. de Montréal & prof. associé, départ. GEGI, Univ. de Sherbrooke, Neurosciences
•Ramin Pichevar, prof. associé, départ. GEGI, Univ. de Sherbrooke & Chercheur Centre Recherche Communications, Ottawa, Codage de signaux sonores
•Jérémie Voix, prof. agrégé, départ. ME ÉTS, Acoustics
Members or collaborators, professors
Students
Tentative definition of an object
Auditory/Visual objects
What are the structures of the perceived object?
How to identify these structures?
How to set up classification /recognition systems for objects extraction?
Widely open questions that we try to circumvent
Physiology/Psychology to constraint potential models
Signal Processing, Artificial Intelligence, Information theory, Machine Learning
Signal Processing & Machine Learning
Tools
Unsupervised Learning
Automatic Extraction of Structures on Large Data
Object based Analysis and recognition of sound/speech
Signal Processing, Physiological and Psychoacoustical knowledge
http://audio-visual-experiment.espaceweb.usherbrooke.ca
Find features and visual objects to be mapped to auditive stimulation
A Sensorial substitution Platform
Visual scene analysis
Virtual auditory scene
Example of a sensory substitution system
M. Auvray et al., Perception, 2007
Peter Meijer, An Experimental System for Auditory Image Representations, IEEE Tr. on Biom. Eng., 1992
Lescal, Rouat & Voix, ICA2013, 3 June 2013, http://www.gel.usherbrooke.ca/rouat
Image analysis with a pulse neural network
3D Virtual scene synthesis with HRTFs
From visual scenes to auditory scenes
Challenges:
Visual scene analysis
Generation of an auditory scene
To reduce the overload on the auditory system
time t
frequency
of
sinus
0
left
right
Lescal, Rouat & Voix, ICA2013, 3 June 2013, http://www.gel.usherbrooke.ca/rouat
Architecture of the ipod demo
Lescal, Rouat & Voix, ICA2013, 3 June 2013, http://www.gel.usherbrooke.ca/rouat
Pixels of the input images
Neural network features enhancement and extraction
Location of active neurons
Define sub-areas (grid of cells) to be mapped on the image
Analyse the neuron's activities inside each cell and decide for the cell to be active
Maps active cells to 3D sounds (auditory scene)
Lescal, Caron & Rouat, IEEE -ISSPA 2012
Play sounds
Demonstration with a 3 x 4 cells configuration
Azimuth : +90, 30, 330 (-30) and 270 (-90) degrees
Elevation : +45, 0, -40
Set-up and experiments
Object location without vision: Office's bin, chair & pair of shoes;
Object location on a table: Computer mouse, iPhone & coffee mug (30 s to find);
Sound source Azimuth & Elevation location
iTunes store : "SeeDifferently" for iPhones, IOS 7
Vision to Sound Platform
Bottom-up
(Feedforward)
Top-down
(Feedback)
Attention
&
Binding
CONCLUSION
Musical Composition & Creation
Games
Assistive technology (one modality complements the other oner)
Training and Reeducation
Health care
Proof of concept available to anyone on the iTunes Store
Demo
with
ipod 4,
IOS 6
Object Recognition
Binding by synchrony
Our Implementation
Vincent de Ladurantaye, Jean Rouat, "Visual Object Recognition with NN Convolutional & Spiking; Test on Traffic Signs", demo session, Neural Information Processing Systems 2010
Robustness to occlusion
https://vdeladurantaye.espaceweb.usherbrooke.ca/recherche.html
Car Recognition
Robustness to occlusion
https://vdeladurantaye.espaceweb.usherbrooke.ca/recherche.html
Download on iTunes store : "See Different Sherbrooke"
http://www.gel.usherbrooke.ca/necotis/en/APP.html
Conclusion
Sensorial Substitution/Completion is multidisciplinar

I have presented the engineering aspect

Many work to be done around that platform

Room for collaboration
CIRMMT : Music
CIRMMT : Music
Laboratoire Neurosciences
Sciences biologiques
Université de Montréal
Prof. Molotchnikoff
???? Questions ????
The new handbook of multisensory processes, edited by Barry E. Stein, MIT Press 2012
Coupling between audition and vision
Coupling between touch and vision
Coupling between touch and audition
Polysensory: Vision altering Audition
Use of auditory streams to learn a language
Hollich & al., Child Development, 2005
The McGurk effect, 1976
Audio-Visual Speech
Plasticity
Sensory substitution and the human--machine interface, Paul Bach-y-Rita & Stephen W. Kercel, Trends in Cognitive Sciences, 2003
Sensorial Substitution by Bach-y-Rita
The VOice (Meijer, 1992), Capelle (1998), Cronly-Dillon (1999), Durette (2010), etc.
Sensorial substitution systems
Evaluation of substitution systems
Amedi, Merabet, Zatorre, Ptito, etc.
The Multisensory Brain
Quick Background
Move pointer
to play the video
Move pointer
to play video
Keywords: See Different Sherbrooke
Vincent de Ladurantaye, Jean Rouat
Cours gradué GEI723, faculté de génie, Univ. de Sherbrooke
Neurosciences computationnelles et applications en traitements de l'information
Médiatisé, possible de le suivre à distance et à son rythme
À chaque hiver
Full transcript