Introducing 

Prezi AI.

Your new presentation assistant.

Refine, enhance, and tailor your content, source relevant images, and edit visuals quicker than ever before.

Loading…
Transcript

Speech-Only Conversational Search

Conversational Search over a

Speech-Only Communication Channel: Challenges and Opportunities

Search over Speech-Only Interfaces:

voice search

A significant problem!

Automatic Speech Recognition has improved substantially!

5.9% Word Error Rate (Oct 2016, Microsoft)

Deep Neural Networks

Connected Cars

Lawrence Cavedon

RealThing, Ltd.

Mark Sanderson

Damiano Spina

bla bla

Bruce Croft

:-)

me

People with Visual Impairment

Johanne Trippas

Ameer Albahem

Illiterate Population

Australian Research Council, Linkage Project (LP130100563)

How can we deliver SERPs over a linear channel effectively?

knowledge graph

context

ranking

diversification

recommendation

query suggestion

Screen readers are ineffectual [Sahib et al. 2012]

Intelligent Assistants with Speech-Only Interfaces

June 16, 2017

UNED

Madrid, Spain

Challenges

References

Interaction

CAIR'17

1st International Workshop on Conversational Approaches to Information Retrieval

Conversational Search: User-System Interactions

User has a broader set of possible actions, that can be performed at any stage of the interaction:

  • "Next" (implicit negative feedback)

Changing Spoken Search Results On-the-fly

  • "More like this": Get similar results (explicit positive feedback)
  • "Less like this": Don't show similar results (explicit negative feedback)

Spoken SERP

  • Why? (conversation with the system)

  • "More about this": Incremental summary

Re-ranking

3

Short Summary

Longer Summary

Title

3

2

4

1

~

Ranking

1

3

https://sites.google.com/view/cair-ws/

Conversational Search: Observational Study

- Iterative feedback for re-ranking

- Negative effect on user satisfaction described by Lee, Teevan and De la Chica [SIGIR'14] disappears due to the linearity of the channel.

bla bla

SEARCH

Speech-only Communication Channel

Information

Need

Retriever

User

query confirmation

query reformulation

SERP presentation/synthesis

...

Trippas, J.R., Cavedon, L., Spina, D., Sanderson M., How Do People Interact in Conversational Speech-Only Search Tasks: A Preliminary Analysis. CHIIR'17 (short paper). 2017.

Trippas, J.R., Cavedon, L., Spina, D., Sanderson, M., A Conversational Search Transcription Protocol and Analysis. CAIR SIGIR'17 workshop. 2017

Johanne R. Trippas

PhD Candidate

[Azzopardi, SIGIR'14] Azzopardi, L. Modeling Interaction with Economic Models of Search, SIGIR'14, 2014.

[Kiseleva et al., SIGIR'16] Kiseleva, J., Williams, K., Awadallah, A.H., Crook, A.C., Zitouni, I., Anastasakos, T., Predicting User Satisfaction with Intelligent Assistants. SIGIR'16, 2016.

[Lee, Teevan and de la Chica, SIGIR'14] Lee, C-J., Teevan, J. and de la Chica, S. Characterizing Multi-Click Behavior and the Risks and Opportunities of Changing Results during Use, SIGIR'14, 2014.

[Sahib et al., JASIST'12] Sahib, N.G., Tombros, A. and Stockman, T., A comparative analysis of the information seeking behavior of visually impaired and sighted searchers. JASIST, 2012.

[Spina et al., JASIST (to appear)] Spina, D., Trippas, J.R., Cavedon, L. and Sanderson, M.. Extracting Audio Summaries to Support Effective Spoken Document Search. JASIST, To appear.

[Trippas et al., SIGIR'15] Trippas, J.R., Spina, D., Sanderson, M. and Cavedon, L. Towards understanding the impact of length in web search result summaries over a speech-only communication channel. SIGIR'15. 2015

[Trippas et al. CHIIR'17] Trippas, J.R., Cavedon, L., Spina, D., Sanderson, M., How Do People Interact in Conversational Speech-Only Search Tasks: A Preliminary Analysis. CHIIR'17. 2017

[Trippas et al. CAIR'17a] Trippas, J.R., Cavedon, L., Spina, D., Sanderson, M., Crowdsourcing User Preferences and Query Judgments for Speech-Only Search. CAIR SIGIR'17 workshop. 2017

[Trippas et al. CAIR'17b] Trippas, J.R., Cavedon, L., Spina, D., Sanderson, M., A Conversational Search Transcription Protocol and Analysis. CAIR SIGIR'17 workshop. 2017

[Walker et al, EACL'97] Walker, M.A., Litman, D.J., Kamm, C.A. and Abella, A. PARADISE: A framework for evaluating spoken dialogue agents. EACL'97. 1997.

August 11, 2017

Tokyo, Japan

Presentation

What are the properties of a "good" audio summary?

Evaluation

Audio Search Summaries: Factors

[Trippas et al., CAIR'17a]

  • Summary length

Minimize interaction cost,

[Trippas et al., SIGIR'15]

  • Spoken Documents (e.g., podcasts): Impact of ASR

[Spina et al., JASIST (to appear)]

maximize user satisfaction

@CAIRWorkshop

  • Phrase boundaries
  • Query-in-Context: Keyword Highlighting in speech
  • How/when to show source (e.g., URL) or date?
  • 'Listenability'

Evaluation of Conversational Interfaces

PARADISE Framework

PARADISE framework [Walker et al., 2000]

- small set of subjective judgments

- 8-9 questionnaire judgments

- prediction target

- several parameters determined from interaction logs

- linear combination, ML

  • Predicting User Satisfaction with Spoken Dialog Systems
  • PARADISE framework [Walker et al., EACL'97]
  • Automatically assess whether the user is satisfied while interacting with a spoken dialog system
  • Weighted linear combination of dialogue costs: efficiency measures (e.g., number of utterances) + quality measures (e.g., ASR)
  • Predicting User Satisfaction with Intelligent Assistants [Kiseleva et al., SIGIR'16]
  • Target: SAT vs. DSAT search dialogues
  • Features from clicks, touch and voice interactions + Machine Learning

Factors

Variables

Q queries

V # results

S # summaries

A # assessed documents

Ts # time spent to listen summaries

...

C # clusters/sub-topics/aspects

L # labels

H # layers in the clustering hierarchy

  • ASR errors
  • Speech confidence ("Sorry, I didn't understand you...")
  • Confirmation ("Right. Searching for ...")
  • Summary quality
  • Ranking quality
  • [Clustering quality]
  • Interruptions for query suggestion/recommendations
  • Non-verbal cues (e.g., beep for highlighting query terms)
  • Strategy decision: Present Answer vs. Ranking vs. Clustering

Cost Models

[Azzopardi, SIGIR'14]

  • Simulated users to understand trade-off between variables
  • E.g., # layers in the hierarchy vs. # query reformulations

Validate with user studies!

Thank You!

bla bla

Conversational Search over Speech-Only Communication Channels

How to deliver SERPs in a speech-only channel?

knowledge graph

context

ranking

diversification

recommendation

audio: linear channel

query suggestion

SERP

Interaction

  • Change ranking on-the-fly
  • Opportunities for system-user conversations (e.g. explicit feedback: less/more like this)

Presentation

  • What is a good audio summary?
  • Hierarchical presentation of search results/summaries to assist browsing SERPs

Evaluation

  • How to evaluate speech-only interaction effectively?
  • User Satisfaction: Cost models + User studies

damiano.spina@rmit.edu.au

@damiano10

.

Recent research in IR evaluation methodologies enabled the study of more complex properties of IR user-system interaction

time-biased gain

summaries

[Smucker & Clarke, SIGIR'12 (best paper)]

conversational

search

search result browsing

cost models

[Azzopardi, SIGIR'14 (honourable mention)]

sessions

card playing model

[Zhang & Zhai, SIGIR'15; Zhai, SIGIR'15 keynote]

Learn more about creating dynamic, engaging presentations with Prezi