Narrative Generation (S:t Andrews)

Narrative Generation

a case study in assistive technology

"I know what it is like to be fed potatoes all my life. After all, potatoes are a good basic food for everyday, easy to fix in many different ways. I hate potatoes! But then, who knew that but me? I know what it is like to be dressed in red and blues when my favorite colors are mint greens, lemon yellows, and pinks. I mean really, can you imagine?"

• Personal communication
• Communication rates
• More dialogue

Natural Language Generation

• Utterance-based
• Automatically generated content

Input

• RFID: Objects, people, locations
• Multimedia: Photos, audio
• External: Timetable, images

• user model e.g. stage
• timetable
• Unexpected events
• "Inherent interestingness"

09:34:00, Voice Recording, A man came to talk to me in gym. I signed an important document.

09:36:00, Voice Recording, My dad was in school today.

09:40:00, Location, Gym.

09:40:00, Object, Skittles

09:40:00, Object, Baton

09:40:00, Person, Mrs Roberts

10:48:00, Voice Recording, I was doing relay racing at school and I joined in really well.

10:48:00, Location, Changing

10:50:00, Voice Recording, I did some high jump and every time I jumped I had a lie down on the mat.

11:08:00, Location, Classroom

11:12:00, Object, Blackboard

11:31:00, Person, Mary

11:36:00, Location, Tutorial Room

11:36:00, Object, Money

11:39:00, Object, Monkey Game

11:58:00, Location, Classroom

12:00:00, Location, Outside

12:08:00, Object, Bike

12:18:00, Location, Classroom

12:29:00, Voice Recording, I asked Jim this morning what he did for his weekend.

12:30:00, Location, Dining Room.

Analysis

Event boundaries

Location-based clutering

Time-based clustering

• Hidding
• Selection
• Embellishments

Then I went to Junior Primary instead of Reading.

- Right! No reading?! Junior Primary?I wonder why that was.

A visitor was there.

- Oh, a visitor, right. I wonder what the visitor was doing?

The Dental Hygienist came to give a talk.”

- Oh! Dental hygienist.

A visitor was there.

- That was the visitor. Okay. That’s why you went to junior primary, mmh.

What did you think of the talk?

- She is nice.

Screenshots

Where to next...

• Increased connectivity
• Increased portability
• Privacy challenges

• Audio analysis
• Social signal processing
• Topic progression
• More domains...

• The children, their parents, and school staff, for their time, help and support.
• RCUK Digital Economy Programme and EPSRC under grants, EP/F067151/1, EP/F066880/1, EP/E011764/1, EP/H022376/1, and EP/H022570/1
• Mobile development: Steven Knox and Alan Clelland
• GUI development: James Christie

I can manage up to 15 words a minute.

Communication Aids

and

Input -> Data analysis -> Editing & narration -> Speech

Voice-recording based clustering

09:34:00, Voice Recording, A man came to talk to me in gym. I signed an important document.

09:36:00, Voice Recording, My dad was in school today.

09:40:00, Location, Gym.

09:40:00, Person, Mrs Roberts

09:40:00, Object, Skittles

09:40:00, Object, Baton

10:48:00, Voice Recording, I was doing relay racing at school and I joined in really well.

10:50:00, Voice Recording, I did some high jump and every time I jumped I had a lie down on the mat.

10:48:00, Location, Changing

12:29:00, Voice Recording, I asked Jim this morning what he did for his weekend.

11:58:00, Location, Classroom

A man came to talk to me in gym. I signed an important document.

This happened at Gym. Mrs Roberts was there. I played with Skittles and Baton. My dad was in school today.

I was doing relay racing at school and I joined in really well. I did some high jump and every time I jumped I had a lie down on the mat. This happened at Changing.

I asked Jim this morning what he did for his weekend. This happened at my home classroom.

Sensors

Location

09:34:00, Voice Recording, A man came to talk to me in gym. I signed an important document.

09:36:00, Voice Recording, My dad was in school today.

09:40:00, Location, Gym.

09:40:00, Object, Skittles

09:40:00, Object, Baton

09:40:00, Person, Mrs Roberts

10:48:00, Voice Recording, I was doing relay racing at school and I joined in really well.

10:48:00, Location, Changing

10:50:00, Voice Recording, I did some high jump and every time I jumped I had a lie down on the mat.

11:08:00, Location, Classroom

11:12:00, Object, Blackboard

11:31:00, Person, Mary

11:36:00, Location, Tutorial Room

11:36:00, Object, Money

11:39:00, Object, Monkey Game

12:00:00, Location, Outside

12:08:00, Object, Bike

12:18:00, Location, Classroom

12:29:00, Voice Recording, I asked Jim this morning what he did for his weekend.

A man came to talk to me in gym. I signed an important document.

My dad was in school today.

I played with Skittles and Baton.

Mrs Roberts was there. I was doing relay racing at school and I joined in really well. This happened at the Gym.

I did some high jump and every time I jumped I had a lie down on the mat. This happened at the Changing Room.

I worked with the Blackboard. This happened at my home classroom.

I played with Money and Monkey Game. This happened at a Tutorial Room. This happened at my home classroom.

I played with the Indoor Bike.

This happened outside.

I asked Jim this morning what he did for his weekend. This happened at my home classroom.

Domain knowledge

Comparison

A man came to talk to me in gym. I signed an important document.

This happened at Gym. Mrs Roberts was there. I played with Skittles and Baton. My dad was in school today.

I was doing relay racing at school and I joined in really well. I did some high jump and every time I jumped I had a lie down on the mat. This happened at Changing.

A man came to talk to me in gym. I signed an important document.

My dad was in school today.

I played with Skittles and Baton.

Mrs Roberts was there. I was doing relay racing at school and I joined in really well. This happened at the Gym.

I did some high jump and every time I jumped I had a lie down on the mat. This happened at the Changing Room.

I worked with the Blackboard. This happened at my home classroom.

I played with Money and Monkey Game. This happened at a Tutorial Room. This happened at my home classroom.

I played with the Indoor Bike.

This happened outside.

I asked Jim this morning what he did for his weekend. This happened at my home classroom.

I asked Jim this morning what he did for his weekend. This happened at my home classroom.

Editing

Microplanning

Simple language desirable

Choose discourse markers, referring expressions, time expressions

Map concepts words:

Smiley -> “he was nice”, “it was fun”

Lexical variation

Realization using simpleNLG

• Locations
• Diversity
• Evaluation
• Multi-device
• Data independance
• Staff independance

• Other practical issues

Stage 3

Stage 2

Stage 1

Questions?

Acknowledgements

SELECTED PUBLICATIONS

R. Black, A. Waller, N. Tintarev, E. Reiter, and J. Reddington.

A mobile phone based personal narrative system. In ACM SIGACCESS Conference on Computers and Accessibility (to appear).

J. Reddington and N. Tintarev. Automatically Generating Stories from Sensor Data. Intelligent User Interfaces (IUI'11).

R. Black, J. Reddington, E. Reiter, N. Tintarev and A. Waller. Using NLG and Sensors to Support Personal Narrative for Children with Complex Communication Needs. Workshop on Speech and Language Processing for Assistive Technologies in association with NAACL-HLT'10 (A demo proposal for this work was also accepted)

How Was School Today....?

In the Wild

Nava Tintarev, Joe Reddington and Ehud Reiter

University of Aberdeen

Rolf Black and Annalu Waller

University of Dundee