Prezi

Present Remotely

Send the link below via email or IM

Copy

Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in the manual

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.

DeleteCancel

Make your likes visible on Facebook?

Connect your Facebook account to Prezi and let your likes appear on your timeline.
You can change this under Settings & Account at any time.

No, thanks

Dissertation: A CONCEPTUAL SMART-PHONE-BASED VIRTUAL REALITY FRAMEWORK TO MONITOR RESPIRATION MOVEMENTS

No description
by Ahmad Abushakra on 17 June 2013

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of Dissertation: A CONCEPTUAL SMART-PHONE-BASED VIRTUAL REALITY FRAMEWORK TO MONITOR RESPIRATION MOVEMENTS

A CONCEPTUAL SMART-PHONE-BASED VIRTUAL REALITY FRAMEWORK TO MONITOR RESPIRATION MOVEMENTS
Basically an alternate world contained inside your PC or Smart phone.

An experience in which a person is surrounded by a three-dimensional computer-generated representation, and is able to move around in the virtual world and see it from different angles, to reach into it, grab it and reshape it.
RESEARCH PROBLEM AND SCOPE

This work focuses on developing novel, efficient and feasible techniques

A.
Classify breathing movements.


B.
Determine the user's lung capacity.


C.
Synchronize the visualization of virtual animations of the lung.


Breathing is one of the most essential bodily functions that humans perform involuntarily.

Nowadays, research on breathing/respiratory functionality has received much attention across the medical fields as well as various cognitive/meditation, and systems/engineering research aspects.
is an attractive means for medical simulations and treatment.
several researches have proven that inducing relaxation responses improves the efficiency of the immune system, while stress retards it.
introducing a virtual environment with real-world dynamic contents to develop an effective tool that can monitor respiratory movements and can assist individuals regulate their breath, appears to be an appealing research application.
RESULTS
Breathing movement Classification
A. Dataset
Ahmad Abushakra

Under the Supervision of: Prof. Miad Faezipour

Committee Members

Prof . Khaled M. Elleithy
Prof. Navarun Gupta
Prof. Prabir Patra
Prof. Saeid Moslehpour
7. CONCLUSION
University of Bridgeport
Computer Science and Engineering Department
Ph.D. Dissertation
Agenda
1. INTRODUCTION
MOTIVATION BEHIND THE RESEARCH
RESEARCH PROBLEM AND SCOPE

2. LITERATURE SURVEY
CONCEPTUAL FRAMEWORK OF A VIRTUAL REALITY THERAPY ENVIRONMENT
BREATHING MOVEMENT CLASSIFICATION
LUNG CAPACITY

5. PRELIMINARY RESULTS

Breathing movement Classification
Lung Capacity

6. THE DEVELOPMENT FRAMEWORK OF VIRTUAL REALITY THERAPY

7.CONCLUSION AND FUTURE WORK

8. LIST OF PUBLICATION
4. RESEARCH PLAN

4.1. PROPOSED CONCEPTUAL FRAMEWORK

4.2. VRT FRAMEWORK COMPONENTS
A. BREATHING MOVEMENT CLASSIFICATION
B. LUNG CAPACITY ESTIMATION
1. INTRODUCTION
In this dissertation, novel conceptual virtual reality framework is presented that monitors breathing movements in real time.
MOTIVATION BEHIND THE RESEARCH

In the recent years, Virtual Reality Therapy (VRT) has been explored by many researchers

VR through simulation has the potential of assisting in breath regulation and post-operative breathing exercises, especially for those with breathing disorders, e.g. lung cancer patients


The interactive immersive features of a virtual environment would provide a rich interactive and contextual setting to support experiential and active therapy

Nowadays, smart-phone devices have also become increasingly ubiquitous

The usabilit, especially the effectiveness and acceptance of smart-phone applications by user has become an appealing matter.

As virtual reality using smart-phones has also evolved rapidly in the scale and scope of developing virtual reality applications for medical purposes, applying virtual reality to help breathing disorders patients via smart-phones is a big challenge.

3.POTENTIAL CONTRIBUTIONS OF THE PROPOSED RESEARCH

A.
Conceptual Framework of a Virtual Reality Framework to Monitor Breathing Movements
B.
Breathing Movement Classification






C.
Lung Capacity
This work describes the conceptual framework of a virtual reality environment that monitors breathing movements in real-time and can eventually aid in breath regulation by motivating users to perform breathing exercises through computer aided visions to increase oxygen intake of their blood.

The individual's acoustic signal of breath is used as an interfacing signal between the user and the smart-phone-based VR framework

The framework analyzes the user's breath in real-time and provides virtually real animations of the lungs as the inhalation and exhalation take place.

The lung capacity is computed simultaneously as the user is breathing and the user is encouraged to take the next coming breath deeply if the previous one was not sufficient.

The framework animations also show the lung organs virtually as the breath is being regulated. The overall VR framework .
We employ Mel-Frequency Cepstral Coefficients (MFCC) to the speech/voice segments of the acoustic signal of respiration to depict the differences between the inhale and the exhale in frequency domain.

MFCC has the ability to fully capture the characteristics of the channel spectrum and simulate the human’s auditory function, whose approximation of speech is linearly spaced in frequency scale. MFCCs are based on the known variation of the human ear’s critical bandwidths with frequency.

Proposed technique is based on segmenting the breathing cycles into speech and silence phases and computing the MFCC features of the speech/voice segments. For segmentation purposes, we apply the Voice Activity Detection (VAD) technique as one of the most important functions for silence and speech detection to the acoustic signal of breath. After segmentation, 13 MFCC features of each voiced segment are computed.

The 6th MFCC clearly shows distinct properties among other MFCC features, in which it has been used to differentiate the inhale and exhale durations by employing a linear thresholding function comparison.

Accurate differentiation between inhale and exhale will lead to a more reliable and precise modeling of the lung functionality during the breathing process in a virtual environment, which will eventually achieve the goal of motivating users/patients to regulate their breath.
Use microphone features to depict the lung capacity via the time period of the signal and the energy of the signal during the breathing cycles.

Our technique is based on segmenting the breathing cycle via speech and silence detection and computing the start and end of each speech cycle in order to determine the time duration and the signal energy for each breathing cycle.

The automated lung capacity computation technique proposed in this part of our intended virtually reality platform to be implemented on a smart-phone to assist lung cancer patients.
2. LITERATURE SURVEY
CONCEPTUAL FRAMEWORK OF A VIRTUAL REALITY THERAPY ENVIRONMENT
Virtual reality is
Breathing regulation therapy is an effective way to assist those with breathing disorders such as lung cancer patients, and patients with asthma.

Virtual reality using smart-phones has also evolved rapidly in the scale and scope of developing virtual reality applications for medical purposes

Applying virtual reality technology to help users via smart-phones is a big challenge
CONCEPTUAL FRAMEWORK OF A VIRTUAL REALITY THERAPY ENVIRONMENT Cont.
Interestingly enough, virtual therapy has proven to increase the chance of survivability of patients with cancer by more than 56% .

The system offers a reason for users to relax while instantly visualizing their immune systems fighting against their diseases, White and red blood cells and malignant cells populate the virtual environment
4. RESEARCH ORGANIZATION
PROPOSED CONCEPTUAL FRAMEWORK
VR FRAMEWORK COMPONENTS

Participants of this study are 123 students from the University of Bridgeport.

Each volunteer was asked to breathe ten cycles in a noise free environment.

The recording of all speech samples took place at the same location for all subjects to provide uniformity.

The microphone was placed approximately 3 cm away from speakers.

All samples were recorded with a sampling frequency of 44.1 KHz. Portions from all volunteers were then copied into MATLAB for normalization and processing.
BREATHING MOVEMENT CLASSIFICATION
Segmentation
Mel-Frequency Analysis
Linear Thresholding
Classification Procedure
LUNG CAPACITY ESTIMATION
Segmentation
Important
Metrics and Factors
Procedure
The following steps show the lung capacity measurement procedure in our work.

First, the recorded signal for each speaker is split into inhales and exhales (speech segments). The splitting process was implemented using the VAD technique.

Second, the start and end point of each speech segment (inhale and exhale duration) is marked.

Third, the time duration between each start and end of speech segments (inhale and exhale) are computed .

Fourth, the energy of the signal between each speech segment (considering all samples within the speech segments) is calculated (Equation 3).
Finally, the lung capacity is computed using the following Equations by substituting the five important factors (gender, age, height, breathing time, and energy).

Estimated lung capacity:
The following metrics are used in computing the lung capacity in our work:
Gender and adult state
Age in years
Height in inches
Duration of which the subject blows air in the microphone
Signal energy: is the amplitude of the acoustic speech signal through time.
The Story begin when a group of researcher in MIT start a program called Staying Alive for cancer patients
Their work is based on the fact that diversion from unwanted feelings such as pain and stress is one of the most effective therapy techniques in dealing with such feelings/diseases.
This work present the development and design of a breathing interface and video game to promote compliance with post operative breathing exercises. An interactive spirometer was introduced to motivate patients to perform post-operative breathing exercises. The work mostly described the early progress of an interaction device, game design, initial play testing and the usability of the game.
In 2006, a post traumatic stress disorder VR therapy was developed with the help of the U.S. Army-funded combat tactical simulation scenario along with the commercially successful X-Box tool for virtual reality exposure therapy Application in the Iraq war .
A virtual reality therapy system was introduced for the treatment of acrophobia and therapeutic cases with the objective of developing an affordable and more realistic virtual environment to perform visual therapy for acrophobia. The visualization was a PC-based framework using a virtual scene of a bunge-jump tower in the middle of a big city. The overall system has proven that the VR therapy environments are successful, realistic and fascinating.
A Virtual Reality Claustrophobia Therapy System treatment of a wide collection of psychological disorders was discussed.




A virtual reality stress therapy web application that was based on image processing/analysis was presented .
Breathing Movement Classification
The breathing cycle is divided into four different phases: inspiratory phase, inspiratory pause, expiratory phase and expiratory pause
One approach is to determine the average energy of the signal in time domain
Voiced and un-voiced speech detection techniques have also been helpful
Identify the maximum amplitude the signal reaches combined with the number of maximum peaks. The highest amplitude indicates the expiration phase
Fine-tuning technique



This technique is hard to implement and is also unreliable, since it requires continuous modifications for the marking coefficients regarding the breath signal





Another segmentation method has been implemented by calculating the average energy of the signal per phase time, This technique is a more reliable approach, since it depends on the whole period of the breathing cycle.
Techniques introduced to differentiate inhale from the exhale.
Breathing Movement Classification
Lung Capacity
Plethysmograph is an equipment for determining changes in volume within the body system

Spirometers are the most widely used devices for such daily measurements, which also provide medical records for the patient.
Traditional spirometer and the The electronic spirometer
Gas flow sensor based on the sound generated by turbulence has been developed as a way to detect the breathing cycle phases using the microphone to capture the flow rate. This work mostly described the hardware and signal processing circuitry involved in the sensor design. The main drawback of this design is ambient noise and vibration, which would require certain filter designs to eliminate the noise. As will be seen, our work is intended to be designed with minimal or even no additional hardware cost to the smart-phone.
The design of this spirometer device was customized for special use, and could be connected to a smart-phone win android application for monitoring the patient spirometry readings, a separate hardware including a pressure sensor was designed to interact with the smart-phone API. Hence, the patient is required to also purchase the peripherals which costs more in addition to the fact that the patient would need to deal with more devices and settings for self monitoring.
Exhale detection by maximum amplitude
Exhale detection by average energy.

Acoustic
Signal
Age
Gender
Height

Building a Virtual
Reality environment

Inputs
The VAD algorithm is adopted by first filtering the signal to remove the undesired low-frequency components, and second, calculating the power with different window sizes of the Fast Fourier Transform (FFT) of the input signal
MFCC has 13 levels of Mel-frequency as a human ear perception. Our technique is based on segmenting the breathing cycles and computing the MFCC features of each speech segment. In our analysis, the 6th MFCC is observed to have certain characteristics, as explained hereafter.

• The relation between the amplitude of the frequencies inside this range and the speed of the speech is prominent. By pronouncing the same letter with different pronunciation speeds of pronunciation, the position of the letters with time domain varies accordingly. By plotting the signal in frequency domain, the resonances change. Also, when the speed of sound is higher, the resonances occur at higher frequencies; the second resonance is shifted right off scale.

• The sound of the speech that majorly exists in this spectrum range can be detected. Assuming that each letter of the speech is dominant at a specific spectrum, certain spectrums will be most similar to the inhale pronunciation and the exhale pronunciation, respectively. The pronunciation of “F” is most similar to inhale, and the pronunciation of the exhale is close to the pronunciation of “H”. Hence, the MFCCs are determined and examined for the most variations.
To separate between the speech phase (inhale and exhale) MFCC values come up with an equation inspired from SVM(Support vector machine )

The threshold value is determined by calculating the average (mean) value of the 6th MFCC related to all speech samples.
The threshold level is computed using the following equation:
The following steps show the classification procedure in our work.

First, the recorded signal for each speaker/subject is split into speech and silence segments. The splitting process was implemented using the VAD technique, as described earlier.

Second, the 13-MFCCs for each speech/voiced segment of the same speaker are calculated.

Third, the i-th MFCC of all samples is determined and kept aside from the other MFCCs.

Finally, all the i-th MFCCs that are related to the same speaker’s breathing segments are plotted.
Lung Capacity
B.Setup and Results
The LC model was also applied on a number of speakers/subjects. Each subject’s acoustic signal of respiration was captured using a microphone in ten breathing cycles. The recordings of all speech samples took place at the same location for all subjects to provide uniformity
A. Dataset
The formulations (Equations 2.6-2.7) were used to estimate the lung capacity (FVC) for numerous subjects. The actual lung capacity was also extracted using a traditional spirometer. The spirometer values and the LC model values were plotted in the two graphs of Figure 31. The x axis represents the number of subjects, and the y axis represents the lung size in liter/minutes. As seen from the graphs, the estimated results are very close to the actual spirometer readings, with and overall error of 0.13% for 20 subjects.
B.Setup and Results
Building a virtual reality therapy environment customized for the patients which allows them to virtually visit their lung cells and encouraging them to regulate their breath, can assist them in disposing of the cancer cells.

This appears to be a promising approach for patients to empower their immune system and eventually fight off the disease.

The objective is to aid lung cancer patients regulate their breath by having a daily basis estimate of their lung size using a hand-held device at home.

The promising results of lung capacity and breathing movement classification using a regular microphone motivate us to deploy a microphone already embedded within a hand-held device for investigating on the VRT development.
FUTURE WORK
Investigate on smart phone implementation

Investigate on run time analysis

Investigate on how to synchronize the animations

Extend our data set include Lung cancer patient
3.POTENTIAL CONTRIBUTIONS OF THE PROPOSED RESEARCH
The Developmental Virtual Reality Framework
Comparison of breathing movement classification among various techniques
Virtual Reality Monitors Respiration Movements Framework Applications
Assisting lung cancer by encouraging them to regulate their breath which will increase the oxygen percentage in their blood.
Monitor the physical breathing situation for the a soldiers in the ground war

Entertain the game player during the video game actions and levels
Monitor breathing cycles for athletes
Hypothesis
1. Breathing Movement Classification
The three-step procedure of Voice Activity Detection, Mel-frequency Analysis and Linear Thresholding applied to the acoustic breathing signal yields a high accuracy (mean accuracy >80%) in classifying inhale and exhale phases considering the statistical significance level of 0.05..

2. Lung Capacity Estimation
The time and energy of the breathing phases in the acoustic signal of respiration along with other factors (gender, height and age) can model the lung capacity with a high degree of accuracy (mean accuracy >80%) considering the statistical significance level of 0.05.

3. Integration.
Run-time (real-time) analysis of the breathing movements requires meticulous integration of the above two breathing movement classification and lung capacity estimation modules/components. For this, a visualization component is also added to virtually display the breathing movements analysis synchronized with the actual respiration movements on a screen. Integrating the framework with run-time breathing movement monitoring/visualization capability adds challenging synchronization research issues to the plate.

The hypothesis claim here is that the mean accuracy of the breathing movement classification and lung capacity estimation module in the integrated framework yields a high mean accuracy of greater than 80% considering the statistical significance level of 0.05..
Thank you
University of Bridgeport
Computer Science and Engineering Department
Ph.D. Dissertation
A CONCEPTUAL SMART-PHONE-BASED VIRTUAL REALITY FRAMEWORK TO MONITOR RESPIRATION MOVEMENTS
As lungs distribute oxygen to the entire body through blood flow via blood cells, any form of exercise that regulates the respiratory system, can improve the lung functionality in providing oxygen to the rest of the body, and as a result, can help diminish breathing/lung disorder symptoms.
Virtual Reality Technology
This research proposes a conceptual framework of a virtual reality environment that deals with analyzing acoustic breathing signals in real-time to monitor respiration movements.

Full-fledge implementation and testing the entire framework on actual patients is not the objective of this research, rather, introducing the underlying components of this conceptual framework is the goal.

In order words, this work suggests that if such a framework were to be fully implemented, the proposed components should be in place in order to have it functional.

Assisting in breath regulation is one of the direct applications of this framework, not the goal of this research itself.

Such application requires clinical trial on lung cancer patients or those with breathing disorders for a long period of time, which is beyond the scope of this dissertation.
With this introduction, the hypothesis of this work is stated as follows:
Visualization
IMPLEMENTATION AND TEST PLAN
Flow of Events in the VRT
The flow of events in each breathing exercise session is described as follows:

(S0) The Start state is the initial state and the beginning of the breathing exercise session. The initial Lung Volume (LV) is initialized to zero (LV = 0) in this state.

(S1) In the Capture state, the acoustic signal of respiration of the patient is being captured via a microphone (which could also be smart-phone embedded microphone).

(S2) The Classification state is the state where the breathing movements are classified into inhale or exhale.

(S3) Then , in the Lung Capacity state, the amount of air that inhaled or exhaled to and from the lungs is computed. This value is denoted as LVnew.

(S4) In the Counter state, the accumulating amount of oxygen represented in the lung is computed from adding up the lung volume values from the beginning of the breathing exercise session; LV = LV + LVnew. If the lung volume has not yet reached the limit (which can be predefined in each breathing exercise session), the system will return to state (S1); the Capture state, otherwise it will go to the End state.

(S5) The End state is the where the breathing exercise session terminates.
VRT Framework Screenshot
Screenshot of the lung capacity parameters
for each breathing exercise cycle in the VRT app
User Information
Lungs before starting the
breathing exercise session
Lungs while performing
the breathing exercise session

Lungs in maximum size
at the end of the session

The list of parameters calculated during the patient's breathing cycle while the system is capturing the acoustic signal of the patient's breath through the smart-phone microphone are as follows:

Current lung size: the air volume of the current breathing cycle (S3)

Lung capacity: This is the desirable air volume that the patient would like to achieve in the breathing exercise session. It is normally set to a predefined default value based on the user’s goal or patient's cancer severity stage.

Total lung capacity: the accumulated value of the lung volume (S4).

Lung capacity in each breathing cycle: This shows the lung volume computed in each breathing cycle of the session separated by commas.

An Interactive Scenario of the VRT framework
An interactive scenario of the VRT framework
VRT Framework Implementation
Software
Microsoft’s Windows Mobile platform (Windows Phone 7.1)

XNA framework is widely used for the development of computer games to be deployed for smart-phone applications or to be run as a Windows .NET application

windows phone emulator on a standard PC .

XNA provides a basic audio and graphic API, sound processing and display and I/O device handling in comparison with other tools.

The system is implemented in C# using the Visual Studio Express 2010 Integrated development environment (IDE).
Hardware
The Smart-phone Breath Regulation Virtual Therapy Framework has been initially implemented on an affordable PC.

The system used is a Windows 7 (x86) PC with Service Pack 2, 4GB of free disk space on the system drive, and 3GB RAM.

Windows Phone Emulator requires a DirectX 10 or above capable graphics card.

The entire system implemented using the Windows phone emulator on a PC is then ready to be installed on a smart-phone.
Virtual Reality Environment
1. Breathing movement classification component.



2. Lung capacity estimation component.



3.Synchronized visualization component.



4. Low Frequency Filter


The low frequency low-pass filter component can be added to the framework to enhance the pre-processing phase. This component would accurately identify the acoustic signal coming from the microphone when the user blows into the microphone
VRT Framework Implementation
Monitoring breath and identifying breathing movements via a microphone to detect and classify breathing movements is the goal of this component. As mentioned earlier, the Mel-Frequency Cepstral Coefficients (MFCCs) along with speech segmentation techniques using Voice Activity Detection (VAD) and linear thresholding are applied to the acoustic signal of breath captured using a microphone to depict the differences between inhale and exhale in frequency domain
The lung capacity estimation component gives an estimate of the air-volume entering/exiting the lungs using the acoustic signal of respiration. The time and signal energy of the breathing phases in the acoustic signal of respiration along with other factors (gender, height and age) can model the lung capacity. The methodology is supported by the mathematical model of lung capacity and highly accurate results achieved in the experiments
The visualization component displays virtually-real high definition (HD) animations of the lungs in real-time that show the inflation and deflation of the lungs as the user/patients breathes. These animations inspire the patient to regulate his/her breath cognitively. This HD model for the visualization component ensures high-quality animations which allow the patient experience the VRT as if it were for real
VRT Framework Integration
Automatic Identification
Voiced/ Unvoiced Decision
Integrated Framework
See the full transcript