Scientific Computing Project

PL
by Prasant Lokinendi3 views·Updated Apr 29, 2013
Powered by AI

Need to present on this topic?

Create your own polished presentation in minutes. Just describe what you need.

Free to try, no credit card required

Transcript

An Analogous, Information Preserving, Spatial

Transformation From Audio To Image

A Presentation by: Beller, Chu, Iwuchukwu, Lokinendi

Background

Motivation

File Formats Used

  • Why would anyone do this?
  • Audio Professionals (Mixing/Mastering Engineers, DJs, etc.)
  • Forensic Analysis
  • Image Processing Tools/Methods for Audio Processing (for free or for cheap)

Table from BMP Specification

BMP File Format

Table from WAVE Specification

WAVE File Format

  • One of the most popular uncompressed image file formats
  • Many bitmap images can be considerably compressed using lossless data compression algorithms because of redundant data.
  • The most popular format to store uncompressed audio
  • Stores 44100 samples per second of audio. Each sample is stored as a 16-bit integer.

Isn't there already a way to do this?

Not really...

Figure from JPEG Specification

JPEG File Format

  • Format that is used for lossy compression of digital images.
  • Typical achieves around 10:1 image compression ratio. Degree of compression can be adjusted.
  • Compression based on discrete cosine transform

These are graphs of digital audio, but the images themselves cannot be manipulated to get a resulting waveform. They're representations, but they're not domain-preserving.

The Idea

  • Design a bijection that translates a range of amplitudes [-32768:32767] to a range of three-dimensional pixel values (R[0:255], G[0:255], B[0,255])
  • Run experiments to determine the extent to which the bijection is domain-preserving.

amplitude = 8096

R:132 G: 205 B:63

References

[1] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Gold-man. PatchMatch: A randomized correspondence algorithm for struc-tural image editing. ACM Transactions on Graphics (Proc. SIG-GRAPH), 28(3), August 2009.

[2] Eric Lee and Jan Borchers. Dimaß:: a technique for audio scrub-bing and skimming using direct manipulation. In Proceedings of the 1stACM workshop on Audio and music computing multimedia, AMCMM'06, pages 107{114, New York, NY, USA, 2006. ACM.

[3] Y. Wexler, E. Shechtman, and M. Irani. Space-time completion ofvideo. Pattern Analysis and Machine Intelligence, IEEE Transactionson, 29(3):463 {476, march 2007.

[4] Richard Boulanger and Victor Lazzarini. The Audio Programming Book. The MIT Press. 2011.

[5] Multimedia Programming Interface and Data Specifications 1.0. IBM & Microsoft. August 1991.

Questions?

What's next?

Experimenting with Audio Processes

Experiments

Transformed BMP image

Transformed Frequency Equalized Image

Transformed BMP image

What is Audio Equalization?

Original Audio File

Conclusions

Audio Frequency Equalization

  • High and low frequencies are analogous in both domains
  • rapid changes in the image correspond to high frequencies in the audio domain, and vice versa
  • All transforms showed this property
  • Find image processing analogues to every audio operation (equalization, dynamic range compression, phase, flange, delay, etc.)
  • Further research texture recognition and explore possible applications in audio forensics.

Experimenting with Compression

Example

Final output WAV File

WAV BMP JPEG BMP WAV

Reconvert to a BMP

Converted to JPEG at PSNR = 95

BMP File After Running Algorithm

Original Song (WAV)

Conclusions

  • Transform is probably not useful for sample-level audio processing
  • Why?
  • Files were visually indistinguishable, but JPEG throws away a lot of information.
  • All of our algorithms performed equally for preserving.

varying compression levels in GIMP (Ranging from 0-100)

As expected, more distortion was observed as we lowered the JPEG quality. For song tracks with vocals, even at lower quality compression levels, song can be distinguished.

Experimenting with Texture Recognition

Naive Attempt at Audio Fingerprinting through Texture Recognition

Segment unidentified audio

Convert to Grayscale

Perform Fourier

Segment Audio

Calculate Euclidean Distance

Convert to Grayscale

Return the audio that has the lowest distance

Perform Fourier

Store coefficients in database

  • The images created by each transform can be considered artificial textures.

Synthetic Textures (produced by our algorithms)

Natural Textures

Algorithms

Beller's Algorithm

Chu's Algorithm

32767

-32767

1111111111111111

1111111111111111

Idea: Distribute empty 8 bits across all three color planes so that images are more visually representative.

R = 11111 = 31, G = 11111 = 31, B = 111111 = 63

R = 111111 = 63, G = 11111 = 63, B = 111111 = 63

10110100||11010001||11001011

Idea: Think of the amplitude as a 16-bit binary string and parse the string into three 8-bit regions and assign one to each color plane.

R: 1101001

G: 11010001

B: 11001011

Iwuchukwu's Algorithm

Lokinendi's Algorithm

  • Goals
  • Domain Preserving
  • Visually Useful Information

R

G

B

Amplitude

0

255

1

-32768

0

254

1

-32767

0

253

1

-32766

...

...

...

...

Take in a signed 16-bit integer from the WAV file

Add 32768 to shift all values to be positive

Take the first 8 bits and represent them as the green value, and take the last 8 bits and represent them as the blue value

The red value is represented as the absolute difference between the two other values. This is strictly for visual purposes.

Cast the three values as unsigned 8-bit integers

1

253

0

32765

  • Methodology
  • Attempt to preserve information by maintaining spatial 'distance'.
  • 44100 = 210 x 210
  • 44100 samples = 1 second of audio
  • Use a look-up table to 'paint' amplitudes into 210 x 210 'blocks'

1

254

0

32766

1

255

0

32767