Showing posts with label spatialization. Show all posts
Showing posts with label spatialization. Show all posts

Tuesday, July 20, 2010

iSonic project at UMD - papers and videos

The project that led to Zhao's dissertation on sonified choropleth maps is described at the iSonic project home page.  Some software is available there, but I have not tried it and do not know what stage of the project it represents.  Video demonstrations are also available there, and some links to papers.  It seems, however, that the most recent papers from Zhao and Plaisant are not listed there. 

Monday, July 19, 2010

Spatial sound background

I had a good lunch-time talk with Anthony Hornof today regarding spatial audio.  Anthony's lab has a research-level spatialized audio setup, which he will allow us to  use if we find it useful.  Primarily I was interested in what might be feasible with off-the-shelf hardware such as we might be able to deploy in a middle-school classroom.  I wondered whether we might get benefit of spatialized sound with standard stereo speakers rather than headphones.

First, Anthony told me that high-end audiophile headphones typically use an "open air" design that does not shut out other sound in the environment.   A typical cost for a really good pair is around $300.

Anthony was skeptical of being able to achieve usable spatialization with ordinary stereo speakers.  Of the two cues our brains use to judge location of sound, only one of them would be effectively controlled with ordinary speakers --- I'll write a little more about this below.  Second, the software that is used for spatialization has been designed for controlling sound near the ear, and would not effectively utilize speakers.

Anthony sent me several papers on spatialization, but for background he particularly recommended a book by Durand Begault which is available online at  Durand Begault's publication web page  . The book can be downloaded from this link:  3D Sound for Virtual Reality and Multimedia.   Chapter 2 provides a concise summary of sound location perception.


We use two cues to judge the location of a sound.  One is the relative loudness of a sound at each ear, called "interaural intensity differences", or IID.  The other is small time differences between sound arriving at each ear, called "interaural time differences", or ITD.    These  time differences range from about half a millisecond up to 1.5 milliseconds. 

In a natural setting, we not only judge the location of a sound by differences between what is received at each ear, but we also actively enhance those differences ... if we really want to know where a particular sound is coming from, we turn our head to minimize interaural differences, whereupon our head is pointed in the direction of the sound.   When our head is still, it is difficult to distinguish whether a sound is in front or behind us, but turning our head apparently disambiguates it (Begault, page 39).    We cannot duplicate this facility with headphones or speakers, so we probably could at best simulate the location of objects left-to-right or in a half-circle centered in front of the listener.  That is the approach that Zhao, Plaisant, and Shneiderman used in a pilot study of a sonified choropleth map.  (I'll write more about the study of Zhao et al in another post.)

Thursday, July 8, 2010

Spatialization and other notes from July 7 meeting

In our informal meeting yesterday (July 7), one of the topics we spent some time on was spatialization --- sound that seems to be coming from a particular point in space.  Here are some notes on that from memory, and a couple things I didn't think of until after we met.

Our initial discussion was about dealing with short-term memory limitations (we need to carefully limit the number of things a user must perceive and remember while using a geographic display), and how to provide some sort of scan or orienting display that helps a user choose to where to "look".  By analogy, we know that human vision has a small central focus (fovea) that jumps around (saccades) to take little snapshots in close detail (fixations).  The amount of information in each snapshot is small.   The peripheral vision notes potential targets for the next fixation ... it provides very little detail but is good at noticing change or motion. 

On a tactile map, perhaps a quick scan with the dominant hand, using the other hand as a locational anchor, is serving a purpose similar to peripheral vision in sighted users.  What might the equivalent "scan" be for an audio display?  We discussed a couple of scanning strategies.  One approach would be to divide the display into rows and columns, and display objects in grid cells in a fixed order (e.g., left-to-right, top-to-bottom).   A variation is a circular scan of objects in the vicinity of the pointer, using loudness to indicate proximity. 

Robert or Matt suggested that spatialized audio could be used to indicate not just a point location, but also additional spatial information ... the example was hearing a sound moving along some part of a river, indicating not just the nearest point in the river but also its direction.  This seems like potentially a neat answer to a question that Andrew Marcus posed in an earlier meeting, about communicating information like the direction of river flow.  One can imagine some indication (perhaps a rising and falling loudness to indicate proximity, or a distinct sound signal like a bell) that also indicates the closest point on the object being traced.  We didn't discuss how or whether polygonal objects (lakes, for example) could similarly be "outlined" with spatialized audio.

In a prior discussion, Amy noted that blind users might not wish to wear headphones, as they shut out other environmental sounds.   We could get some spatialization, at least in the left/right dimension, even with ordinary speakers, but I am not sure how much we give up in that case. 

We discussed allocentric and egocentric views of maps, and tentatively agreed that spatializing an allocentric view (north is always "forward", west is always "left") is probably easier to deal with.  An allocentric view avoids the problem of maintaining a heading, and it avoids requiring the user to remember what the current heading is.  

 There are software libraries for creating spatialized audio, thanks to video games.  I am unsure how compatible the spatialized audio libraries are with the ESRI software developer infrastructure.  After our meeting, I dug a bit into documentation I could find on spatialized audio in Java, and what I found suggests that it should be possible but not easy.  The documentation I found indicated that one obtains spatialized sounds by building a "scene graph" data structure, which means substantially duplicating the work that ESRI has done to place objects in the 2-dimensional screen space.  Currently, when the ESRI software signals us that the pointer (mouse) has moved, we can query ESRI directly about which objects are under the pointer.  ESRI handles the transformation from real-world coordinate system to screen coordinate system, and our software can pretty much ignore it.  Also we can leave it up to ESRI to read and interpret the input data set.  Maybe there is a way to maintain that clean separation of functionality between the ESRI framework and our soundscape software with spatialization, but my preliminary look at the documentation suggests it might become significantly more tangled.  What this suggests to me is that it may be tactically advantageous to separate exploration of spatialization from the main thread of development at first.  I think it's a really interesting possibility, but it will be most efficient to develop soundscape interaction in the mGIS first without spatialization.  If we want to pursue spatialization, it might be worthwhile to develop, in parallel, a small prototype that is independent of the ESRI framework, specifically for demonstrating and evaluating spatialization of geometry.  

In general, one of the things we can be doing, in addition to devising interaction approaches that we hope will be good, is noting relevant questions raised by the design choices.  By analogy, tactile map design is informed by what is known about the minimum feature size that can be distinguished by a fingertip.  Soundscape map and mGIS design ought to be informed by similar knowledge about perception and cognition of sound.  Some of what we need to know is probably already out there, in the research literature on sound perception and cognition and on audio display (including data sonification).  Some of what we need to know is probably specific enough to geographic data and to the tasks that we need to support that we will need to think about how to answer them ourselves.