Showing posts with label ESRI software. Show all posts
Showing posts with label ESRI software. Show all posts

Thursday, July 8, 2010

Spatialization and other notes from July 7 meeting

In our informal meeting yesterday (July 7), one of the topics we spent some time on was spatialization --- sound that seems to be coming from a particular point in space.  Here are some notes on that from memory, and a couple things I didn't think of until after we met.

Our initial discussion was about dealing with short-term memory limitations (we need to carefully limit the number of things a user must perceive and remember while using a geographic display), and how to provide some sort of scan or orienting display that helps a user choose to where to "look".  By analogy, we know that human vision has a small central focus (fovea) that jumps around (saccades) to take little snapshots in close detail (fixations).  The amount of information in each snapshot is small.   The peripheral vision notes potential targets for the next fixation ... it provides very little detail but is good at noticing change or motion. 

On a tactile map, perhaps a quick scan with the dominant hand, using the other hand as a locational anchor, is serving a purpose similar to peripheral vision in sighted users.  What might the equivalent "scan" be for an audio display?  We discussed a couple of scanning strategies.  One approach would be to divide the display into rows and columns, and display objects in grid cells in a fixed order (e.g., left-to-right, top-to-bottom).   A variation is a circular scan of objects in the vicinity of the pointer, using loudness to indicate proximity. 

Robert or Matt suggested that spatialized audio could be used to indicate not just a point location, but also additional spatial information ... the example was hearing a sound moving along some part of a river, indicating not just the nearest point in the river but also its direction.  This seems like potentially a neat answer to a question that Andrew Marcus posed in an earlier meeting, about communicating information like the direction of river flow.  One can imagine some indication (perhaps a rising and falling loudness to indicate proximity, or a distinct sound signal like a bell) that also indicates the closest point on the object being traced.  We didn't discuss how or whether polygonal objects (lakes, for example) could similarly be "outlined" with spatialized audio.

In a prior discussion, Amy noted that blind users might not wish to wear headphones, as they shut out other environmental sounds.   We could get some spatialization, at least in the left/right dimension, even with ordinary speakers, but I am not sure how much we give up in that case. 

We discussed allocentric and egocentric views of maps, and tentatively agreed that spatializing an allocentric view (north is always "forward", west is always "left") is probably easier to deal with.  An allocentric view avoids the problem of maintaining a heading, and it avoids requiring the user to remember what the current heading is.  

 There are software libraries for creating spatialized audio, thanks to video games.  I am unsure how compatible the spatialized audio libraries are with the ESRI software developer infrastructure.  After our meeting, I dug a bit into documentation I could find on spatialized audio in Java, and what I found suggests that it should be possible but not easy.  The documentation I found indicated that one obtains spatialized sounds by building a "scene graph" data structure, which means substantially duplicating the work that ESRI has done to place objects in the 2-dimensional screen space.  Currently, when the ESRI software signals us that the pointer (mouse) has moved, we can query ESRI directly about which objects are under the pointer.  ESRI handles the transformation from real-world coordinate system to screen coordinate system, and our software can pretty much ignore it.  Also we can leave it up to ESRI to read and interpret the input data set.  Maybe there is a way to maintain that clean separation of functionality between the ESRI framework and our soundscape software with spatialization, but my preliminary look at the documentation suggests it might become significantly more tangled.  What this suggests to me is that it may be tactically advantageous to separate exploration of spatialization from the main thread of development at first.  I think it's a really interesting possibility, but it will be most efficient to develop soundscape interaction in the mGIS first without spatialization.  If we want to pursue spatialization, it might be worthwhile to develop, in parallel, a small prototype that is independent of the ESRI framework, specifically for demonstrating and evaluating spatialization of geometry.  

In general, one of the things we can be doing, in addition to devising interaction approaches that we hope will be good, is noting relevant questions raised by the design choices.  By analogy, tactile map design is informed by what is known about the minimum feature size that can be distinguished by a fingertip.  Soundscape map and mGIS design ought to be informed by similar knowledge about perception and cognition of sound.  Some of what we need to know is probably already out there, in the research literature on sound perception and cognition and on audio display (including data sonification).  Some of what we need to know is probably specific enough to geographic data and to the tasks that we need to support that we will need to think about how to answer them ourselves.