Wednesday, July 21, 2010

Things to explore with Jake

Our current plan is that Jake will spend about a day with the computer science sub-team, split into two half-day sessions.  We want to make as good use of that time as possible.  Here is my current list of things we should explore while he is with us:
  • Feature scale and map detail: 
    We should vary the scale and complexity of the map display, to get a better idea of the level of how dense and detailed the display can be.  Our initial belief is that neither fine detail nor a large number of details can be effectively displayed through sound, so we should start with a very sparse display and gradually increase its complexity.  May may do that simply by changing the zoom level.   This may not be enough to distinguish the problem of feature scale, such as the amount of buffering needed around a river to follow it, from density of features, such as the number of rivers shown on a map, but it will be a start. 
  • Device:
    Nafisa has reconfigured the tablet to be an absolute pointing device, rather than a relative pointing device.   We will try the mouse-like puck and then pen again on the Wacom Intuos tablet.  Although we believe the finger-tracking Bamboo tablet is too small for our purposes, we will experiment a bit more with that as well, and try to observe whether there is a real difference between using the pen and using the fingers.
  • Sound selection:
    We won't have spatialized audio to try in August, but there are still several aspects of the sound design that we should informally experiment with, such as how and when to use mimetic sounds such as the sound of water for a river, and when to use speech. 
  • Verbal descriptions:
    We should try providing spoken descriptions of the overall map and the interface.  I am not sure what we should vary, but we should at least give Jake a chance to interact with a map that provides some textual summary information, and get some feedback from him on what kind of overall map description is useful to him.  
  • Hard and soft buttons:
    Nafisa is currently trying to determine whether we can program our own button behavior in Java, or whether we are limited to using the button objects provided through the ESRI developer interfaces.  If possible, we should try corner-based pop-up menus similar to those in Matt's interface mock-up.  We should also try using the hardware buttons on the Wacom Intuos tablet.  I expect that corner-based menus will be the best placement if we need to use the tablet surface, but that the "hard" buttons on the tablet could be even better.  In both cases, what we are trying to do is minimize taking up space when the menus are not in use, and making it very easy to "find" the menus without much memory load or fine control.  With a mouse, a corner is an ideal target because one can "slam" the pointer into a corner by moving further in both the x and y direction than is really necessary ... it can be fast because it doesn't have to be accurate.  The same principle should apply in finding the corner, as long as we make the edges easy to feel, and perhaps provide a physical barrier for the pen.  The hardware buttons may not be quite as easy to find, but it should be possible to locate them by sweeping with the non-dominant hand while keeping the stylus in the dominant hand  (and similarly if the puck or finger-tracking is used).  
This is an initial list, to be refined ... please comment with improvements, things that are missing, potential problems with these topics, etc.

Tuesday, July 20, 2010

iSonic project at UMD - papers and videos

The project that led to Zhao's dissertation on sonified choropleth maps is described at the iSonic project home page.  Some software is available there, but I have not tried it and do not know what stage of the project it represents.  Video demonstrations are also available there, and some links to papers.  It seems, however, that the most recent papers from Zhao and Plaisant are not listed there. 

Monday, July 19, 2010

Spatial sound background

I had a good lunch-time talk with Anthony Hornof today regarding spatial audio.  Anthony's lab has a research-level spatialized audio setup, which he will allow us to  use if we find it useful.  Primarily I was interested in what might be feasible with off-the-shelf hardware such as we might be able to deploy in a middle-school classroom.  I wondered whether we might get benefit of spatialized sound with standard stereo speakers rather than headphones.

First, Anthony told me that high-end audiophile headphones typically use an "open air" design that does not shut out other sound in the environment.   A typical cost for a really good pair is around $300.

Anthony was skeptical of being able to achieve usable spatialization with ordinary stereo speakers.  Of the two cues our brains use to judge location of sound, only one of them would be effectively controlled with ordinary speakers --- I'll write a little more about this below.  Second, the software that is used for spatialization has been designed for controlling sound near the ear, and would not effectively utilize speakers.

Anthony sent me several papers on spatialization, but for background he particularly recommended a book by Durand Begault which is available online at  Durand Begault's publication web page  . The book can be downloaded from this link:  3D Sound for Virtual Reality and Multimedia.   Chapter 2 provides a concise summary of sound location perception.


We use two cues to judge the location of a sound.  One is the relative loudness of a sound at each ear, called "interaural intensity differences", or IID.  The other is small time differences between sound arriving at each ear, called "interaural time differences", or ITD.    These  time differences range from about half a millisecond up to 1.5 milliseconds. 

In a natural setting, we not only judge the location of a sound by differences between what is received at each ear, but we also actively enhance those differences ... if we really want to know where a particular sound is coming from, we turn our head to minimize interaural differences, whereupon our head is pointed in the direction of the sound.   When our head is still, it is difficult to distinguish whether a sound is in front or behind us, but turning our head apparently disambiguates it (Begault, page 39).    We cannot duplicate this facility with headphones or speakers, so we probably could at best simulate the location of objects left-to-right or in a half-circle centered in front of the listener.  That is the approach that Zhao, Plaisant, and Shneiderman used in a pilot study of a sonified choropleth map.  (I'll write more about the study of Zhao et al in another post.)

Thursday, July 8, 2010

Spatialization and other notes from July 7 meeting

In our informal meeting yesterday (July 7), one of the topics we spent some time on was spatialization --- sound that seems to be coming from a particular point in space.  Here are some notes on that from memory, and a couple things I didn't think of until after we met.

Our initial discussion was about dealing with short-term memory limitations (we need to carefully limit the number of things a user must perceive and remember while using a geographic display), and how to provide some sort of scan or orienting display that helps a user choose to where to "look".  By analogy, we know that human vision has a small central focus (fovea) that jumps around (saccades) to take little snapshots in close detail (fixations).  The amount of information in each snapshot is small.   The peripheral vision notes potential targets for the next fixation ... it provides very little detail but is good at noticing change or motion. 

On a tactile map, perhaps a quick scan with the dominant hand, using the other hand as a locational anchor, is serving a purpose similar to peripheral vision in sighted users.  What might the equivalent "scan" be for an audio display?  We discussed a couple of scanning strategies.  One approach would be to divide the display into rows and columns, and display objects in grid cells in a fixed order (e.g., left-to-right, top-to-bottom).   A variation is a circular scan of objects in the vicinity of the pointer, using loudness to indicate proximity. 

Robert or Matt suggested that spatialized audio could be used to indicate not just a point location, but also additional spatial information ... the example was hearing a sound moving along some part of a river, indicating not just the nearest point in the river but also its direction.  This seems like potentially a neat answer to a question that Andrew Marcus posed in an earlier meeting, about communicating information like the direction of river flow.  One can imagine some indication (perhaps a rising and falling loudness to indicate proximity, or a distinct sound signal like a bell) that also indicates the closest point on the object being traced.  We didn't discuss how or whether polygonal objects (lakes, for example) could similarly be "outlined" with spatialized audio.

In a prior discussion, Amy noted that blind users might not wish to wear headphones, as they shut out other environmental sounds.   We could get some spatialization, at least in the left/right dimension, even with ordinary speakers, but I am not sure how much we give up in that case. 

We discussed allocentric and egocentric views of maps, and tentatively agreed that spatializing an allocentric view (north is always "forward", west is always "left") is probably easier to deal with.  An allocentric view avoids the problem of maintaining a heading, and it avoids requiring the user to remember what the current heading is.  

 There are software libraries for creating spatialized audio, thanks to video games.  I am unsure how compatible the spatialized audio libraries are with the ESRI software developer infrastructure.  After our meeting, I dug a bit into documentation I could find on spatialized audio in Java, and what I found suggests that it should be possible but not easy.  The documentation I found indicated that one obtains spatialized sounds by building a "scene graph" data structure, which means substantially duplicating the work that ESRI has done to place objects in the 2-dimensional screen space.  Currently, when the ESRI software signals us that the pointer (mouse) has moved, we can query ESRI directly about which objects are under the pointer.  ESRI handles the transformation from real-world coordinate system to screen coordinate system, and our software can pretty much ignore it.  Also we can leave it up to ESRI to read and interpret the input data set.  Maybe there is a way to maintain that clean separation of functionality between the ESRI framework and our soundscape software with spatialization, but my preliminary look at the documentation suggests it might become significantly more tangled.  What this suggests to me is that it may be tactically advantageous to separate exploration of spatialization from the main thread of development at first.  I think it's a really interesting possibility, but it will be most efficient to develop soundscape interaction in the mGIS first without spatialization.  If we want to pursue spatialization, it might be worthwhile to develop, in parallel, a small prototype that is independent of the ESRI framework, specifically for demonstrating and evaluating spatialization of geometry.  

In general, one of the things we can be doing, in addition to devising interaction approaches that we hope will be good, is noting relevant questions raised by the design choices.  By analogy, tactile map design is informed by what is known about the minimum feature size that can be distinguished by a fingertip.  Soundscape map and mGIS design ought to be informed by similar knowledge about perception and cognition of sound.  Some of what we need to know is probably already out there, in the research literature on sound perception and cognition and on audio display (including data sonification).  Some of what we need to know is probably specific enough to geographic data and to the tasks that we need to support that we will need to think about how to answer them ourselves. 

Monday, June 28, 2010

Sound design dimensions to evaluate

While our interactions with Jake will not be controlled experiments, we should nonetheless think about what design choices we will need to make in soundscape design, and try where practical to create alternative versions that vary along that dimension for evaluation and feedback.  Here are some dimensions that come to mind so far:

  • Natural (mimetic) sounds vs symbolic (non-mimetic) sounds:  I conjecture that natural sounds (like water flowing for a water feature) may be easier to initially learn, but we can make symbolic music sounds that are faster to distinguish (allowing a faster scan of a map display) with practice.  Also I think it will be easier to design systematic variations among musical sounds (e.g., different piano intervals are all recognizably piano and not marimba, but can be distinguished from each other).  I would like in an initial prototype to include a soundscape that uses only natural sounds, one that uses only musical sounds, and one that layers an initial very short musical sound over a mimetic natural sound. 
  • Fewer or more classes of sounds.  For example, we have three classes of water feature for Yellowstone (stream, river, and lake, as I recall).  Should they be distinguished in sound representation?  I am not sure whether we want to create variations along this dimension right away ... but perhaps it would not be too hard.  
  • Sound groupings.  If we do want larger sets of sounds with finer distinctions (e.g., stream vs river), then we should consider ways of thematically grouping sounds.  For example, different classes of water feature could all be represented by piano, with pitch (or interval) distinguishing the particular feature class. 
  • Sound duration.  How long does a clip need to be to be recognizable?  This is surely tangled up with the kinds of sounds we use, and how distinct they are.  As mentioned above, my initial thought is that a percussive musical instrument that is identifiable by timbre (e.g., piano, xylophone, tympani, glockenspiel, plucked violin string, etc.) will allow scans that are quick relative to either musical sounds with a less rapid attack and decay (e.g., oboe, flute, bowed violin) or non-percussive natural sounds (water, car horn, phone touch-tone).   Levitin's book "This is Your Brain on Music" notes that our near-instantaneous recognition of familiar recorded music depends primarily on timbre (as versus the sound dimensions pitch, tempo, rhythm, loudness).  If mimetic natural sounds are indeed more learnable, we may need to consider both the required duration of an initial symbolic sound and a longer mimetic sound. 
These are initial thoughts.  What are some other design parameters we should be considering? 

Thursday, June 24, 2010

Suitable Audio Cues

In the current version of soundscape mGIS, we have identification with speech. Whenever the mouse enters into a new region, the program identifies the region and the speaker enunciate the name of the region and be silent until the mouse moves inside the same region. In order to make this soundscape feature more expressive for a blind user we can include some audio cues.

Michal and I talked about this today and came up with the idea that apparently we need to have three different audio cues (naturalistic or musical or both):

1) A tiny little sound that represents entering into a new region. (also can be thought of region changed)
2) One sound that can be played continuously while the mouse is hovering inside a region. Our current simple yellowstone park map basically has two different types of regions:
a)Water (including river, lakes and creeks),
b)Park base or land.
So for type 2 we need to make two different kinds of sounds that can represent these regions more significantly.

Proximity, headsets, and audio input

In our Monday meeting, we briefly discussed direction and distance of sound cues.  These are tied to the use of headphones, and indirectly to Jake's idea about voice input:

There are different levels of "proximity" information we could provide:
  • Distance only (no direction) - just with loudness.  We could scale loudness based on a realistic function (and we should at least find out what that is), or we could use an artificial function to either increase or decrease the range at which proximate regions become audible.  This is further broken into two sub-cases: 
    • Continuous distance scaling.  If the loudness of an object differs as a continuous function of distance, then one can judge direction by moving the mouse (maybe ... this may be difficult). 
    • Discrete scaling.  The simplest version of this is to have an extra buffer around each object, in which its sound is audible but less loud.  With discrete scaling, mouse motion does not reveal the direction to an object unless the boundary of a surrounding buffer is crossed. 
  • Directional.  The most accurate directional audio requires headphones and computed with an individual head-related transfer function ( HRTF ), but some left-right directionality can be achieved using a generic model and even with ordinary stereo speakers.  We are better at sensing direction of high-pitched sounds than lower pitches (which is why a surround-sound system has more tweeters and mid-range drivers than woofers, and why high-end stereo systems often use a single sub-woofer).   High quality directional audio is a good deal more computationally intensive than simply varying loudness, but directional audio is supported program libraries (including for Java) because it is used in some games. 
Amy noted that she has never seen a blind person wearing headphones, and she conjectured that blind people might find them objectionable because they "close off" the wearer.

The indirect link between this and Jake's idea about audio input is that the most accurate audio input comes from headset microphones.  The built-in microphones in most computers are poor quality and/or have a problem with noise from the computer.  It should be possible to provide a good-quality microphone that is not part of a headset, but it could be a challenge to keep it well-positioned relative to a blind user.