Showing posts with label research questions. Show all posts
Showing posts with label research questions. Show all posts

Wednesday, July 21, 2010

Things to explore with Jake

Our current plan is that Jake will spend about a day with the computer science sub-team, split into two half-day sessions.  We want to make as good use of that time as possible.  Here is my current list of things we should explore while he is with us:
  • Feature scale and map detail: 
    We should vary the scale and complexity of the map display, to get a better idea of the level of how dense and detailed the display can be.  Our initial belief is that neither fine detail nor a large number of details can be effectively displayed through sound, so we should start with a very sparse display and gradually increase its complexity.  May may do that simply by changing the zoom level.   This may not be enough to distinguish the problem of feature scale, such as the amount of buffering needed around a river to follow it, from density of features, such as the number of rivers shown on a map, but it will be a start. 
  • Device:
    Nafisa has reconfigured the tablet to be an absolute pointing device, rather than a relative pointing device.   We will try the mouse-like puck and then pen again on the Wacom Intuos tablet.  Although we believe the finger-tracking Bamboo tablet is too small for our purposes, we will experiment a bit more with that as well, and try to observe whether there is a real difference between using the pen and using the fingers.
  • Sound selection:
    We won't have spatialized audio to try in August, but there are still several aspects of the sound design that we should informally experiment with, such as how and when to use mimetic sounds such as the sound of water for a river, and when to use speech. 
  • Verbal descriptions:
    We should try providing spoken descriptions of the overall map and the interface.  I am not sure what we should vary, but we should at least give Jake a chance to interact with a map that provides some textual summary information, and get some feedback from him on what kind of overall map description is useful to him.  
  • Hard and soft buttons:
    Nafisa is currently trying to determine whether we can program our own button behavior in Java, or whether we are limited to using the button objects provided through the ESRI developer interfaces.  If possible, we should try corner-based pop-up menus similar to those in Matt's interface mock-up.  We should also try using the hardware buttons on the Wacom Intuos tablet.  I expect that corner-based menus will be the best placement if we need to use the tablet surface, but that the "hard" buttons on the tablet could be even better.  In both cases, what we are trying to do is minimize taking up space when the menus are not in use, and making it very easy to "find" the menus without much memory load or fine control.  With a mouse, a corner is an ideal target because one can "slam" the pointer into a corner by moving further in both the x and y direction than is really necessary ... it can be fast because it doesn't have to be accurate.  The same principle should apply in finding the corner, as long as we make the edges easy to feel, and perhaps provide a physical barrier for the pen.  The hardware buttons may not be quite as easy to find, but it should be possible to locate them by sweeping with the non-dominant hand while keeping the stylus in the dominant hand  (and similarly if the puck or finger-tracking is used).  
This is an initial list, to be refined ... please comment with improvements, things that are missing, potential problems with these topics, etc.

Thursday, July 8, 2010

Spatialization and other notes from July 7 meeting

In our informal meeting yesterday (July 7), one of the topics we spent some time on was spatialization --- sound that seems to be coming from a particular point in space.  Here are some notes on that from memory, and a couple things I didn't think of until after we met.

Our initial discussion was about dealing with short-term memory limitations (we need to carefully limit the number of things a user must perceive and remember while using a geographic display), and how to provide some sort of scan or orienting display that helps a user choose to where to "look".  By analogy, we know that human vision has a small central focus (fovea) that jumps around (saccades) to take little snapshots in close detail (fixations).  The amount of information in each snapshot is small.   The peripheral vision notes potential targets for the next fixation ... it provides very little detail but is good at noticing change or motion. 

On a tactile map, perhaps a quick scan with the dominant hand, using the other hand as a locational anchor, is serving a purpose similar to peripheral vision in sighted users.  What might the equivalent "scan" be for an audio display?  We discussed a couple of scanning strategies.  One approach would be to divide the display into rows and columns, and display objects in grid cells in a fixed order (e.g., left-to-right, top-to-bottom).   A variation is a circular scan of objects in the vicinity of the pointer, using loudness to indicate proximity. 

Robert or Matt suggested that spatialized audio could be used to indicate not just a point location, but also additional spatial information ... the example was hearing a sound moving along some part of a river, indicating not just the nearest point in the river but also its direction.  This seems like potentially a neat answer to a question that Andrew Marcus posed in an earlier meeting, about communicating information like the direction of river flow.  One can imagine some indication (perhaps a rising and falling loudness to indicate proximity, or a distinct sound signal like a bell) that also indicates the closest point on the object being traced.  We didn't discuss how or whether polygonal objects (lakes, for example) could similarly be "outlined" with spatialized audio.

In a prior discussion, Amy noted that blind users might not wish to wear headphones, as they shut out other environmental sounds.   We could get some spatialization, at least in the left/right dimension, even with ordinary speakers, but I am not sure how much we give up in that case. 

We discussed allocentric and egocentric views of maps, and tentatively agreed that spatializing an allocentric view (north is always "forward", west is always "left") is probably easier to deal with.  An allocentric view avoids the problem of maintaining a heading, and it avoids requiring the user to remember what the current heading is.  

 There are software libraries for creating spatialized audio, thanks to video games.  I am unsure how compatible the spatialized audio libraries are with the ESRI software developer infrastructure.  After our meeting, I dug a bit into documentation I could find on spatialized audio in Java, and what I found suggests that it should be possible but not easy.  The documentation I found indicated that one obtains spatialized sounds by building a "scene graph" data structure, which means substantially duplicating the work that ESRI has done to place objects in the 2-dimensional screen space.  Currently, when the ESRI software signals us that the pointer (mouse) has moved, we can query ESRI directly about which objects are under the pointer.  ESRI handles the transformation from real-world coordinate system to screen coordinate system, and our software can pretty much ignore it.  Also we can leave it up to ESRI to read and interpret the input data set.  Maybe there is a way to maintain that clean separation of functionality between the ESRI framework and our soundscape software with spatialization, but my preliminary look at the documentation suggests it might become significantly more tangled.  What this suggests to me is that it may be tactically advantageous to separate exploration of spatialization from the main thread of development at first.  I think it's a really interesting possibility, but it will be most efficient to develop soundscape interaction in the mGIS first without spatialization.  If we want to pursue spatialization, it might be worthwhile to develop, in parallel, a small prototype that is independent of the ESRI framework, specifically for demonstrating and evaluating spatialization of geometry.  

In general, one of the things we can be doing, in addition to devising interaction approaches that we hope will be good, is noting relevant questions raised by the design choices.  By analogy, tactile map design is informed by what is known about the minimum feature size that can be distinguished by a fingertip.  Soundscape map and mGIS design ought to be informed by similar knowledge about perception and cognition of sound.  Some of what we need to know is probably already out there, in the research literature on sound perception and cognition and on audio display (including data sonification).  Some of what we need to know is probably specific enough to geographic data and to the tasks that we need to support that we will need to think about how to answer them ourselves.