Sonitunes (Digital Lines App): University AT Project

As we transition to digital education materials, developers are facing new types of accessibility challenges. Traditionally, graphics have been produced tactually; how can students with visual impairments access these graphics in a digital format? There are many pieces that have to come together before new standards are created. Let's take a look at the process.

Process 

  1. Identify the issue
  2. Come up with a creative, innovative solution
  3. Field test solution with students, TVIs and educators
  4. Improve the solution (Repeat steps 1 - 4 as many times as necessary)
  5. Pair new solution with traditional tactile training and bridge between the two formats
  6. Train TVIs on new solution and the bridge between tactile and digital formats
  7. Train students using all formats (tacile, bridge, and digital formats) and then apply the new tech skills into mainstream educational tasks
  8. Build a systematic curriculum that grows the new tech skill along with the mainstream educational skill

Example

In theory this sounds good: identify an issue and solve it. Talking with mainstream software developers who are creating new types of technology, I learned that it can often be a 10 year process from the time a robust project - developing cutting edge tech - is started to a full public release. And once launched, the software continues to evolve and be updated regularly.

Let's take a look at a common task for preschool/early elementary students: Following and creating 'lines'. Take a look at traditional worksheets for young students. Students are asked to draw a line between two matching items. The line might be from a bird to a nest, three apples and the number 3, or the letter A and an apple. Traditionally, matching activities were put into braille/tactile format for students who are visually impaired. There are literally hundreds of apps that include the same activity; however, these apps,like most preschool and early elementary apps, are currently not accessible for students who rely on a screen reader.

Note: Drawing a physical line teaches students with vision more than just matching - it also provides practice with eye-hand coordination and emerging writing skills. Digital line drawing will also teach additional skills.

1. Identify the issue

Create an accessible digital line.

2. Come up with a creative, innovative solution

Current available options: There are several Android based apps (Feelif and Vital) that are using haptics (vibrations) and sonification (sounds) to enable a student to follow the visual line that has been tagged. iPads do not have the vibration feature; however, there are several ways to make an accessible 'line' on an iOS device. In Pages, you can create an accessible bar chart; touch anywhere on the bar to hear the bar labels. (Learn more about accessible bar charts here.) SAS Graphics Accelerator uses sonification to make charts and graphs accessible. (Learn more about the Accelerator here.) For decades, students who are visually impaired have been using sonification for higher math with talking graphing calculators; however, this type of sonification has not been yet been made available for activities appropriate for young students and young students have not been systematically taught how to use sonification. Just like children learn to walk before they run, it makes sense to mirror traditional skills and sequence such as drawing a line before graphing a parabola.

Solution: Create a proof of concept app that enables a young student using a screen reader to locate, follow and create digital lines using sonification. Vertical lines will use pitch (dragging up the line increases the pitch; dragging down the line decreases the pitch), horizontal lines will use tempo (speed of sound increases to the right and decreases to the left) and diagonal lines use both pitch and tempo. Each color will be associated with a different sound. 

The app will include a tutorial, activities to practice creating a line, then applying the skills to create a line between two matching items. Additional academic skills will be incorporated, such as the terms right/left, horizontal/vertical/diagonal, short/long, etc.

Computer Science Team at the University of North Carolina

In a recent post, Hidden Resources: Collaborative AT Projects with Universities, we learned about teaming with university computer science and/or biomedical engineering departments to create accessible educational tools. The Sonified Lines app project was presented to the Serious Games class at the University of North Carolina, spring semester, 2019. A dedicated three-person student team took on the project with support from their professor. This passionate and talented group of students began researching how to create a sonified line while running VoiceOver and began fleshing out an app that would teach the desired skills. 

Hmm, this all sounds good in theory, but will a sonified line really work for students who are visually impaired? Take a look at the demonstration videos! The first video introduces the Sonitunes app and demonstrates how to find and follow a digital line that has been sonified.

 
Pretty amazing, huh? Honestly, finding and following a sonified line is fairly easy after a few tries! Think of all the skills you can teach through this app in addition to finding and following a digital line. Can you apply concepts learned through Sonitunes to O&M and math? You bet!
 
Keep in mind that Sonitunes is in the toddler stage - the concept of sonified lines has been developed and the basic structure of the app is up and running. However, a semester is a very short time to develop an app and this passionate team has made incredible progress! The app has not yet been perfected; it can, however, be used as a 'proof of concept' that students with visual impairments can indeed find and follow a digital line.
 
The next video demonstrates how students can create a digital line. Note: Currently, when VoiceOver is on, the line is hidden from view and is not accessible. (We are working on it!)
 
 

The third video demonstrates drawing a line to match two items.

Next Steps

Let us know what you think of using Sonification and of Sonitunes! Do you have ideas on how to improve the app? Additional concepts that can be taught? Additional ideas for teaching sonification?

The semester is over and the three-person team is moving on. One team member (who graduated in May 2019) is volunteering to work on Sonitunes this summer, before starting his new job. Sonitunes is being shared as a proof of concept with interested app developers. Sonitunes may be pitched to another team in the fall as well. Please stay tuned to find out more about Sonitunes!

Thank you!

A big thanks to the dedicated Sonitunes team (Amit Enand, Christopher Burras, and Dasol Lee), their supporting computer science professor (Dr. Diane Pozefsky),and the University of North Carolina's computer science department for strongly supporting accessibility-related projects!

Related Post

Accessible Number Line App: University AT Project

Sonitunes collage