PalmOS takes a tour through 'Rome'

The Backstory:

The PalmOS operating system user experience - designed from the beginning in 1998 for touch - was refreshed from PalmOS 5 to PalmOS 6, but with no operator support. In 2003, PalmSource fell back to extending PalmOS 5, and decided to innovate in its architecture and user experience. After a few design experiments, PalmSource hired me in 2004 to validate the new design, internally named Rome, or create another design, while delivering on overlapping releases for customers.

The dominant mobile device UI at the time was Nokia's Symbian (non-touch): like PalmOS, employing a five-key rocker hardware button (left, right, up, down, center). But unlike PalmOS, the Symbian UI reserved a space at the bottom of the screen to display conditional 'soft key' labels for physical buttons. This provided scalability (the buttons could perform whatever function they are labeled on the screen, rather than on the hardware) but introduced other problems, such as designing 'reverse locations' for operations the user would regret if agreed by mistake. Nokia would not ship a Symbian touch device until 2008 (5800 XpressMusic, 'Tube' - you can read a prescient review here).

The Brief:

Validate the updated navigation style and ensure that it is easy to use, and presents no problems for users transitioning from a Nokia-style interface. The test must determine whether previous soft key usage affects users' ability to operate the new interface. Validate the claims that the new interface:

Overcomes disorientation common to mobile UX (e.g., avoids tunneling menus)
Supports multitasking (e.g., play music while browsing)
Manages interruptions (e.g., receive call while checking calendar)
Enables access to background tasks from all screens (ACCESS Panels)
Provides background activity awareness to the user
Provides predictable navigation model and globally available “back”
Provides one handed interaction model via 5 way navigation & dedicated hard keys
Supports optional touch screen

The strategic issues:

Updating the OS is costly and can be polarizing to existing users; proceed carefully
Usability tests find problems, not goodness; even great results don't prove that the UI will be successful

The result:

We tested in Salzburg, Austria and Cape Town, South Africa, with the same results. Users had just a little difficulty using the new user interface. The architecture challenges were much bigger than the user interface challenges, and PalmSource dropped the investment. We continued to update the classic system with operator- and customer-requested features. By 2008, Apple introduced the iPhone, Palm invested in WebOS, and the Palm OS became history.

The following narrative is based on an unpublished academic paper with co-author and test moderator Kris Mihalic, describing the South Africa test. Test crew included me, Hiroyuki Toki (Access), Kris Mihalic (ICTS/University Salzburg), Leo Ah Kun (University of South Africa), and our hosts Prof. Manfred Tscheligi (University of Salzburg) and Prof. Gary Marsden (University of Cape Town).

Why do overlapping releases matter? Take a look at this sample timeline.

Feature updates are easier to estimate, when a customer is specifying exactly what they want. But estimation becomes art and craft when a mature system adopts architecture changes and updates its user experience.

The Rome project focused on new capabilities that work well for users and attract developers. The chief architect of Rome was George Hoffman, who generated some of the original intellectual property.

What are soft keys and why are they outdated?

The soft key interaction paradigm on mobile phones has been a de facto standard for years. Similar to another successful but aging metaphor - the desktop for personal computers - little innovation has happened beyond the predominant soft key concept. We tested an alternative hard key driven user interface, which shows promise toward advancing ease of use for increasingly complex personal mobile interaction.

The soft key interaction paradigm on mobile phones has been a de facto standard for a decade now. First introduced in the Nokia Series 30 UI style in the mid 1990s, the soft key approach was designed to provide intuitive and consistent interaction. Like the desktop metaphor for personal computers' graphical user interface, little innovation has appeared beyond the soft key paradigm prevalent in commercial handsets. Some devices feature only one soft key (Nokia calls their implementation Navi-Key) [12], the dominant approach employs two soft keys, and some manufacturers have scaled the concept, producing three (e.g. Motorola A820, Nokia 6650) or even four soft keys (e.g. Siemens C/M 35). But aside from the PDA 'mini-desktop' metaphor and the use of directional shortcuts that map to applications on a mobile phone's introductory screen, little innovation has occurred to advance the interface paradigm beyond soft keys.

A soft key is a function expressed in an onscreen, or 'soft', key that maps directly to a physical hard key on the device. The soft key is positioned so that the relationship between the hard key and its onscreen counterpart is clear. Soft keys are almost always used in opposing pairs located on the bottom left and right of the device screen. Three button soft key arrangements that span the bottom of the device screen are becoming more common. The hard keys are located immediately below the screen, positioned to correlate closely with the soft keys. The opposition in two-soft key placement echoes opposition in function; one button usually negates an action, and the other promotes or extends the number of potential actions. The three and four soft keys models attest to how limiting the original concept has become as more and more functionality is available in today's handheld devices.

Soft keys are commonly used to invoke single commands or a list of menu items. Because screen real estate on mobile phones is limited, this construction has been useful to clearly identify multiple options using a minimum of buttons. While soft keys are most often used with mobile phones and some PDAs, other devices and appliances make use of this interaction paradigm, including Automated Teller Machines, Multi-Function Displays, etc. Touch screens employ on-screen buttons; these are not termed soft keys since they have no hard key counterparts.

As opposed to soft keys, hard keys (or labeled keys) are mapped to only one function. The advantage of this is that using the soft keys produces predictable results. Lately, some manufacturers have introduced hard keys for Home and Back on their devices (e.g. T-Mobile SDA or Cingular cell phones running Microsoft Windows Mobile OS), in addition to the soft key interface. However, devices with many uniquely mapped hard keys suffer: too many buttons increase the user's cognitive load. For example, consider the remote control for home electronics; many of the buttons are never used or are poorly understood because the labeling is poor, the function is seldom necessary and therefore difficult to remember, or the user finds a more memorable workaround. Thus while we claim that limiting complex interaction to two soft keys is problematic, spreading interaction across many dedicated hard keys is equally problematic.

Much of the research in the field of mobile HCI is concerned with alternative text entry methods for various applications [6, 13, 15]. Likewise, there is a great body of research on menu structures and accessing various functions of a mobile phone in a usable manner [1, 5, 8, 14]. Also, different types of soft key interaction have been compared previously [10]. The soft key interaction style is considered an improvement over a simple phone with hard keys only; soft keys have been found to be more intuitive and usable compared to the former hard key interaction. However, as Kiljander points out [10], this type of interaction is only one aspect of improvement, others being better and larger displays capable of presenting more information, more processing power, better connectivity, and further advances in technology. Mobile phones in general have expanded in the availability of features, devices targeted to specific needs and wants, and advances in telephony from 2G networks to the promise of 4G networks. However, alternatives to the prevalent soft key interaction paradigm have not been proposed nor have we witnessed adoption of alternatives by the industry on a large scale.

Findings

Navigation concepts

Our results are based on qualitative findings with support by quantitative information gathered during the formal usability inspection. The quantitative aspects are usually covered by time on task, successful completion rate, and affective response to the test. Although we gathered this information, we were interested in particular navigation concepts on a finer level of granularity.

In our approach, we have identified four navigation concepts as indicators for the hardkey interaction paradigm as follows:

Using the Back key for navigating back. The functionality of the right hardkey was to bring the user one screen back to where she was before. This is a common navigation pattern found in other systems.

Using the Options Key to invoke and dismiss the Options panels. The left hardkey was mapped to opening and closing the Options panel (toggle function). Horizontal navigation in the Options panel was only possible once the Options Bar was active and a panel was open.
Horizontal navigation in tabs. Tabs were used in the Contacts applications to separate different levels of detail for particular information, e.g. summary, details, and full-frame image of a phonebook contact.
Horizontal navigation in the Options Bar. This concept allows the user to navigate left and right through currently open applications, e.g. accessing a running music player.

All tasks were broken down into atomic actions. The level of detail for the actions was fine enough that the same actions could be identified across different tasks. If it were too narrow, there would have been too many distinct actions to track. On the other hand, a vague high-level action would not provide any valuable information on the navigation concept. We have defined a total of 20 distinct atomic actions. They were of the form "Options panel navigate right," "Call state switch hold," or "Contacts edit."

Based on these atomic actions we have identified relevant instances of these actions across all tasks and participants. A relevant instance describes an instance of an action relevant for a particular task that can be accomplished by the participant in the order of executing the actions. If a task is aborted before the particular action, e.g. due to timeout, then this is not considered a relevant instance of the action, since the participant didn't get the chance to perform it. However, if the action is the last action before aborting the overall task, then the instance is considered relevant, though not successful. With this method we were able to track the success not only on a per task level, but on a more detailed level of actions.

Using the Back key

In general, the results show that the hardkey interaction pattern was successful and was picked up quickly by the participants, despite little familiarity with personal computers and web browsers. Using the Back key was straightforward and they did not report, nor did we observe, any difficulties using it. The navigation concept was clear to the participants. In the interview, all subjects stated that they knew how to us the Back key and that in case they got stuck in the system, they would use the Back key to navigate to a familiar state (usually the Launcher). While this might sound logical, it is important to stress that only a few of the participants had previous experience with using back navigation. Many subjects had little or no computer experience and were thus not familiar with the concept of navigating back. We probed users for their anticipated feelings about certain system elements before the test and asked the same questions again at the end of the test. Almost one third of the subjects (28%) where doubtful at first, expressing that they expected the Back key to be difficult and confusing. However, after using the system in the test, all participants felt that the Back key was easy to use, clear, and efficient (Figure 4).

Motivation for the test

Soft keys have benefits and trade-offs. The primary benefit to users is a consistent clear location to invoke commands; the trade-off is that since the buttons are usually oppositional, one will Exit, Quit, or Return to a previous screen, and the other is left to do all the 'heavy lifting' presenting all of the menu items available to the user in a single, tunneling list. Where hierarchy is used, the menu becomes a complex series of one menu item sub-launching another list, with this pattern repeating; the user is left to navigate cycling menus or use a Back key to return to the previous level. This arrangement makes terminology and recognizability increasingly important as users have to memorize their path, remember both how to arrive at their target operation, and memorize what it is called. The introduction of a third soft key reduces the complexity of the individual keys, but still presents the user with cognitive complexity in the shell game of 'where's my option?'

We believe that a new paradigm is necessary to help users cope with this memory problem. In identifying the chief problems with navigation on mobile phones, we developed a different interface style to address the following common complaints about mobile device interaction:

Where am I in the interface?
How do I remember how to get where I want to go?

The switch from soft keys to hard keys presents a paradigm shift. In line with Kuhn's view [11], it is a change from one way of thinking to another. The introduction of the personal computer and the Internet, as well as the switch from the desktop to the mobile is a catalyst for a paradigm shift, allowing the change from an industrial society to an information-centered society. As Kuhn states, the "awareness is prerequisite to all acceptable changes of theory" [11]. New assumptions require the reconstruction of prior assumptions and the reevaluation of prior facts. Historically, this is a difficult and time-consuming process that is strongly resisted by the established communities.

Some early evaluators of our system design were particularly skeptical that users accustomed to a soft key paradigm could successfully use, much less enjoy, a departure from what has become the norm. Our first user test of the system gave us sufficient positive feedback to continue to develop the capabilities of our prototype. This paper reports our second test of the system. Our challenge was to address to what degree familiarity with the soft key paradigm interfered with successful use of the non-soft key interface. Our study showed that users enjoy and understand the interface after they have used it and had it explained, but they had difficulty understanding it at first. Our findings confirmed that the interaction model itself is valid, and task completion and success rates will likely dramatically improve with some very specific changes to the visual design of system feedback.

Prototype details

Our approach presents an alternative to the common soft key paradigm by introducing a hard key interaction that relies on five labeled keys with defined functionalities: Home, Back, Options, Off Hook, and On Hook, and a five-way rocker (Up, Down, Left, Right, Center Press).

The key mappings are:

Home always navigates the user to the beginning of the prototype. In our case, that is a launcher screen that permits users to select applications. In most mobile phones, the Home screen is an idle screen that precedes the mobile operator's own main menu. The home screen usually offers some user customization options as well as some operator-locked options.
Options activates the Options bar at the bottom of the screen and permits the user to navigate left and right across the available Options panels. Options are essentially lists of menu choices. This button most closely maps to the concept of a 'positive' (enabling) navigational soft key (as opposed to the 'negative' or exit soft key).
Off Hook is almost always a glyph of a telephone receiver shown in green; the Off Hook key on all mobile phones is used to initiate a phone call. It is also sometimes mapped to invoking the telephone call log or, on a long press, to dialing the last called number.
On Hook is almost always a glyph of a telephone receiver shown in red; the On Hook key on all mobile phones is used to end a phone call. Some handsets double-map the On Hook key to a Quit or character-level Clear function in addition to ending the call; in all cases, it is a "negative" action. Most handsets double-map the On Hook key to a power key to turn on or wake up the device.
Back is a "return to previous" key familiar to users who have experience navigating in web browsers.

In addition to these hard keys, the interface depends on navigating in four directions with a five-way rocker permitting left/right and up/down movement through the interface, with a press on the center button mapped to select and/or activate an onscreen object. We ran our prototype on a real mobile phone that can boot into our environment. We developed a subset of common mobile phone functionalities to exercise our navigation concepts. The functionalities include:

Idle screen for quick starting of four predefined applications by pressing one of the 5-way rocker directions (left, right, up, down), as well as accessing the Launcher (center press).
Launcher for accessing phone applications. While we display a full screen of application icons, only three are activatable.
Contacts application for managing the phone's address book.
Photos application for browsing and viewing images.
Music application for managing and playing MP3 audio files.
Call handling application for handling calls initiated by the Contacts application or received by the handset.

In addition, the prototype features initiating new calls (from the Contacts application only - direct dialing was not implemented), receiving calls, and call handling (placing calls into active, mute, or hold states).

Our prototype used no soft key/hard key mappings. Figure 1 (a) shows the Photos application; Figures 1 (b) shows the Music application. The bar at the top of the screen is a status bar showing various system status elements such as battery and network signal strength. The bar at the bottom of the screen is an Options bar. The Options bar, when invoked, animates up from the bottom of the screen and partially covers the foreground application.

Figure 1 (a). The first round icon represents the Options panel for the Photo application; the second round icon represents an always-present panel for the use of the mobile carrier. Carriers' business depends on generating revenue associated with their brand. We included this persistent panel in recognition of access to operator services and branding requirements that the operator often stipulates in the design of a commercial handset user experience.

Figure 1 (b). The Call Handling Options panel is opened over the Music application. The panel shows that the user is currently in an active phone call and has selected End Call to terminate the call. In our prototype, the On Hook button also maps to ending the call; navigation to an Options item isn't necessary but is provided so that users focusing on the screen region don't have to remember to hit the On Hook hardkey to end their call.

Note that in 1 (b) the first Option icon refers to the Music list; the second Option icon shows that music is playing in the Player; the third Option icon shows that there is currently a call in progress; and the last Option icon provides access to the mobile operator's panel.

The Options hard key is used to activate the Options bar and open the first Options panel. The Options bar contains the menus for active applications. The Options hard key toggles the state of the Options bar: the first press activates the bar and opens the first panel; a second press on the Options hardkey closes the panel and returns the navigation to the main form (the foreground application). The first Options panel generally provides the functions for the currently active (foreground) application. All choices on Options panels are navigated vertically; lateral navigation switches focus to a neighboring panel. By navigating left and right on the open Options panel, the user can successively access all currently active applications.

This new concept entirely eliminates soft keys from the interaction model. It allows navigating through the system by relying exclusively on the predefined hard keys and the navigational concept behind them. We provided a Home key for faster access to the application launcher; our commercial design based on this paradigm maps the Home key to the idle screen following the pattern used by most mobile phones.

Study and Methodology

We wanted a diverse group of participants, yet all participants had to be able to speak English. For our test in Salzburg, we recorded and transcribed participant discussions from German to English for analysis. For our second test in South Africa, all participants spoke English (and one, two, or three additional languages). We tested in these two locations because they met our needs, and were graciously offered to us by academic colleagues: Professor Manfred Tscheligi, University of Salzburg, with assistance from his staff, Kris Mihalic; and Professor Gary Marsden, University of Cape Town, with assistance from his staff, Leo Kuhn. Observers attended from ACCESS Japan, including Hiroyuki Toki, who assisted with data collection and analysis.

Test setup and hardware: Salzburg, Austria

The Salzburg study was conducted in a formal usability lab. The user operated a PC simulation, following printed instructions. Assistance was available if needed. The on screen display, overhead view, profile view, and context view were projected in a conference room for observers.

Test setup and hardware: Cape Town, South Africa

The Cape Town study was conducted in two joining rooms at the University, connected via video. We used one room as the experiment room, and the other room as the observer room. The observer room also served as the 'from caller' location for the call handling and multitasking parts of our test.

A monitor and live audio in the observation room was critical for researchers to be able to follow the test, take observation notes, and participate authentically in the live calling tasks since the test was based on the local telephony network and was not a 'wizard of Oz' type of constructed experience.

Figure 2. Observation and experiment room

Figure 3. The prototype phone with painted keys:

Options key (dotted circle in the upper left)
Back key (arrow in the upper right)
five-way rocker
center press (OK)

WhiteOut paint job courtesy Professor Gary Marsden, our congenial host!

Our prototype was run on an actual mobile phone equipped with a five-way rocker and multiple hard keys, not all of which were used. We had no control over the physical device's industrial design. The hard keys had screened-on graphical labels which we found confused users. After clearing our tasks through two pilot participants, we commenced the test, but after three more participants we recognized that users were paying a lot of attention to the irrelevant screened-on glyphs. Most notably, the hard keys on the phone did not map directly to available functions in the prototype. We mitigated this problem by painting over the device's screened graphics and drew new glyphs that accurately represented our hard key operations. This had two effects: users could much more easily predict the outcome of key presses, and they also realized that despite very realistic interaction, the device was, after all, only a prototype (Figure 3). This was important because the prototype's responsiveness was uneven, and rapid successive key presses could present seemingly irregular results. We asked all participants to press keys somewhat slowly in order not to confuse the prototype, since it was still in development. This had the added benefit of helping users understand that we were testing the prototype, not the participants. In general, the painting approach can be seen as a very useful strategy when exactly matching hardware prototypes are not available - which is often the fact with mobile devices

Equipment

Several tasks of the test involved live calls. We fixed the prototype to the sled of a Noldus camera that sat on the desk in front of the participant. We captured audio and visual data through a portable lab, piped that into the observation room, and afterwards burned a DVD of each session which we filed with observation notes, participant questionnaires, and task result coding sheets filled out by three of the four researchers for each participant. The prototype was permanently set to speakerphone so that we could hear both sides of conversations through our observation equipment. This allowed us to record the entire communication between the participant and the researchers. The speakerphone was permanently set to a high level to ensure we would be able to track the participant's conversations and activities. Over the course of running twenty participants through the test, we realize it would have been sufficient (and less painful) to set a lower volume level.

Test Objective

The main objective of the study was to evaluate the new interaction paradigm using the hard keys defined above as opposed to the dual soft key approach. We were interested in how easy it is for users to acclimate to the new concept. In addition, we wanted to know to what extent does the switch from soft keys to hard keys present a burden for the participants? Conducting a test in a region where many of the participants did not have previous computer experience was essential: we hypothesized that users with little or no computer experience were able to quickly pick up the concept. We were both satisfied, and surprised, at some of the results.

Participants

One of the main reasons for choosing to conduct the study in South Africa was the diversity in the population in terms of mobile phone usage. We targeted mobile phone users covering a broad mix of mobile phone experience, educational, and professional background. All participants were recruited on site in Cape Town based on our screener. Participants were each paid 40 South African Rand (approximately USD$7) for two hours with us. Out of the 20 recruited participants, 15 were counted in the test results.

We worked with 7 female and 8 male participants ranging between 21 and 41 years old (mean of 29.8 (sd=5.7)); 7 participants spoke English as their first language, and for 8 participants English was their second language; 4 of the subjects spoke Xhosa as their first language, 3 Afrikaans, and 1 Sesotho.

Although all subjects had used mobile phones before, their experience with the handsets and the breadth of their experience varied. Their mean duration of mobile phone usage was 5.7 years (sd=3.1 years), changing to a new phone every 1.6 years (sd=0.8 months). Three of the participants (all university people) had previously used handhelds other than mobile phones. In the pre-interview the participants self-assessed their mobile phone experience: 3 participants categorized themselves as novice users, 7 as intermediate, and 5 as sophisticated. All 15 participants used their mobile phones for handling calls, sending and receiving SMS, clock and alarm functions, as well as managing contacts. All but one participant had their mobile phones with them all the time. Nine of the participants purchased their phone.

The profession of the participants varied greatly as was intended by the study. We wanted to cover a very broad range, skewing towards non-sophisticated users as much as possible. There were six students of various disciplines, two secretaries, an administrative assistant, a librarian, a cashier, a janitor, a grill cook, a food service assistant, and one associate professor who ran through the tasks as a time trial.

Some context about Cape Town and South Africa

Cape Town is a beautiful, vibrant, and dangerous place. Following is data collected in a census in 2011 and added here for context (our study was conducted in 2005, with no recent census data available at that time).

University of

Cape Town

Cape Town is a large metropolitan area. But explore beyond the city center and you'll find the townships. Whether home is a shanty or a high rise apartment, most people (91.3% as of 2011) have a cellphone.

91.3% have a cellphone
37.9% have a computer
49.3% have access to the Internet (either through a computer or a cellphone)

78.4% are in formal structures (houses or flats)
20.5% are in informal structures (shacks)

94.0% of households use electricity for lighting
87.3% of households have piped water to the dwelling
12.0% have piped water through a communal tap.
94.9% of households have regular refuse collection service
91.4% of households have a flush toilet or chemical toilet
4.5% still use a bucket toilet
82.1% of households have a refrigerator
87.3% have a television
70.1% have a radio

South Africa's Languages

Throughout apartheid, a period of intense racial division, South Africa claimed only two official languages: English and Afrikaans. Nine more indigenous African languages were officially recognized with equal status when Nelson Mandela came to power and mandated it in the 1997 Constitution: Afrikaners were unwilling to give up Afrikaans as an official language. Therefore, for equality among other African langugaes, the 11 languages listed here were all made official languages.

Today, many South African children grow up in households where more than one African language is spoken; it’s not unusual for different family members to consider different languages to be their first.

According to the 2011 census, the 'home' or 'first language' breakdown looks like this:

22.7% isiZulu
16.0% isiXhosa
13.5% Afrikaans
9.6% English
8.0% Setswana
7.6% Sesotho
5.0% one of the remaining languages

(From Project Literacy. Read more here.)

Cape Town's population

With 3.7 million residents, this is the second most populous South African city after Johannesburg, according to the South African National Census of 2011.

The sex ratio is 96, meaning that there are slightly more women than men
45.4% of the population described themselves as "Coloured"
42.7% as "White"
8.6% as "Black African"
1.4% as "Indian or Asian"

Education

Of those residents aged 20 or older, 46.6% have at least a Grade 12 education. Of those aged between 5 and 25, 67.8% are attending an educational institution. Overall:

1.8% have no schooling
8.1% have some schooling but did not finish primary school
4.6% finished primary school but have no secondary schooling
38.9% have some secondary schooling but did not finish Grade 12
29.9% finished Grade 12 but have no higher education
16.7% have higher education.

Cape Town city logo update

City Council nominates ad agency to create 'more inclusive, less passive' city identity.' From 'we work for you' to 'let's do it together, losing the iconic Table Mountain profile triggers community debate.

Cell C is a mobile operator offering community chat at kiosks like this one, made from a half-size cargo container. Neighbors drop in to charge their phone or use a provided device. Mobile phones are objects of desire, and represent sophistication, financial security, and style.

Many township residents hop on a jitney bus for work in the morning for a commute to the city. Despite primitive conditions at home, they live in an otherwise first world environment, with mobile phones providing the dominant means of connecting to the internet. After our test, cell phone use surged in Africa. Pew Research estimated that by 2014, 34% of South Africans had a smart phone compared to 64% in the US. For more details see the study here.

Listen to Miriam Makeba sing 'The Click Song' (Xhosa)

South Africa • land of contrasts

Procedure and Tasks

After conducting an in-depth interview prior to the test, the facilitator explained the system including the UI components and the corresponding hardkeys. The subjects were shown how to use the system and encouraged to refer to a poster explaining the interaction fixed to the wall immediately in front of them. There was no training or free exploration phase, since we were interested in how easy it was to adopt the system and how quickly the users can pick up the new concept.

We had participants perform eight tasks in total. The tasks were diverse in complexity and the length of time we expected each to require. The sequence of the tasks was randomized for each participant in order to prevent learning effects. Four of the tasks involved live calls with one or two people (made to or received from researchers in the observation room).

After the tasks, a qualitative interview was conducted on how the users felt about their experience with the system, and what they understood about how it worked. We answered any questions and illustrated successful methods to complete tasks when asked.

Using the Options key

Participants had almost no difficulties in using the Options key to access (open and close) the Options panel. In very few cases (4 out of 104 relevant instances), participants did not manage to use the Options panel. They were unsure whether using the Options key would produce the right effect. In comparison to the Back key, the participants were even more skeptical about the Options key. Prior to the test, more than one half of the participants (56%) expected the Options key to be tricky, confusing, difficult or unintuitive. They did not know what functionality to expect. However, after the test, all participants had positive valuations of the Options key, stating that it was clear, easy, efficient and a nice mechanism (Figure 5). We were surprised that even users who clearly did not grasp the concept and had difficulties completing tasks responded with enthusiasm when they were shown how to use the Options key after the test was concluded.

Figure 5. Valuations of the Options and Back keys before and after the test.

Horizontal navigation in tabs

We found that the participants had some slight difficulties with horizontal navigation in the tabs employed in the Contacts (phone book) application. In 7 out of 30 relevant instances, participants did not manage to navigate through the tabs. The main reason was that they did not visually recognize the tabs or if they did, the metaphor had no particular meaning for them. Participants expressed that navigating left and right in the tabs was straightforward, once they recognized the tabs as neighboring dividers. This suggests that the navigation concept itself is not difficult. Rather it is a design issue to make the tabs more visually meaningful to the users. We feel that in a full system where this mechanism is repeatedly employed, consistent exposure to the same visual definitions would increase the ability of users to use them successfully.

Horizontal navigation in the Options panel

The subjects had the most difficulty navigating horizontally across the Options Bar. While opening and closing the panel using the Options key was not a problem, navigating from the open panel right to the next panel was not at all obvious to the users. In 29 out of 72 relevant instances, participants were not able to navigate across the Options Bar. The reason for this was that the users did not recognize that it was possible to move left and right once the panel was open; additionally, many users who were unfamiliar with a five-way rocker never pressed the right or left rocker keys. They expressed that they could clearly see the icons in the Options panel, as well notice them appear and disappear, and they knew that was where they wanted to navigate, but they could not figure out how to get there. Once the moderator showed them how to move left/right from the open Options panel, participants found it fairly easy to use. Similar to the horizontal navigation in tabs, the horizontal navigation in the Options panel can be overcome by providing clear (visual) design, making the navigation possibilities more obvious to the users. The recommendations from the collective results suggest that the entire Options Bar should highlight when the Options button is pressed; and adding a left-facing arrow to the left of the Options icons and a right-facing arrow to the right of the Options icons would reinforce the navigation possibilities.

Focus on novice users

We were particularly interested in the results of novice users. The users with the most computer experience had no difficulty at all with the system and beat our time trial results by a significant margin with ease. But we expected that. More interesting is how the novice users fared. Comparing the results of novice users to all users shows that novice users were able to pick up the concept to a similar extent as sophisticated users. Users' cell phone experience was self-assessed during the introduction. Novice users had little or no computer literacy, did not own a computer, and had no e-mail address. Figure 6 shows the success rates of novice users when compared to all users of the test. The rates for using the Options key as well as navigating with the tabs are slightly lower, but still comparable to the overall results of all users. The very low rate for the horizontal navigation in the Options panel is obviously a consequence of the 5-way rocker difficulty that some users had experienced. During the tests, we observed that users who had difficulties in using the 5-way rocker for navigation (predominantly novice users), were very cautious and considerate, often lingering over the buttons before pressing them. The missing experience with such navigational patterns combined with the design issues for the navigation in the Options panel contributed to the low success rate.

Discussion

We have been using the Standard Usability Scale (SUS) [2] questionnaire in order to capture subjective impressions based on users' experiences with the system. We have learned through the pilots that the participants had difficulties to understand the questions contained in SUS. Problems occurred with the terminology and they were not able to associate it with a mobile phone. A probable reason is that many participants had little or no exposure to personal computers and other consumer systems. Thus for the future, we have to consider extended methodologies to be adapted before they can be applied in regions with population that has entirely different background experiences. One of our tasks requested users to navigate to the About Box and tell us what year an application was copyrighted. It did not take too many blank stares for us to realize that an About Box is meaningless to individuals with little or no computer experience, yet most still completed the task successfully.

We realized that the main method of controlling the interface is extremely important: if users had never seen or used a 5-way rocker, it really doesn't much matter that arrows are drawn on the buttons indicating the direction in which focus will move. It simply did not register. In particular, some participants had a hard time confirming a selection using the center press. In future tests, we will consider providing a more in-depth and experiential, rather than pedagogical, training on issues that we might consider daily routines.

Since many of the tasks involved live calls, we bought three local prepaid SIM cards for the test. We used a total of three handsets during the test - one for the participant running the prototype, as well as two other phones for the researchers who communicated with the test participant providing tasks and questions to them. Our prototype system did not permit editing of the phonebook contacts (although we did manage to delete one mid-test). For the test, we used fixed numbers assigned to the local SIM cards. We mapped the SIM card numbers to the existing phonebook entries prior to the test. It soon became obvious that the credits were used very unevenly with different participants: depending on the task, calls were initiated from either the participant or the helpers. We had to introduce a rather sophisticated management of credits and SIM cards. We purchased credit vouchers with different values so that we could opportunistically recharge the phones that had run down the most after each session. After each session, we checked the credits of all SIM cards and re-charged the minutes when necessary. Simply switching the phones did not work, since the numbers assigned to the persons in the phone contacts would then no longer match the tasks. Establishing local long-term usage numbers was not practical for us - but now we have a better idea of how well to prepare for each individual two-hour session.

Our tasks did not assume that users understand how network telephony works, but we observed some strong biases not to trust the ability to put a call on hold and take or initiate another call at the same time. Multi-line calling and conference calling are activities that were alien to most of the participants. In one task, participants were asked to call for instructions, and then were told to put the researcher on hold, and call another researcher to get some information to pass back to the first researcher. Most frequently our participants successfully put the original call on hold, called the second researcher, then failed to navigate back to the original call. Somewhat frustrated, they simply placed another independent call to the first researcher. The effect this had on the test was interesting: participants could indeed see that two calls were active from the feedback on the screen, but felt they satisfactorily completed the task if they were able to pass on the information, regardless of how they managed to do so. They frequently left the original call on hold until the end of the task. Needless to say, this became expensive for us in terms of telephone minutes and we found ourselves making daily trips to the local shop to purchase top-up coupons to add minutes to our local SIM cards.

Based on our experiences during this evaluation, we can express following recommendations:

Carefully consider the tools and methods to be applied during a test, especially if tests are conducted with an unfamiliar user population. Adapting methods and tools might be necessary prior to applying them.
Check whether the participants are familiar with the important concepts of the system under testing. Some obvious things might not be as obvious for the participants as one might expect.
Get acquainted with local infrastructure and its quirks. Some things might need more preparation and management for which resources should be planned and available.

End Notes

Our study highlighted many issues particular to telephony and call handling, and significant interactions between the users' mental model of the interaction paradigm, the hardware design, and users' mental model of call handling in general. We conclude that a close and successful match of hardware to software is imperative. This brings up an interesting issue: in a market where both handset manufacturers and mobile operators compete through distinctive differentiation, is it possible to engage with a new interaction paradigm? What is the most important differentiator: device industrial design, graphic treatment, or interaction design?

We submit that users can acclimate to a new interaction paradigm with ease. In our pre-test questionnaire, it was clear that industrial design ("I want a slim phone that will fit in the pocket of my tight jeans") is something that is important to users that they understand. However, users don't generally have much choice in the interaction paradigm, and few users had ever thought an alternative was possible. Most said they would purchase this phone if it were commercially available because they found it surprisingly easy to use. But creating an easy to use mobile navigation model requires the cooperation of both handset manufacturers (providing appropriate hardkeys appropriately labeled) and operators, without whom new handset designs only rarely come to market.

It is worthwhile to understand the ecology of device mobility. The dominant operating system for mobile phones is Symbian, an 'invisible' OS that is customized for each handset model. Competing operating systems include Windows Mobile and Palm OS in the wireless handheld category of devices - these are, however, PDA experiences with telephony features rather than mobile phones with extensive add-on applications. According to a recent Global Wireless Research Sector Report, "for a manufacturer looking to invest in an operating system to support the increasing complexity of its current and future handsets, there are essentially five choices: BREW, Microsoft, Symbian, Linux-based OSes and Nucleus-based stacks. Most Tier-1 manufacturers are today investing significant R&D efforts on Linux-based OSes." [4] Operating systems such as Qualcomm's BREW or Trolltech's Qtopia provide extensive customization opportunities to mobile operators and device manufacturers. But all of them use the de facto softkey paradigm despite a broad variety of custom visual effects, branded screen layouts, and carrier-specific tools and on line services. Changing an interaction paradigm means introducing some measure of financial risk. A new handset design generally takes 12 to 18 months to come to market. That is a considerable expense. The introduction of a new interaction paradigm requires mobile operators and manufacturers to support the concept. To do this with justification, all partners in the mobile product development space must be willing to try something new. However, our experience is that although each strongly desires to be unique and clearly differentiated from one another, they are equally risk averse.

Overall the reaction to our prototype was positive; some flaws have to be corrected, mostly visual design issues; the navigation concept is promising for a broad market segment ranging from entry-level to high-end. But ultimately it will be the ecosystem that approves or disapproves. Subsequent prototype development and user exposure will add to the body of evidence that it is time to change the paradigm.

Conclusion and Future Work

Our next steps are to correct the visual design issues in our prototype and re-evaluate. We are also investigating the scalability of the overall concept; research questions include how to make deep navigation structures much shallower through distribution across neighboring panels, employing tabs or "related views" within an application. We are also considering regional distinctions in successful use of this paradigm, and we are looking at other distinct user populations to evaluate the concept, primarily in Europe and Asia where more services are commonly used than in North America. This will also result in an international comparison of the novel interaction concept.

Acknowledgments

We would like to express our thanks to Leo Ah Kun from the University of Cape Town, and Hiroyuki Toki from ACCESS Co., Ltd. for their support.

References

Ahlström, D. (2005). Modeling and improving selection in cascading pull-down menus using Fitts' law, the steering law and force fields. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Portland, Oregon, USA, April 02 - 07, 2005). CHI '05. ACM Press, New York, NY, 61-70.
Brooke, J. (1996). SUS: a "quick and dirty'' usability scale. In: Jordan, P. W., Thomas, B., Weerdmeester, B. A., and McClelland, I. L. (Eds.). Usability Evaluation in Industry. London: Taylor & Francis. 189-194.
Cellular (2006). Statistics Of Cellular in South Africa. http://cellular.co.za/.
Constantinous, A. and Lewis, M. (2006). Mobile Operating Systems: The New Generation. Global Wireless Research Sector Report, ARCchart Ltd, London, UK.
Geven, A., Sefelin, R., and Tscheligi, M. (2006). Depth and breadth away from the desktop: the optimal information hierarchy for mobile use. In Proceedings of the 8th Conference on Human-Computer interaction with Mobile Devices and Services (Helsinki, Finland, September 12 - 15, 2006). MobileHCI '06, vol. 159. ACM Press, New York, NY, 157-164.
Gong, J. and Tarasewich, P. (2005). Alphabetically constrained keypad designs for text entry on mobile devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Portland, Oregon, USA, April 02 - 07, 2005). CHI '05. ACM Press, New York, NY, 211-220.
Goodman, J. (2005). Linking mobile phone ownership and use to social capital in rural South Africa and Tanzania. In: Moving the Debate Forward: The Vodafone Policy Paper Series, No. 2, Africa: The Impact of Mobile Phones (Vodafone Policy Paper Series).
Hyvärinen, T., Kaikkonen, A., and Hiltunen, M. (2005). Placing links in mobile banking application. In Proceedings of the 7th international Conference on Human Computer interaction with Mobile Devices & Services (Salzburg, Austria, September 19 - 22, 2005). MobileHCI '05, vol. 111. ACM Press, New York, NY, 63-68.
Jones, M. and Marsden, G. (2006). Mobile Interaction Design. John Wiley & Sons.
Kiljander, H. (2004). Evolution and Usability of Mobile Phone Interaction Styles. PhD thesis, Helsinki University of Technology, Espoo.
Kuhn, T. S. (1974). The structure of scientific revolutions. Chicago, Ill.: University of Chicago Press.
Lindholm, C., Keinonen, T., and Kiljander, H. (2003). Mobile Usability – How Nokia Changed the Face of the Mobile Phone. New York et al.: McGraw-Hill.
Lyons, K., Starner, T., Plaisted, D., Fusia, J., Lyons, A., Drew, A., and Looney, E. W. (2004). Twiddler typing: one-handed chording text entry for mobile phones. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vienna, Austria, April 24 - 29, 2004). CHI '04. ACM Press, New York, NY, 671-678.
Marsden, G., Thimbleby, H., Jones, M., and Gillary, P. (2002). Data Structures in the Design of Interfaces. Personal Ubiquitous Comput. 6, 2 (Jan. 2002), 132-140.
Pavlovych, A. and Stuerzlinger, W. (2004). Model for non-expert text entry speed on 12-button phone keypads. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vienna, Austria, April 24 - 29, 2004). CHI '04. ACM Press, New York, NY, 351-358.

<< BACK TO PORTFOLIO