An International Localization Case Study in Four

Successful Mobile App Localization Requires User Research

Parts; Part Three

What are the requirements for a mobile app to be successful in multiple languages? This is the third post in which we address that question as we describe UX research on localization  that TecEd and Cisco worked on together.

Cisco Systems developed a mobile application, SalesConnect, for its internal sales force of tens of thousands, as well as partners across the globe. Before investing in costly localization, the team wanted to learn:

What is most important to sales users for a localized app?
When and why do these users want materials in English versus in their local language?
When do they search using English versus their local language?
This case history, originally presented at the 2016 ACM SIGCHI conference, describes how Jennifer Lee Carlson and a multidisciplinary team at Cisco worked with TecEd on this research project to learn what constituted a good localized application for sales users’ content needs. It describes the methods we used, the challenges we overcame, and what we learned.

Part 1:  Introduction

Part 2:  Project Work Process, What We Did Steps 1-4

Part 3:  Project Work Process, What We Did Steps 5-9

Part 4:  Lessons Learned and Key Takeaways

Part 3: Project Work Process, What We Did Steps 5-9

The demands and constraints of a multinational project had a major impact on our work process. Some activities required additional steps, and almost all were more complex than we initially anticipated. To summarize, we:

1. Designed the research
2. Engaged bilingual partner researchers in different countries
3. Coordinated with Cisco cross-functional teams
4. Created a “translation-ready” research protocol
5. Designed data-logging templates for consistent reporting
6. Ran pilot-test sessions in each language
7. Conducted 22 participant sessions
8. Translated the session recordings
9. Analyzed and reported the research results

5.    Data-logging templates for consistent reporting

We wanted to collect comparable data from our three groups, despite their differences in language and culture.  To manage inter-rater reliability as much as possible, we created a data-logging template in Excel, with detailed instructions to the partner researchers for filling out the Excel data logs. For example, we instructed:

–        Enter participants’ answers into the data-logging template in English. You can take notes using the protocol script and data-logging template in whatever way works best for you, but the data must be provided to us in English using the Excel spreadsheet when you send it to us.

–        If a participant has no answer for a particular question (for example, because the question does not apply to that participant), please enter N/A (for “Not Applicable”) in the cell for that answer.

–        If you did not ask a question (for example, because there was no time remaining in the session), please enter No Data in the cell for that answer.”

Having structured data logs facilitated the data analysis, and helped maintain consistent reporting from researchers conducting sessions in three different languages.

6.    Pilot-test sessions, one in each language

TecEd had many discussions with the partner researchers and with Cisco about the best methodology for pilot-testing.  Among the options we considered were simultaneous (real-time) translation by a second bilingual speaker and consecutive translation with the researcher summarizing in English every minute or two.

Unfortunately, simultaneous interpretation was both costly and posed technical challenges because researcher, participant, and stakeholder teams were in multiple time zones. However, consecutive translation would have extended the session length past the amount of time the sales participants could devote.

We ultimately chose a simpler approach of the researcher debriefing in English with the TecEd and Cisco teams immediately after the pilot-test session, using his or her notes from the session to describe what happened.

7.    Conducting the sessions

After extensive scheduling—and rescheduling—our final research sessions included 22 participants: 7 speakers of French, 7 of Mandarin Chinese, and 8 of Spanish.

The French speakers were located in France and Quebec. The Chinese speakers were located in China, including one in Hong Kong. The Spanish speakers were located in Chile, Colombia, Costa Rica, and Puerto Rico, as well as one in Miami (US).

The participants represented several roles within the Cisco sales organization; most were account managers or systems engineers. About half the participants worked directly with Cisco customers, and the rest worked with Cisco partners (companies that sell Cisco products).

To manage the session facilitation process, we maintained a detailed spreadsheet of participant sessions. Each participant’s session time was listed in four time zones: the participant’s, the facilitator’s, the TecEd researcher’s, and the Cisco stakeholders’.

8.    Translation of Session Recordings

After the sessions in each language were completed, we had the WebEx session recordings translated into English. The method we chose was Interpreted Transcription, in which the original-language audio files are directly interpreted by a professional translator into English without the use of an initial source-language transcript.

This approach was faster and more cost-effective than a two-step process (transcribing in the native language, then translating the transcription). Although not suitable for legal documents, interpreted transcription is a good research tool, especially since the partner researchers were available to resolve any questions about the transcriptions.

9.     Data analysis and results reporting

The first results deliverable to stakeholders was the compiled English-language data logs, because those did not require translation. After reviewing the transcriptions, we then drafted the project results report and sent the relevant portions of it to each partner researcher for review. After receiving the feedback from the partner researchers, we completed the detailed results report. The final report included 15 graphs of participant preferences and hundreds of participant quotations.

Next Installment  Part Four: Lessons Learned and Key Takeaways

The case study “What Makes a Successful Localized Mobile App” that is presented in four separate blog posts was written by Stephanie Rosenbaum, CEO of  TecEd, Inc. and Jennifer Lee Carlson,  User Experience Manager,  Cisco Systems, Inc..  For more information on localization or other user research project needs you may have please Contact Us.