Role: UX Researcher
Timeline: 2 weeks
Mobile Platform: iOS + Android
Timeline: 2 weeks
Mobile Platform: iOS + Android
Puzzle Hospital is a mobile Medical Puzzle Match 3 game, available on both iOS and Android. Players have the ability to customize their hospital and follow a gripping storyline through playing over 200+ puzzles.
The problem
Data showcased a dip in retention on Day 7.
The game team wanted to understand why our retention numbers were not reaching our target.
To bring qualitative understanding to the data presented, the UX team and I had to come up with research methods to answer the teams' questions to figure out how to better their players' experiences.
The game team wanted to understand why our retention numbers were not reaching our target.
To bring qualitative understanding to the data presented, the UX team and I had to come up with research methods to answer the teams' questions to figure out how to better their players' experiences.
Our Constraints
2 weeks, financial budget and a bandwidth issue.
We had to present the results in 2 weeks as we wanted to produce high quality results, quickly and efficiently, to provide the dev team with enough time to implement changes before their next release. There was also limitation on the type of research tools that we could use due to the financial budget provided for us.
This was a project that I took ownership on. As our UX department is a centralized team that consists of 3 members, my other team members were preoccupied with another project and a 3-day training. Of course, we had a great support system and helped one another to the best of our abilities to lessen the workload of each others'.
My Role
I took full ownership of this user research project.
I took full ownership of this user research project.
As written above, the UX team was low in bandwidth. As a result, I had to manage the research from recruitment to interviews and to compiling all the results on my own, while remaining in constant communication with the game teams and their research needs. However, I had my teams' help for the first day in brainstorming the best research to understand why there is a dip in Day 7 retention.
Team collaboration
We brainstormed on the best research method to understand the dip in retention, as our previous in-house play study methods were not suitable.
We brainstormed on the best research method to understand the dip in retention, as our previous in-house play study methods were not suitable.
We realized that having recruited players come to our office to play for 7 days did not provide an authentic understanding of their interactions with the game. This setup failed to capture how players would engage with the game in real-life scenarios, such as during transit, at home, or in other natural environments. To truly understand user behavior and retention, we needed a research method that better reflected their everyday experiences.
Our previous multisession play study we had for another game.
We had previously attempted to create a natural setting in the office by allowing players to watch Netflix or read while playing, and used Tableau to analyze median play session times. Despite these efforts, the setup still failed to accurately reflect how players interacted with the game or felt about it in their everyday lives.
Remote study was essential.
It was clear that we needed a research tool that would allow players to play the game at their own time, wherever and whenever. We needed a tool that allowed us to see how they play our game on their phones while allowing us to hear their thought process, simultaneously.
We dispersed to look for reliable remote tools within our budget.
I researched various remote research tools by reviewing Google feedback, consulting with user researchers on DesignX, and comparing pricing packages. I also arranged demo calls with companies to test their products. After evaluating the options, I presented my findings to the team. We decided that Lookback best met our needs with its essential features and affordability.
✅ Record players' screens in real time across various environments (ex. transit, library)
✅ Captures video and audio to hear players' thoughts during gameplay.
Recruitment
I led the recruitment process and analyzed the in-game survey for Puzzle Hospital to understand our players' demographics.
I led the recruitment process and analyzed the in-game survey for Puzzle Hospital to understand our players' demographics.
Through looking into our in-game surveys, I was able to look into their age, gender, games they play, and other details to understand who our demographics were.
I began recruitment for fresh players who matched our players' profile, who have never played Puzzle Hospital before. It was important to recruit for players who have never played the game so that we would be able to gain a fresh perspective on their experience of the game.
How I recruited players:
I leveraged a list of interested players that I compiled through previous Facebook campaigns with the User Acquisition team. This list included players from various game genres. When recruiting for Match3 playtesters, I used this existing list to quickly identify and contact the right candidates, skipping the need for new advertisements and saving time.
I began recruitment for fresh players who matched our players' profile, who have never played Puzzle Hospital before. It was important to recruit for players who have never played the game so that we would be able to gain a fresh perspective on their experience of the game.
How I recruited players:
I leveraged a list of interested players that I compiled through previous Facebook campaigns with the User Acquisition team. This list included players from various game genres. When recruiting for Match3 playtesters, I used this existing list to quickly identify and contact the right candidates, skipping the need for new advertisements and saving time.
Minimizing the room for error
I organized orientation day for smooth remote playtesting.
Excerpt of my orientation slides
Given the tight deadline and our first remote study, I organized an orientation day to guide playtesters on how to conduct the study using Lookback. This session aimed to minimize errors by explaining the process and addressing any questions. I also tested the presentation with colleagues to ensure it was clear and user-friendly for the playtesters.
Communication with the Game Team
I ensured ongoing collaboration with the game team for timely insights.
I ensured ongoing collaboration with the game team for timely insights.
During the study, I regularly updated the game team with daily summaries of my findings. Since they were making continuous changes to the game, I provided high-level insights to guide their next release. Staying in constant touch ensured I could address any additional questions they had throughout the 7-day study.
The Dilemma
Players faced ongoing technical issues with Lookback during remote playtesting.
Screenshot of our remote study with Lookback.
Lookback encountered issues, causing game lag or failures to load recordings. To address this, I contacted Lookback multiple times to report these problems and seek solutions to ensure we could successfully capture the playtest sessions.
Despite contacting Lookback for solutions, we couldn’t resolve the technical issues. With players' videos not recording, I quickly sought alternatives. I explored other recording apps but found them too complex to teach players, as assuming they were tech-savvy might lead to further complications.
Rerouting for a quick solution
I opted for players to use their phones' screen recording feature, despite the lack of audio recording capability. At the time, iPhones couldn't record audio simultaneously with the screen.
I opted for players to use their phones' screen recording feature, despite the lack of audio recording capability. At the time, iPhones couldn't record audio simultaneously with the screen.
I had to choose between capturing players' in-game actions or their verbal feedback. Given that we already collected their thoughts through diary journals, I decided that observing their gameplay was more crucial for understanding their user journey.
To address the missing audio
I contacted players by phone daily to gather their feedback.
As we needed to know their thought process leading up to Day 7, when retention drops, it was important to understand deeply on how they feel about the game each day. Since Lookback wasn't working for some players, I took more time analyzing their user journey through the screen recordings and wrote down questions that I had, wondering why they decided to take a specific route/action that they need, and etc.
I contacted players by phone daily to gather their feedback.
As we needed to know their thought process leading up to Day 7, when retention drops, it was important to understand deeply on how they feel about the game each day. Since Lookback wasn't working for some players, I took more time analyzing their user journey through the screen recordings and wrote down questions that I had, wondering why they decided to take a specific route/action that they need, and etc.
I also looked into the surveys we send everyday as part of our diary study component. Through the surveys, I analyzed their responses, read through them, and put their responses side by side with their behaviour as they played the game. Through analyzing that way, it allowed me to gain a deeper understanding on why they play the game the way that they did and why they thought the way that they thought.
Another Dilemma
Due to technical bugs, some daily journalling surveys didn't get sent, but the Lookback videos did.
To gain insights into their decision-making, I created follow-up questions over the phone to understand their actions better, asking why they took certain steps or spent time on specific details. This approach allowed me to address the missing survey data and provided valuable insights into their thought processes. By thoroughly exploring these aspects, I ensured that the study's findings were comprehensive and actionable, even with the technical challenges.
To gain insights into their decision-making, I created follow-up questions over the phone to understand their actions better, asking why they took certain steps or spent time on specific details. This approach allowed me to address the missing survey data and provided valuable insights into their thought processes. By thoroughly exploring these aspects, I ensured that the study's findings were comprehensive and actionable, even with the technical challenges.
Time management
I had to schedule out how the calls would work as 4 out of 5 players were working individuals with limited time availabilities.
Since I was on my own, I had to manage my time and figure out how I would conduct these studies by myself since some of the players' time availabilities overlapped one another.
For example, on some days, I had 2 players were only available from 1pm-2pm. As a result, I would only have 30 minutes to talk to each player (my individual calls with each player would normally be for an hour). As a result, I had to prioritize which questions were the most vital to be answered within the limited time span.
Not only that, but I had to ensure that I watched their remote videos that would range from 10 minutes to 1.5 hours long, before the phone calls.
Not only that, but I had to ensure that I watched their remote videos that would range from 10 minutes to 1.5 hours long, before the phone calls.
I created a to-do list every morning to ensure that I had proper time management to conquer this situation.
After the 7-day remote study was over, I had our playtesters come into our office on the 8th day and had them do a wrap-up session to inquire about their overall 7-day experience. For the wrap-up, I also invited Producers and the Lead Game Developers to join in so that they could observe and ask any specific questions that they may have.
I led the meeting allowing them to know all my findings and what I found insightful throughout the 7 days. By then, my team member was back from her workshops and we decided to divide the remaining work between the both of us. She took all my findings and created presentation slides to present to the game team.
The return of my teammate
By the time I compiled a majority of the results, I booked a meeting with my team members and presented them my findings.
By the time I compiled a majority of the results, I booked a meeting with my team members and presented them my findings.
I led the meeting allowing them to know all my findings and what I found insightful throughout the 7 days. By then, my team member was back from her workshops and we decided to divide the remaining work between the both of us. She took all my findings and created presentation slides to present to the game team.
As she was creating the presentation slides, she often asked me questions for any detail she was unsure of. It was easy to be in constant communication as we sat right next to each other.
While she was creating the presentation slides, I rewatched some of the recordings again. I collaborated with the Data Analytics team to see which levels our players failed at and why, as they had also been tracking their journey through their Player ID. We wanted to understand why players failed at certain levels that they failed at and if it was related to any usability issues.
As a result, I watched over 95 videos that were around 30 minutes to 1.5 hours long, each.
While she was creating the presentation slides, I rewatched some of the recordings again. I collaborated with the Data Analytics team to see which levels our players failed at and why, as they had also been tracking their journey through their Player ID. We wanted to understand why players failed at certain levels that they failed at and if it was related to any usability issues.
As a result, I watched over 95 videos that were around 30 minutes to 1.5 hours long, each.
Our Findings
I will only be able to share high-level findings, due to the restrictions of my NDA.
● A strong conflict is needed in the beginning of the storyline to capture the players' attention.
● Micro-animation of gems being matched together is important for players to feel a celebratory moment over their achievement.
● Readability of the puzzles is highly important to ensure there is enough contrast or else it will be difficult for players to differentiate between different gems.
● Players like a high degree of customization features in the game.
● Players say the puzzles in our game is too easy. However when the team decided to tune to the game into a harder difficulty, data showed that there was a dip in retention.
Impact on the Game Team
Based on the user research, the game team realized that the cost and long return on investment (ROI) for improving Puzzle Hospital were too high, leading them to abandon the game and start a new project in a different genre.
Based on the user research, the game team realized that the cost and long return on investment (ROI) for improving Puzzle Hospital were too high, leading them to abandon the game and start a new project in a different genre.
With the user research findings that was found, it was obvious to the game teams that the changes needed for the improvements would cost too much. Especially when connecting with the marketing team, the game team recognized that the amount to buy puzzle players is too high that our company would not be able to see a return in investment and make a profit until after a year.
Even though Puzzle Hospital has high retention and engagement, it also has a long Return on Ad Spending (ROAS) and low Average Revenue of Daily Users (ARPDAU). Typically puzzle players in the market don't spend in games, until mid to late levels, which means our team would need to improve the engagement of the game for a number of active days in order to see that ROAS and heightened ARPDAU.
As a result, the team decided to drop Puzzle Hospital and start a new game in a different genre.
Even though Puzzle Hospital has high retention and engagement, it also has a long Return on Ad Spending (ROAS) and low Average Revenue of Daily Users (ARPDAU). Typically puzzle players in the market don't spend in games, until mid to late levels, which means our team would need to improve the engagement of the game for a number of active days in order to see that ROAS and heightened ARPDAU.
As a result, the team decided to drop Puzzle Hospital and start a new game in a different genre.
What I learned
Collaborating with other departments is key.
Through this project, I realized that many departments have resources that can be offered to help create the most suitable research method. Having collaborated with Player Support and the Data Analytics team, I'm excited to see what other supporting departments have to offer, such as the Marketing team, I.T., Quality Assurance, etc. Not only does it help make our studies better, but it helps the company become more unified as we work together in improving our games.