Role: UX Researcher
Timeline: 4 weeks
Idle Playbook is a competitive analysis of 4 top idle mobile games in the market.
An idle game is a type of game where players make progress by performing simple actions, such as clicking, to earn resources. Over time, the game continues to generate resources even when not actively played, allowing players to achieve goals with minimal continuous effort.
Objective
We wanted to understand why these top games have high day 7 retention numbers.
By understanding the why, we would be able to understand how we can make our idle games just as great and better.
My role and reasoning for this book
I wanted to help the game teams create a successful idle game from scratch. As a result, I took initiative and created this playbook as the sole UX researcher.
UX research in the company, at the time, was not fully valued from the entire company. As a result, I wanted to create a competitive game analysis that was user friendly and insightful enough that the game teams won't be able to ignore it, because of how easy it is to read.
My purpose for this book was to help the game teams:
1. Understand what the competitive games have done well.
2. Understand why their retention numbers are so high.
3. Understand patterns of how players interact with games and why? (Behaviour, motivation, etc.)
4. Answer other questions from game teams on idle games.
Research Method
I created a 7 day idle game comparison, split into 2 groups.
We wanted to create 2 study groups, with 1 group playing Idle Miner and Idle Supermarket Tycoon, whereas the 2nd group played Hotel Empire Tycoon and Sushi Bar Idle.
This is a research method that the game teams, creative director and I all agreed on, because Idle Supermarket Tycoon and Hotel Empire Tycoon are actually created from the same company. As a result, the two games are relatively similar in terms of game play, they just had different themes. Whereas Idle Miner and Sushi Bar Idle are idle games that have a more strategic element to it. Idle Miner and Sushi Bar Idle are more repetitive in terms of their game play and it's more focused on earning more money, whereas Idle Supermarket Tycoon and Hotel Empire Tycoon, there's more visual progression within the game.
As a result, we wanted our study groups to play 2 different types of idle games to understand which style of games they prefer and why.
As a result, we wanted our study groups to play 2 different types of idle games to understand which style of games they prefer and why.
I conducted a remote study for 7 days using Lookback, Surveymonkey and phone calls.
Through discussion with the game team and creative director, we had decided on a 7-day remote study to delve into player behaviour in their natural settings. Our goal was to uncover the reasons behind our competitor's impressive day 7 rentention rates, as indicated by AppAnnie's data. Traditional office observations wouldn't have provided the complete picture. In this study, we allowed participants to play for extended durations within their daily routines to obtain an authentic representation of their gameplay habits.
I used Lookback as our chosen platform for usability testing and interviews.
Lookback is a customer research platform for usability testing and interviewing. I opted for Lookback as our chosen platform for our user research because it enabled users to record their real-time interactions on their mobile device while playing our competitors' games. This feature allows us to capture their live thought process as they navigate through the game, providing us with invaluable insights into their behaviour and motivations.
Lookback is a customer research platform for usability testing and interviewing. I opted for Lookback as our chosen platform for our user research because it enabled users to record their real-time interactions on their mobile device while playing our competitors' games. This feature allows us to capture their live thought process as they navigate through the game, providing us with invaluable insights into their behaviour and motivations.
I had our users fill in questions regarding their engagement, expectation and likes/dislikes of the games daily on Surveymonkey.
Our users were required to complete questionnaires everyday on Surveymonkey, because we had mandatory questions that needed daily responses to precisely gauge their feelings about the game by the end of the day.
Each day, I would review their video recordings and surveyed responses. If I had questions, I would call them.
At the beginning of the study, I had informed participants that I might call them for detailed response clarifications. I requested their schedules to determine the most suitable times for these calls. This daily communication option allowed me to get their responses quickly, preventing them from forgetting important game-related details of how they felt on the particular day I was asking about.
Constraints
I dealt with a number of constraints that impacted my research method.
🥵Exhausted Idle Community - Lack of suitable idle players for user test
In the past, we had conducted many play tests to understand Idle Miner and Idle Supermarket Tycoon. However, because our research objective is different from last time, we need players who have never played any of the 4 competitive games before.
However, because we conducted so many play tests for Idle games in the past, we were exhausting the Idle community in Vancouver. As a result, it was highly difficult for us to have a proper amount of female/male ratio between the two groups. And since the 4 competitive idle games are the most highly played in the market, many individuals within the idle community have already played it as well. We needed people that did not play it at all.
We had a total of 66 playtesters, and the greyed-out rows in the table indicate those who were not suitable as playtesters.
Solution: I had to work with the limited amount of players that I had.
🥵Bandwidth issue - I was the only User Researcher for this study
During this time, I was a 1-person department. As a result, I had to manage everything from recruitment, interviews, compiling results, presenting the research to the teams, and etc.
Solution: I simply had to work with the resources that I had, even if that meant lack of suitable players. I also had to firmly manage my time schedule to ensure that I provide all my insights to the teams in a timely manner.
🥵Time Constraint - Game teams needed learnings ASAP
Game teams were already in the beginning phases of developing their game. As a result, I needed to provide information to them as soon as possible if I wanted to help support the teams in ensuring they are heading the right direction.
Solution: I decided I would provide 'daily' insights that I was getting from players each day. These daily insights would be enough for them to work with while I continue to compile the results and create a finalized book of all the themes I have gathered.
As the teams were highly busy and would often find it difficult reading a lot of content within their day, I made sure to make things concise within a single title, and if the teams wanted to read more information on it, then it would be at the bottom. I also made sure to add emoticons into my research, so that it was obvious which points were "good" and which points were "not good" about the competitor games.
As the teams were highly busy and would often find it difficult reading a lot of content within their day, I made sure to make things concise within a single title, and if the teams wanted to read more information on it, then it would be at the bottom. I also made sure to add emoticons into my research, so that it was obvious which points were "good" and which points were "not good" about the competitor games.
The Result
The game teams saw immense value of UX research, leading them to refine key game features before release, significantly enhancing the final product.
To ensure the game team read my findings during the Box Office Tycoon production phase (a new idle game they were creating for launch), I kept them short and to the point, knowing that extensive text often went unread. I used emojis—like a checkmark for positive experiences and a neutral face for negative ones—to make the insights more digestible. This approach led to highly positive feedback and helped the team recognize the value of user research.
The Findings
After 4 weeks, I completed the playbook and compiled common patterns observed from all 10 players throughout their 7 day participation.
Within the playbook, I prioritized the readers' experience by:
1. Implementing a table of content with clickable links for easy navigation.
I understood that game developers have demanding workloads and limited time to go through extensive material. Therefore, I aimed to simplify their experience by enabling them to quickly access the specific information they needed in the playbook.
I understood that game developers have demanding workloads and limited time to go through extensive material. Therefore, I aimed to simplify their experience by enabling them to quickly access the specific information they needed in the playbook.
2. Ensured the information remains concise, using images to effectively convey key points.
I recognize that many people are visual learners. Therefore, I aimed to reduce textual content by incorporating images and screenshots, while adding 'boxes' to the images to allow the readers to know where to focus their attention on.
I recognize that many people are visual learners. Therefore, I aimed to reduce textual content by incorporating images and screenshots, while adding 'boxes' to the images to allow the readers to know where to focus their attention on.
Below are samples from the 43-page playbook, with most content blurred.
