User experience analysis of Peduli Lindungi-applications using heart metrics in Eid Al-Fitr travellers

INTRODUCTION

As implementation of COVID-19 surveillance, the government has created a mobile application "Peduli Lindungi" which functions as tracing, tracking, warning and fencing in the territory of the Republic of Indonesia, and in order to facilitate a new order of life (new normal).¹ This also applies to surveillance during the 2022 Eid travel period where many people going to their hometown. The government requires them to fill out the E-HAC menu in the "Peduli Lindungi" application. In addition, there is a vaccine certificate for those who have vaccinated in this application also has, record of COVID-19, and a scanning feature to enter certain public places.²

The "Peduli Lindungi" application is an application that is commonly used by people during the COVID-19 pandemic. The Peduli Lindungi application can be accessed on smartphones. Using this application on a smartphone can make it easier for users to access this application, and can also be used whenever needed because smartphones have a long battery life, are easy to carry, and can be connected to medical application platforms.³ However, this application is new for the community, where people used to feel free to travel and go to public places, while during a pandemic people must use the "Peduli Lindungi" application to track their trip if they are going to travel and go to public places. User Experience measurement is needed to ensure that a product provides sufficient service, so that it can satisfy its users when using the product.

One of methods measuring user experience is the HEART metrics method. HEART metrics is a user-centred tool to identify goals and measuring success.⁴ According to Rodden et al HEART consists of: happiness, engagement, adoption and retention, task success.⁵ To process the results from the HEART Metrics measurement, the importance performance analysis method is used to find out what aspects need to be improved.

The purpose of this study is to describe the user experience of the Peduli Lindungi application using heart metrics for Eid Al-Fitr travellers.

METHODS

The study was conducted from December 2022-April 2023 in the capital city of Jakarta. This research is a quantitative study using a cross sectional design. Data is collected and assessed at one time. The variables are happiness, engagement, adoption, retention, task success. Sample were 102 respondents that travelers from Jakarta, Bogor, Depok, Tangerang and Bekasi to their home town using the Peduli Lindungi application during their trips.

Inclusion criteria are respondents traveling outside the Jakarta, Bogor, Depok, Tangerang and Bekasi area for Eid Al-Fitr and respondents using both public transportation and private vehicle for their trips. Exclusion criteria are that the respondent is not willing to be interviewed and the respondent not in good condition. The sample takes by accidental sampling technique. Data collected by using a questionnaire. There is no ethical approval in this study

Descriptive analysis was carried out to explain or describe the characteristics of each research variable. Data were analyzed using the HEART metrics framework. HEART metrics is the easiest framework for identifying goals and measuring success that focuses on User-centered. The HEART metrics approach is not used with a focus on certain variables, but rather there is a structured method to organize the variables that must be used in evaluation by determining a series of goals, signals, and metrics.⁵ To find out the suitability of the expectations and reality of each question item, the importance-performance analysis technique was used. The user is said to be satisfied if the performance value minus the goal value ≥0.

RESULTS

Demographic of the respondents can be seen in Table 1.

The table showed that more than half of respondents are female (69.6%), 55.9% respondents live in Jakarta (55.9%) and the most widely used transportation is private vehicle (52.9), 3.9 percentage respondents live in Depok and the least used form of transportation is by ship.

Determination of goal-signal-metrics of heart metrics framework can be seen in Table 2.

Table 2 below is user experience data using the Heart Metrics framework. It can be seen that among 5 variables only 1 variable has reached the Peduli Lindungi user. The variable is the task success (80.14%). While the variable with the lowest goal achievement is the Retention (70.39%) (Table 3).

Measuring the level of suitability of goals and the performance of caring applications is done by comparing the goal (expectation) that has been set in the previous stage (80 percentage) with the results of the assessment from the respondents (performance). The application is said to have fulfilled the user's wishes if the level of conformity between performance and goal is ≥0. On the other way, it means that if the subtraction results from the level of performance and goal ≥0, it means that the user is satisfied with the questioning item. In the table it can be seen that from 27 statement items assessed by respondents, only 7 items had a value ≥0. This means that only about 25.92 percentages of the items met the expectations of the user. Items that meet user expectations are among the engagement variable (statement number 4), adoption (statement number 1) as well as the task success (statement number 2, 3, 5, 6, and 7). The results of the conformity level can be seen in Table 4 below.

DISCUSSION

User experience analysis using the heart metrics framework aims to measure the success of a program or application that has been assigned to users.⁶ Users are the only determinant of whether the application or program used can function properly, meet expectations, and answer needs. The questions needed to find out the level of product efficiency, the level of product recommendations, and how users compare products with competitors, product deficiencies, and how to improve can be identified with a user experience measurement tool.⁷ There are 2 main aspects in the user experience of health applications, namely the experience of being sick and the experience of health services. First, the experience of illness includes the physical and psychological limitations caused by the illness. Second, health service experience includes user perceptions during the health service process.²

From 102 respondents, it was found that the score of the happiness variable was 71.37%, engagement 74%, adoption 74.71%, retention 70.39%, task success 80.14%. Based on the goals that have been set, only the task success variable meets the target user that 80% of the community has succeeded in using the features in the Peduli Lindungi application. The results of this study are in line with research by Hanum et al that the task success variable in the Peduli Lindungi application has a good category (105.18%) and also Khakim and Sharif's research found that the task success variable in the Go-Jek application has a good category (72.93%).^4,8

The task success variable describes the level of efficiency, effectiveness, and error rate of a system.⁴ From 7 statement items given to respondents, 5 items indicated satisfaction with the effectiveness and efficiency of the Peduli Lindungi application. These items are “I can scan the QR code on the Peduli Lindungi application”, “I can register vaccines in the Peduli Lindungi application.”, I can see vaccine history and tickets in the Peduli Lindungi application.”, “I can find the vaccine certificate in the Peduli Lindungi application”, “I can find important information about COVID-19 in the Peduli Lindungi application”. This result shows that in terms of the function of the Peduli Lindungi application, it is able to function properly. Users are satisfied with the barcode scanning feature and the vaccine certificate that can be accessed whenever they need it.

Meanwhile the variable retention is the variable that has the lowest score (70.39%) among the five variables. Retention is related to how many users from a certain time period are still exist in several subsequent time periods.⁴ None of the items met the goal of the 5 items asked. According to the researcher's assumption that most people use this application only to scan QR to enter public places. So that people who rarely go to public places will install the application when they are in the public and will uninstall the application when they don't use it anymore (do not travel to public places). It will affect the number of users in a period of time.

The statement that has the highest score is found in the task success variable. the item is "I can see the vaccine certificate in the Peduli protect application" with a percentage of 86.67%. In the Peduli Lindungi application, if the community has been vaccinated with the COVID-19 vaccination, the vaccination certificate will be automatically provided in their application and can be accessed and downloaded without requiring a long time. Vaccine certificates are needed as a requirement to enter public places if the QR scan is not successful, such as in transportation facilities, recreational areas, or public facilities such as schools, campuses, places of worship, hospitals, etc.

The item that has the lowest score is found in the happiness variable. Happiness is used to describe metrics that relate to subjective aspects of user experience, such as satisfaction, visual appeal, possibility for recommend, and perceived ease of use.⁴ The item that has the lowest score is "I feel I have to use the Peduli Lindungi application every time" with a percentage of 60.20%. Moore found that a good application is an application that can provide value in user activities. Not only does it provide useful features, applications must have a supporting structure in the user's environment to increase motivation, encourage peer involvement, and adapt to user preferences.⁹

According to the researchers' assumptions, in line with the phenomenon of using the Peduli Lindungi application whose use focuses on preventing and controlling COVID-19, people only access this application if they have needs related to the COVID-19 handling protocol. In addition, the features of the Peduli Lindungi application are limited to COVID-19 information. This is the reason why people do not use the Peduli Lindungi application all the time.

Two other items that meet user satisfaction are the engagement and adoption variable. Engagement means user involvement with a product. This term is often used to refer to the frequency, intensity, or depth of interaction over a period of time. Adoption focuses on new products and features such as calculating how many new users are using the product at a certain point time and measure the level of user activity over a period of time so that manufacturers can continuously improve the service.⁴ The 2 items are "I can spend time using the Peduli Lindungi application" and "I know how to use the Peduli Lindungi application". These two items indicate that the Peduli Lindungi application is a user-friendly application so that users have no difficulty accessing the services contained in it. According to Suryanto et al in order to create good relationships between users and service providers, features in services can be adapted to functions and with user expectations. Thus, there will be repeated use of the service. This is one indicator that application meets user satisfaction.¹⁰

User intent is influenced by user experience. Perceptions of quality among different user experiences are strongly influenced by psychological needs. Therefore, user experience is very necessary in a digital concept map of an application.^11,12 Continuous application development is a factor in user satisfaction. The more added value there is to a user's life, the more someone's interest in using an application will increase.

Limitations

This study was limited to measures user satisfaction using the heart metrics framework. Respondents were also limited to only traveling home for Eid Al-Fitr, not on tourism, business or educational trips. This research also did not explore user complaints or expectations regarding the Peduli Lindungi application. This research was conducted when COVID-19 cases were starting to show improvement and there might be some areas that neglected to monitor COVID-19 through the Peduli Lindungi application, but only in minor cases. So that this research states in inclusion criteria that people who have used Peduli Lindungi application can be respondent in this study.

CONCLUSION

From 102 respondents who have traveled for Eid in 2022, it was found that when using Peduli Lindungi application, the happiness variable score was 71.37%, engagement 74%, adoption 74.71%, retention 70.39%, task success 80.14%. The largest score of the statement in the questionnaire is accessible to the vaccine certificate (86.67%). The statement with the lowest score is the continuity of using Peduli Lindungi application (60.2%). It would be better if the Peduli Lindungi application adds features related to general health information so there will be more user and level of application access can increase.

ACKNOWLEDGEMENTS

Author would like to say thank you to research dan community service department Esa Unggul university, as well as all of those who have assisted in the implementation of this research.

Funding: No funding sources

Conflict of interest: None declared

Ethical approval: The study was approved by the Institutional Ethics Committee

REFERENCES

1. 1. Ministry of Communication and Informatics. Guideline of the PeduliLindungi Application. 2021.
2. 2. Ministry of Communication and Informatics. Safe and Healthy Ied-Fitr Travel. 2022;1. Available at: https://indonesiabaik.id/public/uploads/post/5795/Mudik-2022.pdf. Accessed on 15 January 2023.
3. 3. Ryu WHA, Kerolus MG, Traynelis VC. Clinicians’ User Experience of Telemedicine in Neurosurgery During COVID-19. World Neurosurg. 2021;146:e359-67.
4. 4. Latifah Hanum A, Karunia Miranti T, Fatmawati D, Frendi Diyon M, Joyo Prawiro C. Analysis of User Experience on Peduli Lindungi Mobile Application Using Heart Metrics. J Heal Sains. 2022;3(2):362-72.
5. 5. Rodden K, Hutchinson H, Fu X. Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications. 2010. Available at: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36299.pdf. Accessed on 15 January 2023.
6. 6. Kokkinaki A, Flaounas I. On the Evaluation of OLEs Using the HEART Framework Ilias Flaounas. 2015. Available at: file:///C:/Users/Fannya/Downloads/ ASCILIATE_2015_Heart_Framework.pdf. Accessed on 15 January 2023.
7. 7. Tullis T, Albert T. Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics: Second Edition. Morgan Kaufmann. 2008.
8. 8. Khakim ML, Sharif OO. Analysis of user experience of go-jek applications using heart metrics. 2018. Available at: https://openlibrarypublications.telkomuniversity.ac.id/index.php/management/article/view/6339/6317. Accessed on 15 January 2023.
9. 9. Moore K, O’Shea E, Kenny L, Barton J, Tedesco S, Sica M et al. Older adults’ experiences with using wearable devices: Qualitative systematic review and meta-synthesis. JMIR mHealth uHealth. 2021;9(6):1-19.
10. 10. Suryanto TLM, Fauzi A, Setyohadi DB. Identifying the effect of emotions in government-citizen online (G2C) tourism based on the HEART metrics. Int J Data Netw Sci. 2021;5(4):641-8.
11. 11. Brodeur M, Margo-Dermer E, Chouinard MC, Hudon C. Experience of being a frequent user of primary care and emergency department services: A qualitative systematic review and thematic synthesis. BMJ Open. 2020;10(9):1-11.
12. 12. Rohles B, Backes S, Fischbach A, Amadieu F, Koenig V. Creating positive learning experiences with technology: A field study on the effects of user experience for digital concept mapping. Heliyon. 2022;8(4).