Quantitative Analysis Reveals Geographical Skew in User Studies within the Security and Privacy Field

News Highlights:

TOKYO - September 3, 2024 - NTT Corporation (Headquarters: Chiyoda Ward, Tokyo; Representative Member of the Board and President: SHIMADA Akira; hereinafter "NTT") and Cybersecurity Laboratory of the National Institute of Information and Communications Technology (NICT, President: TOKUDA Hideyuki, Ph.D.) conducted a systematic review of papers published as user studies (studies involving participants) in the security and privacy field. The review quantitatively revealed that many existing studies focus on populations from limited geographical and cultural regions, primarily in the West. This finding highlights the limited generalizability¹ of previous security and privacy research, suggesting that people from different geographical and cultural contexts, including Japan and other parts of Asia, may not fully benefit from these research outcomes. Our study also emphasizes the importance of identifying differences across diverse populations and proposes research methodologies to enhance understanding of such diversity. This work was presented at USENIX Security 2024, one of the premier international cybersecurity conferences, held in Philadelphia, USA, from August 14-16, 2024 [1].

Figure 1 Overview of the Study

1. Background of the study

In research fields centered on human subjects, such as psychology and human-computer interaction (HCI), user studies have been used to uncover people's psychological and behavioral traits. However, these studies have been criticized for their Western-centric approach, with a significant skew toward "WEIRD" populations [2], [3]. "WEIRD" refers to people from Western, Educated, Industrialized, Rich, and Democratic societies. Previous research focused on geographically skewed populations has not thoroughly analyzed whether the results are universally applicable or if there are geographical differences, and if so, what those differences might be.
　On the other hand, while some studies in the security and privacy research field—analyzing psychological, behavioral, and decision-making processes and applying findings to the design, implementation, and operation of computer systems—have shown that geographical and cultural differences influence results, the extent of the skew towards "WEIRD" populations in previous research remains unclear.

2. Outline of the study

Based on a systematic literature review methodology², we comprehensively investigated and analyzed user studies in security and privacy research papers. From 7,587 papers presented at prominent international conferences on cybersecurity and human-computer interaction³ over the past five years (2017 to 2021), we identified 715 papers that conducted user studies in the field of security and privacy. We analyzed these 715 papers regarding participants' countries of residence, demographics, recruitment methods, study methods, and research topics using a process that ensured inter-rater reliability⁴ by multiple analysts.

3. Research findings

In this study, we found that in the security and privacy field, the proportion of user study samples⁵ targeting non-Western populations decreased from 25% to 20% over the past five years (2017-2021), indicating an increase in Western skew (Figure 2).
　In contrast, a similar study in the HCI field [3] showed that the proportion of samples from non-Western countries increased from 16% to 30% over five years (2016-2020). This result indicates a trend toward reducing Western-centric skew in HCI research. This comparison highlights that the Western skew in the security and privacy field is more pronounced.

Figure 2 Temporal Changes in the Proportion of Countries/Regions Represented in User Study Samples

Additionally, our investigation of the normalized ratios of participant samples by the country's population (Ψs) revealed that Western countries such as the United States, the United Kingdom, and Germany are overrepresented relative to their global population ratios. In contrast, many non-Western countries, including those in Asia, the Middle East, Africa, and South America, are underrepresented relative to their global population ratios (Figure 3). Furthermore, we found a positive correlation between Ψs and the degree of education, industrialization, richness, and democracy in each country. This statistically significant relationships indicate that user studies tend to be skewed toward populations from countries with higher levels of E, I, R, and D (educated, Industrialized, Rich, and Democratic).

Figure 3 Worldwide Distribution of Normalized Ratios of Participant Samples by Country's Population (Ψs)

One factor contributing to this skew toward "WEIRD" populations is the geographical skew of the authors themselves. Among the analyzed papers, 86.5% were authored solely by researchers affiliated with institutions in Western countries. Researchers tend to recruit participants who are more accessible due to geographical and linguistic barriers, leading to convenience sampling⁶, which makes it difficult to study populations from different countries. This practice has been found to exacerbate the skew toward "WEIRD" participants in user studies.

We propose the following approaches to address the skew identified in this study and develop methodologies for understanding diverse populations.

4. Outlook

These results are expected to promote international collaborative research to understand diverse countries and cultures in the security and privacy field. We will contribute to advancing diversity and inclusion by fostering the development of security and privacy technologies that benefit a broader range of people.

< References >

^[1]Ayako A. Hasegawa, Daisuke Inoue, Mitsuaki Akiyama. "How WEIRD is Usable Privacy and Security Research?" USENIX Security 2024.
https://www.usenix.org/conference/usenixsecurity24/presentation/hasegawa

^[2]Henrich, Joseph, Steven J Heine, and Ara Norenzayan. "The WEIRDest people in the world?" Behavioral and Brain Sciences 33, no. 2-3 (2010): 61-83.

^[3]Sebastian Linxen, Christian Sturm, Florian Brühlmann, Vincent Cassau, Klaus Opwis, Katharina Reinecke. "How WEIRD is CHI?" ACM CHI 2021.

^1.Generalizability: The extent to which research findings can be applied to different situations or populations. The more widely applicable the findings are to various groups or contexts, the higher the generalizability of the study.

^2.Systematic literature review: A method for systematically and comprehensively searching, evaluating, and integrating existing literature within a specific research field. Papers are selected based on clear criteria to minimize bias when deriving conclusions.

^3.The ten international academic conferences include the top four cybersecurity conferences (USENIX Security, IEEE S&P, ACM CCS, NDSS), top HCI/CSCW conferences (ACM CHI, ACM CSCW), and conferences focusing on human-centered security and privacy (SOUPS, PETS, EuroUSEC, USEC).

^4.Inter-rater reliability: A consistency among multiple analysts when independently analyzing data. In this study, we confirmed that the reliability exceeded the standard threshold.

^5.Number of samples: The number of times samples are extracted from a population. For example, if user studies are conducted once in the U.S. and once in Japan, the sample count is 1 for the U.S. and 1 for Japan.

^6.Convenience sampling: A method where user study participants are not randomly selected but are chosen based on their accessibility to the researcher. For example, a university researcher might recruit students from the same university for a user study.

^7.Replication study: A study conducted using the same methods and conditions as previous research to test the reproducibility and generalizability of the findings. If a researcher cannot replicate the results in different populations, it suggests that the findings may be limited to specific conditions or groups.

About NTT

NTT contributes to a sustainable society through the power of innovation. We are a leading global technology company providing services to consumers and businesses as a mobile operator, infrastructure, networks, applications, and consulting provider. Our offerings include digital business consulting, managed application services, workplace and cloud solutions, data center and edge computing, all supported by our deep global industry expertise. We are over $97B in revenue and 330,000 employees, with $3.6B in annual R&D investments. Our operations span across 80+ countries and regions, allowing us to serve clients in over 190 of them. We serve over 75% of Fortune Global 100 companies, thousands of other enterprise and government clients and millions of consumers.

About NICT

The National Institute of Information and Communications Technology (NICT) is Japan's sole National Research and Development Agency specializing in the field of information and communications technology and is charged with promoting the ICT sector as well as research and development in ICT, which drives economic growth and creates an affluent, safe and secure society.
For more information, visit https://www.nict.go.jp/en/