Synthetic users: is there a place for “AI-generated users” in UX Research?

Trusting AI over real users? Here’s why it might cost you more than you think.

It is now common to see synthetic users being widely used in UX research. After all, with the rise of AI, synthetic data generation is becoming the new norm for gaining insights.

For context, the synthetic data generation market size is expected to grow at a CAGR of 31.1% [1]. Meaning it’s not just growing, it’s accelerating every year. By 2030, this market could be worth an eye-popping USD 2.34 billion. In simple terms, synthetic data is becoming too big for businesses to ignore.

Of course, synthetic users, or AI-generated personas, are just one application of synthetic data generation. Synthetic data has a much larger landscape. It’s everything from fake users to fake behavior, interactions, and even fake datasets that AI models are trained on. In UX research specifically, synthetic users are created to mimic real user behavior, which is one way to leverage synthetic data for design and testing.

This is something businesses cannot ignore. For instance, a major e-commerce company can easily slash its UX research budget by a whopping 30% by heavily relying on interactions with these AI-generated personas.

Yes, they may get the plaudits for revolutionizing the industry in an economic sense [2]. But at what cost? Some go even further — Mark Ritson praised synthetic data as a game-changer for research. He highlighted new studies where AI-generated consumer data produced results “around 90% similar” to those from real surveys. [3]

With such success, companies can end up encountering a wave of customer complaints for poor design and unaddressed needs. This is because synthetic data has its limitations and risks.

Yes — AI-driven users are synthetic personas based on real user data, and they simulate interactions with a feature, providing initial insights. However, features that perform well with AI personas can still fail with real users, as AI-driven feedback lacks emotional depth and unpredictability, making real-world validation essential. Hence, one shouldn’t be surprised to see a decline in user engagement and a costly redesign that relies only on AI-generated insights.

This now raises a crucial question: Can AI-generated personas truly replace human intuition in UX research, or are we risking innovation by relying too much on synthetic data?

This article helps dive deep into these questions, examining both the opportunities and limitations of using synthetic users in UX design and research.

Moreover, designers will have a clearer understanding of when and how to integrate synthetic users effectively in UX research.

What are synthetic users?

According to Neilsen/Norman Group, they define synthetic users as “an AI-generated profile that attempts to mimic a user group, providing artificial research findings produced without studying real users.”

Synthetic users are used in UX research to gain user insights, test interfaces, workflows, and design elements without relying on human participants. As we can see, AI continues to revolutionize industries, and UX research is no exception.

67% of technology enterprises are now using synthetic data in their development workflows, up from just 23% in 2019 [4]. Many of these companies are increasingly turning to synthetic users to accelerate testing and study user behaviors across global markets so that they can save time and money in the process. Nevertheless, this growing reliance on AI-generated personas raises critical questions about the accuracy, ethics, and effectiveness of synthetic user research (all of this will be covered later in the article).

But first, let’s understand more about synthetic users.

How are they created after all? These virtual users are generally built using AI models trained on vast datasets of real user interactions, behavioral patterns, and decision-making processes. Basically, data from the whole internet!

While traditional UX personas are based on qualitative and quantitative research about real user groups, synthetic users are entirely data-driven and generated through AI or trained LLMs.

We are aware of the fact that traditional personas require manual creation and validation based on market research. Meanwhile, synthetic users can dynamically adjust their settings based on AI-generated insights. Hence, why companies are attracted to the prospect of using synthetic users as it’s seen as a scalable and cost-effective alternative.

Regardless, synthetic users do lack the depth of real human emotions and unpredictable behaviors that human participants bring to UX research. But it’s wise to learn the potential synthetic users can have by learning about its pros.

Synthetic users do come with benefits in UX research

Synthetic users may evoke negativity in the design space, but one cannot deny that it can be a valuable tool in UX research — sometimes. Here are a few benefits that can come with incorporating synthetic users into your UX research:

Pro #1- Cost and time efficiency

This is the most obvious advantage that we needed to start with. Conducting UX research with real human users can often be time-consuming and costly depending on the nature of the project, and sometimes you need to encounter awkward conversations. However, synthetic users eliminate these barriers by allowing us to carry out rapid testing at a much lower cost.

Think from the perspective of a fintech startup looking to develop a new mobile banking app. Instead of spending weeks conducting user interviews, they can generate thousands of synthetic users within hours.

In this case, having synthetic users helps them to iterate quickly in a very competitive market where time and budget constraints are critical.

Pro #2- Scalability and diversity

If you want access to a wide range of user demographics instantly, then synthetic users help with that cause. UX teams that are eagerly looking to explore various cultural and accessibility considerations can benefit from incorporating synthetic users in their research.

Imagine a global e-commerce company and the leverage they have when they can generate synthetic users to test their websites across different countries, getting insights on language preferences and browsing habits.

Synthetic users have the ability to model diverse backgrounds without accessing real-world user pools.

Pro #3- Exploring edge cases and extreme scenarios

Let’s face it — there will be some UX challenges that involve rare, extreme, or even dangerous situations that are difficult to replicate with real-world users. That’s why AI-generated users can assist UX researchers with identifying pain points that may not surface in standard user testing.

Let’s say a cybersecurity firm wants to test users on how they respond to phishing attempts. An often sensitive and extreme case. However, companies can generate users with different levels of tech literacy to learn how they would react to such scenarios.

Pro #4- Privacy considerations

Privacy can be a concern for UX teams when gathering data on real users. However, once you eliminate the need for real user data, synthetic users can help comply with privacy regulations while still gaining valuable user insights.

Understand this from the perspective of a healthcare company that wants to optimize its patient portal without handling sensitive medical records. By using synthetic users, they get to test different features in their portal while avoiding privacy concerns.

In a nutshell, synthetic users can help bypass regulatory issues related to real user data collection.

Even though I wanted to share the positives synthetic users have to offer and make them sound like they are the only answer, I would also like to raise crucial questions about their limitations.

The limitations and risks of using synthetic users

Synthetic users may have their pros, but they do often come with certain drawbacks, and this can impact the quality and reliability of UX research.

AI cannot exhibit genuine human emotion.

Let’s say a company wants to test out their mental health app with synthetic users in simulated therapy conversations, would they really get reliable insights if they did so?

To an extent, AI-generated profiles will only give generalized opinions. But they often lack deeper meaning that can genuinely help designers make an impact for real users.

Synthetic conversations fail to capture the emotional depth and unpredictability that we humans have. As a result of not exhibiting a full range of human emotions, synthetic users’ findings can often be misleading.

One key area where AI-generated personas may lack insights is not in attitudinal research but in behavioral research. AI simply cannot experience real emotions such as delight, frustration, fatigue, etc. However, we know that by observing real users’ behavior, we can pick out these cues and develop solutions to their pain points.

More interestingly, synthetic users can even struggle to replicate certain irrational decision-making or spontaneous actions that researchers observe naturally in human behavior. Hence, synthetic users can only provide insights at a surface level.

AI is a Bit Biased!

Remember when I said earlier in this article that AI models rely on existing datasets (a.k.a the internet)? This means they introduce certain biases and reinforce stereotypes rather than challenge them.

For example, AI models inherit biases such as gender bias and generalized stereotypes from their training data, and this can sadly lead to incorrect assumptions and findings.

This basically means an AI model trained primarily on Western internet habits will surely struggle to simulate accurate UX behavior in different emerging markets. Hence, real users are more insightful in addressing cultural and other subjects with deeper meaning.

AI can easily respond without context

Suppose a company specializing in smart home automation wants to learn about lighting adjustments and preferences. If they use synthetic users rather than actual users, they can easily miss out on cultural and psychological nuances about home comfort.

This depicts the problem from an applicational point of view. As we know, AI-generated feedback fails to depict emotions and the unpredictability factor, and this also means it lacks real-world experiences.

AI models lack real-world intuition, and this can often lead to incomplete UX insights. This can be dangerous if entities are over-reliant on AI-generated feedback.

Also, you might have noticed this, but synthetic users or AI in general just want to “please” researchers–a phenomenon known as sycophancy — and this does not really represent human behavior well. Think about it!

Is it really ethical to rely on AI for UX practices?

Using synthetic personas and labeling the results as user-tested or presenting UX findings based on these datasets as research raises ethical concerns. Those practices can deceive stakeholders about the authenticity and reliability of the insights. Above all, their use should be disclosed, especially when making broad UX decisions, to ensure transparency and avoid misleading claims.

Delve AI is a good example. They openly discuss the creation and application of synthetic personas in their research, providing clarity on their methodologies and the role of AI-generated data in their processes.[5]

This comes to show that by adopting such transparent practices, organizations can navigate the complexities of integrating synthetic data into UX research while upholding ethical standards.

Synthetic users vs. real users: who would win if these two went head to head?

So here’s the thing: Instead of arguing if synthetic users are better or worse than using real users, I will play devil’s advocate and make a case for both.

Based on my research and testing, I found that interestingly, both have a place in UX design, and here are a few use cases and scenarios that you will find intriguing:

Scenarios:

Early-stage ideation and hypothesis testing
Synthetic Users: Highly cost-effective and scalable
Real Users: Takes a lot of time and can be expensiveUsability testing for common UX patterns
Synthetic Users: Provides faster feedback loops
Real Users: Requires real users for validationTesting extreme or rare use cases
Synthetic Users: AI can simulate outliers
Real Users: It can be difficult to find diverse participantsEmotional response and satisfaction research
Synthetic Users: AI lacks human emotions
Real Users: Real users provide genuine reactionsAccessibility testing
Synthetic Users: Can simulate disabilities
Real Users: Real users provide deeper insightsCultural context and social norms
Synthetic Users: AI struggles with nuance
Real Users: Real users provide authentic perspectivesPrototyping new features for existing products
Synthetic Users: Fast iteration cycles
Real Users: Provide insights into feature adoption barriersTesting ethical implications of design decisions
Synthetic Users: AI may lack moral reasoning
Real Users: Real users offer real-world perspectives on ethical concerns

Is combining synthetic and real user research worth it?

As you can see from the table above, there is no clear winner. In fact, both have their pros and cons, which makes this discussion interesting.

There is a place in UX design and research where both synthetic users and real users can work together and achieve a more optimized solution.

I suggest using synthetic users mainly for early-stage ideation and hypothesis testing, as this is cost-effective and not time-consuming.

For validation, use real users to validate things before any significant product launch. And lastly, for real-world insights, real users are the way to go, but that doesn’t stop you from using improved AI-generated models that provide better real-world insights. In truth, it’s all about staying updated with the latest AI models.

Best practices for integrating synthetic users in UX

Before wrapping up this article, I’m going to put out a few practices that you can use to integrate synthetic users into your UX workflows. I will provide some actionable steps that you can later use as a checklist for your future projects.

1. Use synthetic users as a supplement, not a replacement

Make this clear — synthetic users will NEVER replace real users in research and testing. Instead, these personas should enhance your real user research. Always complement AI-generated insights with real-world testing to see significant results.

One should never use only AI-generated insights for the entire UX process. But you shouldn’t ignore that it has its perks for particular tasks in your UX workflows. For example, using synthetic personas for A/B testing early prototypes can help provide great insights and help make efficient decisions.

2. Validate AI findings with real user testing.

I mentioned this before, but remember that any “major” UX decision you are going to make should be verified with human testing before implementation.

Never fully trust AI-generated insights because that might end up haunting you in the future. Always cross-check them with real user feedback.

3. Utilize improved AI models with Real data

Remember that AI can have certain biases that may hinder the quality of our research and testing. Hence, keep up-to-date with improved AI models with actual user behavior data, as this can help to improve accuracy.

Instead of relying on generic tools like ChatGPT, explore tools specialized for generating synthetic personas for UX research and testing, such as Synthetic Users, UXtweak, Tonic.ai, and MOSTLY AI.

4. Always maintain ethical transparency

This could often be ignored, but always be transparent about when and how synthetic users are used in your research. I recommend disclosing such information wherever necessary. This helps you stand on your ethical ground and address concerns before implementing a major UX decision.

My Final Thoughts…

To summarize, we have seen the major perks of using synthetic users: they can be cost-effective and provide scalable UX insights. Nevertheless, they come with significant limitations that can be mitigated if we pay close attention to them.

Synthetic users should never fully replace real users. Instead, it should be complemented to enhance the research and testing process.

The best way to go about this would be using a hybrid model where one can leverage synthetic users for speed and scalability while validating insights with real participants. This can depend on you, but I would personally like to implement an 80/20 or 70/30 split, where real users are prioritized while utilizing synthetic data as a supplementary tool.

The future of UX research will likely involve AI-enhanced methodologies rather than full automation. If everyone can integrate synthetic users thoughtfully and ethically, then businesses can seriously optimize their UX strategies without sacrificing genuine human insights.