In-app usability testing means watching real people try to complete real tasks inside your product, from first login to the first result, and recording where they hesitate, fail or leave. For a SaaS trial, the fastest way to start is simple: recruit five to eight people who have never seen your product, give them one realistic goal, stay quiet, and note every place they stall. This guide shows how to plan that test, choose a format and tools, test prototypes before launch, and turn what you see into a ranked list of fixes.
TL;DR
- Usability testing measures whether people can finish a specific task, not whether they like the design. Start from the activation event, not the home screen.
- Test the entry point a new trial user actually uses, with people who are unfamiliar with your product and an account that is empty, as theirs will be.
- Use moderated sessions to learn why people struggle and unmoderated sessions to measure how many do. Most teams need both.
- Test prototypes before development, then test the live flow again after each change.
- Rank findings by how close they sit to the first win, and validate every fix with an experiment.
Table of Contents
- What in-app usability testing is
- What it finds in onboarding and activation
- How to build a usability test plan
- Test prototypes before the flow is live
- Moderated or unmoderated testing
- Remote testing and choosing tools
- From findings to fixes you can validate
- Our perspective
- Get a second pair of eyes on your trial
- FAQ
- Sources
What in-app usability testing is
Usability testing is a research method in which representative users attempt realistic tasks while an observer records where they succeed, hesitate or fail. According to this overview of the method, its purpose is to measure how easily a task can actually be completed, not to collect opinions about the design.
"In-app" adds one constraint that matters for SaaS: the test happens inside the product, in the state a new user would really be in. That means a fresh account, no sample data, no coaching from a teammate and no shortcut link to the screen you want to examine. Your internal accounts are full of data and your team knows where everything lives. Neither is true for a person who just clicked "Start free trial".
This is different from analytics. Dashboards tell you that 60% of trial users stopped at step three. A usability session tells you that they stopped because the field label used a term only your engineers use. GitLab's UX handbook puts the split well: experiments tell you what is happening, while user research tells you why. You need both, and the order matters: analytics point at the step, testing explains it.
What it finds in onboarding and activation
The problems that surface in trial flows are rarely exotic. They tend to be the same handful of patterns:
- Setup before value. A new user reaches a configuration step, such as connecting an account, importing data or inviting teammates, and never finishes it. Nothing useful happens until it is done, so nothing useful happens.
- The blank state. The user lands on a screen with nothing on it and no idea what to put there. One guide to the method calls this the most common onboarding failure, and notes it never appears in internal review because internal accounts are populated.
- The repeat visit. A trial user opens the same page twice and leaves. They were looking for something and did not find it.
- The plan comparison with no decision. A buyer reads the pricing page, compares tiers and closes the tab, because the difference between plans is not tied to anything they want to do.
- The upgrade prompt before the first win. The product asks for money before the person has seen a result.
- The late total. A checkout price that appears only on the last screen, after the user has already committed effort.
A test of a 21-step onboarding flow described by one product team found that only about 1% of trials completed it and more than half dropped before the second step. Treat that as an illustration of how steep the cliff can be, not a benchmark for your own product.
For more on what a good first-result path looks like, see our onboarding flow examples and the onboarding case studies.
How to build a usability test plan
A plan fits on one page. A professional guide to unmoderated testing lists six steps: define study goals, select testing software, write task descriptions, pilot the test, recruit participants and analyze results. The same source adds an important point: do not pick the tool first. Know what you want to learn, then choose software that meets those requirements.
For a SaaS trial, adapt those steps like this.
1. Define the activation event
Write one sentence describing the moment a user first gets value. For a reporting tool it might be "sees their own data in a chart". For an AI product it might be "gets a useful output from their own input". Everything in the test is measured against reaching that moment. If you have not pinned it down, our guide to time to value shows how to define and measure it.
2. Set scope, expected results and measures
Decide which flow you are testing (sign-up to first result, say, not the whole product), what you hope to learn, which method fits, and how you will judge success. Two measures come out of most sessions: task completion rate and error rate, which together show where a specific step breaks. Add time on task, since a flow that technically works but takes twenty minutes will lose trial users.
3. Recruit for unfamiliarity
Your best participants look like your trial sign-ups and have never used your product. Existing customers have learned your workarounds and will not hit the walls a new user hits. Five to eight participants per round is enough to see the major patterns. If you serve distinct roles, such as an admin who configures and an end user who consumes, recruit each role separately, because their problems differ.
4. Write quiet tasks
A task describes a goal, not a path. "You want to see how your last month of sales is trending. Show me how you would do that" works. "Click Reports and create a new chart" gives the answer away. Avoid your own interface vocabulary in the task text. Pilot the script with a colleague first, so you catch confusing wording before it costs you a real participant.
5. Start at the real entry point
Begin where a trial user begins: the ad, the landing page or the sign-up form, not a logged-in dashboard. Many conversion problems sit in the handoff between marketing pages and the first screen of the product, and a test that starts inside the app never sees them.
6. Look for patterns, not anecdotes
One person getting lost is a story. Four of six getting lost at the same field is a finding. Log every task as success, success with difficulty or failure, and tally where problems cluster.
Test prototypes before the flow is live
The cheapest usability problem is the one you find before an engineer builds it. A prototype test follows the same logic as a live test, with a few differences:
- Pick the fidelity to match the question. A rough clickable sketch can test whether people understand the structure of a setup flow. A polished prototype is better for testing wording, layout and trust.
- Prototype the path, not the product. Build only the screens needed for the task. Tell participants up front that some elements may not work as expected, so a dead link is not mistaken for a design failure.
- Share it as a link. Most prototyping tools produce one, so remote participants can open it on their own device.
- Test, review, refine, repeat. Prototype testing pays off most when it is a loop, not a one-off event. Run a round, fix the top problems and run another before handoff.
When the change is a new feature in an existing product, include both current users and new ones. Current users reveal whether the change disrupts habits. New users reveal whether it makes sense cold. That second group is the one your trial depends on.
Moderated or unmoderated testing
The choice comes down to the kind of answer you need.
| Moderated | Unmoderated | |
|---|---|---|
| How it runs | A facilitator guides the session live | Participants complete tasks alone, recorded |
| Best for | Understanding why, exploring new problems, complex flows, early prototypes | Measuring task success, comparing two designs, quick checks, larger samples |
| Typical sample | 5 to 8 people per round | Dozens, if you need numbers |
| Speed | Slower to schedule and run | Results can arrive within a day |
| Main risk | The facilitator leads the participant | Nobody can ask a follow-up question |
A practical sequence for a trial flow: run five moderated sessions on the current onboarding to find the problems and their causes. Fix the obvious ones. Then run an unmoderated round to confirm that task completion improved across a wider group. Fidelity and budget also matter. An early sketch rarely suits unattended testing, because participants need someone to explain what is a placeholder.
In moderated sessions, the facilitator's discipline decides the quality of the data. Watch, do not lead. Ask what they expect to happen before they click. Ask what they are looking for when they pause. Never rescue a stuck participant in the first minute, because that silence is the data.
Remote testing and choosing tools
Remote testing is the default for SaaS because your trial users are remote anyway. For a live product, you give the participant a public URL, or the name of the app in a store for mobile, and make sure it is accessible without special setup. For prototypes, you share a link.
When you compare tools, sort them by what they do rather than by brand. The categories most teams choose between:
- Panel-based testing platforms that supply participants and record screen and voice. They are fast for recruiting strangers who match a profile, and costs run from modest monthly plans to enterprise contracts that are not published.
- Prototype testing tools that measure clicks, paths and task success on design files before code exists.
- Session replay and analytics tools that record what real trial users do in the live product. They show where people stall at scale but cannot tell you what the person was thinking. Pricing for some is tied to session volume, which matters if your trial traffic is high.
- Video call software plus a spreadsheet. For moderated rounds with five people, this is often enough.
Selection criteria worth writing down before you shop: can it recruit your target role, does it handle your device mix, can it test a live product and a prototype, how do you share clips with stakeholders, and what is the real annual cost at your volume. Pricing changes often, so confirm it on each vendor's page before budgeting.
If your audience is enterprise buyers, recruiting is the hard part. Administrators and procurement-level users are scarce in general-purpose panels. Recruit from your own sign-up list, your customer success team's contacts or a professional network, and offer a fair incentive for their time.
From findings to fixes you can validate
A report with forty observations will not get acted on. A ranked list will.
Rank by distance to the first win
For each problem, score three things: how many participants hit it, how severe it was (blocked, slowed or annoyed), and how close it sits to the activation event. A blocker at step two outranks a cosmetic issue at step eight, since fewer people ever reach step eight. Our in-app guidance article covers the fixes that tend to follow, such as replacing a blank state with a guided first action.
Fix fast, then check again
The RITE method makes very rapid changes to the interface and evaluates them right after implementation, sometimes within hours, so problems actually get fixed rather than filed. You do not need the full method, but the habit helps: when three of five participants fail at the same spot, change it and put the next participant in front of the new version.
Confirm with an experiment
Usability sessions show you what to change. They do not prove the change moves conversion. Release the fix as an A/B test or a staged rollout and watch activation and trial-to-paid. GitLab's team runs usability studies on key flows about every quarter to add qualitative feedback to launched experiments. Another team reports that onboarding completion rose 29% after it used analytics and in-app guides to find the exact drop-off point and redesign the flow. Your results will differ, but the sequence is the model: locate, understand, fix, measure.
Our perspective
Most teams test the wrong thing. They test the interface, when the trial is lost earlier: in the order of the steps. A setup that comes before the first result, an upgrade prompt that appears before the first win, a total that shows up late. None of these is a button problem.
That is why we run a trial the way a buyer does, with a stopwatch and a screenshot of every step, before anyone is recruited. The stopwatch tells us where the time goes. The screenshots show what the user saw when they decided to leave. Then usability sessions have a precise question to answer, instead of a vague one.
We also insist that each recommended change is tied to documented user behavior, not taste. If a session shows four of six people stalling at the same field, the fix is justified. If the only argument is that another product does it differently, it goes lower on the list.
This is the approach behind our trial-to-paid work, and it is how we have approached product flows from a startup setup experience to an 18-year-old governance platform at Swiss Re, where the goal was to modernize without breaking decades of user habits.
Edijs
Get a second pair of eyes on your trial
If your team has sign-ups but too few paying customers, the cheapest next step is to find out where the trial loses people before you commission a round of testing. Our trial to paid service covers exactly that: we go through your trial as a buyer, find where users leave between first login and payment, and redesign the path. You can see how we approached a regulated enterprise product in the Swiss Re project.
Engagements are scoped per company, so there is no published price. Start from the services overview to see which fits.
FAQ
How many participants do I need for in-app usability testing?
Five to eight people per round is enough to surface the major problems in a flow, provided they match your target user. Run several small rounds rather than one large one, fixing issues between rounds. If you need a reliable completion rate, an unmoderated study with a larger sample is the better tool.
What is the difference between usability testing and A/B testing?
Usability testing explains why people struggle, using a handful of observed sessions. A/B testing measures whether a change improves a metric, using live traffic. Teams get the best results by using the first to decide what to change and the second to confirm that it worked.
Can I test a prototype with real users before building anything?
Yes, and it is usually the cheapest time to find problems. Build only the screens needed for the task, tell participants some elements may not work, and watch them attempt a realistic goal. Fix the top issues and test again before handing designs to engineering.
How do I test usability for B2B or enterprise products?
Recruit the actual roles that use the product, such as admins, analysts or approvers, since a general consumer panel will not reflect their tasks. Source them from your sign-up list, customers or professional networks. Test each role on its own tasks, because an administrator's setup flow and an end user's daily workflow break in different places.
Sources
- Usability Testing: Methods, Metrics and a 7-Step Process
- Onboarding, Keith Mason
- Unmoderated User Tests: How and Why to Do Them, NN/G
- How SaaS Teams Use Usability Testing to Decide What to Fix
- Experimentation for the User Experience Team, GitLab Handbook
- Using the RITE method to improve products
- How a research platform improved onboarding and engagement, Pendo