Product-market fit survey: the Sean Ellis test, how to run it and how to read the score

A product market fit survey asks current users one fixed question, "How would you feel if you could no longer use [product]?", and the score is the share who answer "very disappointed," with 40 percent as the line Sean Ellis set after comparing nearly 100 startups. Send it to people who used the core product recently, aim for at least 30 responses and ideally 100 or more, and read the score alongside retention.

What is the product-market fit survey and what does a 40 percent score mean?

The product-market fit survey, often called the Sean Ellis test, is one question you ask people who already use your product. Sean Ellis published his PMF survey methodology in 2010. The wording is fixed: "How would you feel if you could no longer use [product]?" (fitsignal).

Respondents pick one of four answers:

• Very disappointed
• Somewhat disappointed
• Not disappointed
• N/A, for people who no longer use the product

The score is the share of respondents who pick the first option. The published formula divides "very disappointed" responses by total responses and multiplies the result by 100. You exclude "N/A" responses from the total, since those people are not current users.

Where the 40 percent line comes from

Ellis set the line by comparison, and he called it soft. In his words: "Admittedly this threshold is a bit arbitrary, but I defined it after comparing results across nearly 100 startups." He went on: "Those that struggle for traction are always under 40%, while most that gain strong traction exceed 40%." (fitsignal).

Read that sentence twice. In our reading, a score under 40 percent is a firmer warning than a score over 40 percent is a green light, because only most of the strong-traction companies cleared the line.

The bands Ellis described look like this:

Score	What Ellis described
Above 40%	Growth tended to come more easily, with word-of-mouth doing real work
25 to 40%	Growth was possible but effortful, and churn kept reappearing
Below 25%	Growth was difficult at every level

The bar is high in practice. When Slack was growing fast in 2015, an independent survey of 731 Slack users run by Hiten Shah found 51% would be very disappointed without it. Those respondents were recruited through volunteers’ email lists and networks, so treat the figure as an indication.

A worked example

These are round example inputs, not a benchmark. If 40 of 100 qualifying respondents answer "very disappointed," your score is 40 percent, right on the line. If 100 of 400 respondents give that answer, your score is 25 percent, the bottom edge of the middle band.

Your step this week: write the question and the four options exactly as above, and decide who counts as an active user before you send it. Then put the score next to your retention numbers. We treat the survey as one input, read alongside how many people keep coming back.

Where the 40 percent benchmark comes from

A score above the line feels like permission to spend on growth. It can also be wrong, and the failure modes are specific enough to check for before you act.

Start with what the line was meant to gate. The Hacking Growth notes say companies with more than 40 percent "very disappointed" have high growth potential, and the book sets a condition on top of the score: you must not move into the high-tempo growth experimentation push until you know your product is a must-have, why, and for whom. A number without the why and the who does not meet that condition.

The early-product false positive

He describes a team that scored above 40% even though the product did not work well and the market was too small.

Respondents in that case were rating a hope. Our rule for early products: discard any wave where most respondents have not yet used the working product through at least one complete usage cycle, then resurvey only those who have. For a weekly tool, that means several weeks of real use. For a monthly workflow, it means at least a full month.

Small groups and borrowed samples

A high score from a handful of people tells you less than it appears to. Learning Loop notes that a 45% with very few total users might mean a small group really loves it, and you still have to test whether that extends further. The often-quoted Slack result is a reminder of sampling too: its respondents were recruited through volunteers' email lists and networks rather than sampled from Slack's active users.

The retention cross-check

Hacking Growth names a second signal: your retention rate is good compared to other competitors in your space, and stable over time. Learning Loop adds willingness to pay, advising you to check it against retention and willingness to pay, then go for growth. In our reading, a high score that the usage data contradicts should not be acted on.

Warning sign	What it means	What to do this week (our reading)
Respondents joined before the product worked	They rated the promise, not the product	Resurvey only users with a full usage cycle on the working product
High score from very few users	A small group loves it, and reach is untested	Keep surveying new cohorts before you spend on acquisition
Respondents came through networks, not your active base	The sample may flatter the score	Draw the next wave from active-user records
Score high, retention unstable	The two signals disagree	Hold growth spend until retention flattens
No clear answer to why and for whom	The must-have condition is not met	Code the open-ended answers for core benefit and user type

The four-question PMF survey template you can copy

The disappointment question gives you the score. The follow-ups give you the reasons, and the reasons become the work plan. Fitsignal pairs the core question with three follow-ups and advises keeping the survey to "the core question plus 2 or 3 follow-ups at most". That gives the four-question shape below.

Who receives it matters as much as what you ask. Superhuman followed Ellis's advice to "focus on those who used the product at least twice in the last two weeks" (First Round). Its first run came back with only 22% choosing "very disappointed".

Question	Answer format	Why it is there	How to code replies (our method)
The disappointment question covered above	Single choice, the fixed options from earlier	Produces the score	No coding. Use this answer to split every other reply into groups.
"What type of people would most benefit from [product]?"	Free text	It helps you refine your ideal customer profile	Tag the role, company type or situation named. Count tags within the very disappointed group first.
"What is the main benefit you receive from [product]?"	Free text	From the very disappointed group, it tells you what to double down on	Tag one benefit per reply in the user's own words. Merge near-duplicates once at the end.
"How can we improve [product]?"	Free text	From the "somewhat disappointed" group, it reveals the blockers keeping them from becoming advocates	Tag the missing feature or friction. Rank tags by count within that group.

Copy this sheet into a spreadsheet and fill one row per respondent:

respondent_id | disappointment (very / somewhat / not) | persona_tag | benefit_tag | improvement_tag | verbatim_quote

Summary tab:
theme | count_very | count_somewhat | count_not | example_quote

Code free text the same day it arrives. One person tags every reply so the themes stay consistent. A second person spot-checks a sample.

As an example, not a benchmark: if 40 of 100 "very disappointed" respondents name the same main benefit, that theme carries 40 percent of that group's benefit answers. In our reading, that theme belongs in your homepage headline and onboarding before any new feature work.

On volume, Ellis says 30 responses makes the survey directionally useful, and he is much more confident at 100 or more. Do not pay people to answer. As Fitsignal puts it, "you want honest answers, not people clicking for a reward".

If the improvement themes are vague or scattered, a survey alone will not explain them. Book short calls with a handful of "somewhat disappointed" users. Our guide to customer interviews by cold email covers how to get those calls this week.

Who should you send the PMF survey to?

Who you ask decides what the score means. The same product can look strong or weak depending on the list you send it to, so set the sampling rules before you write the invite.

Survey people who reached the core experience

Typeform describes the usual filter: people who have experienced the core of the product recently, meaning they have used it more than once and within the last few weeks. In practice, define one core action that signals someone got real value, then pull everyone who did it more than once in your recent window. Our activation definition is a good place to start.

Leave out signups who never got there. Typeform puts the risk plainly: including people who downloaded (but never opened) your app will unfairly punish your score. The opposite error also distorts it: if you only hear from your loudest fans, your score will flatter you. Rahul Vohra made a related point at Superhuman: early marketing pulls in users who won't be well-qualified and have no real need for the product.

If your base is small, Typeform suggests surveying in rolling waves as new users cross the active threshold. Tag each wave as a cohort so you can compare rounds later.

Pick the moment and the channel

Typeform advises against surveying right after a pricing change, an outage, or a big launch. Email reaches people too, but it brings fewer responses and a skew toward strong feelings in either direction. Whatever channel you choose, keep it the same from round to round.

Decide on N/A before you count

Tools differ on the denominator. Satismeter, for example, computes the score as a percentage of all responses that picked very disappointed. The method covered earlier drops N/A answers. Here is how much that choice moves the result, with example inputs, not a benchmark: 400 responses, 100 of them N/A, 40 very disappointed.

• Counting all 400 responses, 40 very disappointed answers is 10 percent.
• Removing the 100 N/A answers from 400 leaves 300 responses, and 40 of 300 is about 13 percent.

Neither number is wrong. The problem comes when you mix them across rounds. Record the rule once and keep it attached to every result:

PMF survey sampling log
Round:
Date sent:
Eligibility rule (core action + window):
Excluded groups:
Channel (in-product / email):
Responses received:
N/A responses:
Denominator rule (all responses / excluding N/A):
Very disappointed count:
Score:
Unusual events that week (pricing, outage, launch):

In our view, a score you cannot reproduce is worse than no score. Fill in this log before you read the result, and use the same rules next quarter.

How many responses do you need?

A 40 percent score is a point estimate. Whether it means anything depends on how many people answered, and that is a calculation you can run in a minute.

The standard formula for the margin of error of a proportion, as Qualtrics sets it out, is MOE = Z-score * SQRT [ p-hat * (1 - p-hat) / n ]. The z-score for 95 percent confidence is 1.96.

The calculation

Paste this into any Python shell. The comments show what it prints. The second step uses the round example inputs 40, 100 and 400, which are an example, not a benchmark.

import math

Z = 1.96  # 95 percent confidence

def moe(p, n, population=None):
    se = math.sqrt(p * (1 - p) / n)
    if population:  # finite population correction
        se *= math.sqrt((population - n) / (population - 1))
    return Z * se

# margin of error, in points, for a 40 percent score
for n in (30, 50, 100, 200, 400):
    print(n, round(100 * moe(0.40, n), 1))
# 30 17.5
# 50 13.6
# 100 9.6
# 200 6.8
# 400 4.8

# 100 replies from a base of 400 active users
print(round(100 * moe(0.40, 100, population=400), 1))  # 8.3

# a 38 percent score on 50 replies: low and high end
m = moe(0.38, 50)
print(round(38 - 100 * m, 1), round(38 + 100 * m, 1))  # 24.5 51.5

How to read it

The margin shrinks with the square root of the sample. Quadrupling your responses only halves the margin, which is why the jump from a few dozen replies to a few hundred matters more than any later jump.

The last block is the case that trips founders up. A score just under the line on a small wave produces an interval that runs from the bottom band to well above the threshold. You cannot tell it apart from a pass, and you cannot tell it apart from a clear fail either.

The finite population correction matters when you survey a large share of a small user base. For small populations, a higher percentage needs to be sampled, but every reply you get also covers more of the whole group, so the correction tightens the margin. Use it when your eligible base is a few hundred people.

Two checks before you trust the number

• Ask how the sample was drawn. Margin of error is used when you have a random or probability sample. Qualtrics adds that it won’t be relevant to your market research data if your sample has been selected in a non-random way. A survey that only your keenest users answered carries bias that no sample size fixes. In our reading, this is the more common failure, so the targeting rules from the previous section come first and the arithmetic second.

How to segment the answers when you score under 40 percent

A score under the line tells you where to look next. Typeform's guide to the test gives each answer group its own job: the very disappointed group defines your real market, and building for the not disappointed group pulls you away from your core.

The five steps

1. Profile the fans. Pull every "very disappointed" response. Read their answers to the persona and main-benefit questions, then write a one-paragraph profile in their words: role, situation, and the job they use the product for.
2. Tag every respondent. Mark each person as fitting or not fitting that profile. Use their persona answer plus whatever account data you hold, such as role or company size.
3. Re-score the segment. Count the very disappointed answers among fitting respondents only, then divide by the number of fitting respondents. As an example, not a benchmark: if 100 respondents fit the profile and 40 of them answered very disappointed, the segment score is 40 percent. Report this next to the overall score.
4. Find the blockers. Filter to somewhat disappointed users inside the segment whose main benefit matches what the fans named. Their answers to the improvement question are your blocker list. Typeform puts the condition plainly: these requests count only where their requests align with what the very disappointed group already loves.
5. Split the roadmap. Run two tracks. One doubles down on the benefit fans love. The other fixes the blockers from step 4. Score items in both tracks in your growth experiment backlog with an ICE score so neither track takes over by default.

Leave out requests from somewhat disappointed users who value a different benefit. Chasing every request from lukewarm users dilutes the product. Log the not disappointed feedback and set it aside.

A scoring sheet to copy

Columns:
respondent_id | answer (very / somewhat / not) | persona_answer | main_benefit
| fits_profile (Y/N) | benefit_matches_fans (Y/N) | improvement_answer

Overall score   = count(answer = very) / count(all)
Segment score   = count(answer = very AND fits_profile = Y) / count(fits_profile = Y)
Blocker list    = improvement_answer WHERE answer = somewhat
                  AND fits_profile = Y AND benefit_matches_fans = Y
Fan-love list   = main_benefit WHERE answer = very AND fits_profile = Y

Then repeat the survey on a schedule. Typeform suggests that re-surveying fresh users each quarter shows whether your roadmap is converting lukewarm users into people who'd genuinely miss you. We think the segment score is the number to watch across those rounds, because it tracks the market you chose to build for.

Worked example: a 31 percent score segmented to a 44 percent core

This is our own example, built to show the method end to end. The figures are illustrative, not a benchmark.

The team does not read it as failure. It reads it as a signal to segment.

Step 1: filter to the fans' role

The persona question asks what type of people would benefit most. Among the very disappointed group, one role comes up most often: operations managers. Filtering the whole sample to respondents in that role leaves 64 responses, and 28 of them answer very disappointed. That is 44 percent (28 divided by 64).

The smaller base widens the margin to about plus or minus 12 points, so the core's true score could run from 32 to 56 percent (44 minus 12, 44 plus 12). In our reading, this does not prove fit. It does show where the product already matters most, and that tells the team where to point the roadmap.

Step 2: the filled template

PMF survey readout, Q3 wave (example data)

All active respondents
  Responses:            140
  Very disappointed:    43
  Score:                31% (+/- 7.7 pts, 95% confidence)

Core segment: operations managers
  Responses:            64
  Very disappointed:    28
  Score:                44% (+/- 12 pts, 95% confidence)

Main benefit themes (very disappointed, core segment)
  1. Fewer manual handoffs between teams
  2. One place to see request status
  3. Faster weekly reporting

Top blockers (somewhat disappointed, core segment)
  1. Setup takes too long
  2. Missing integration with the team's ticketing tool
  3. Reports cannot be shared outside the account

Decision: build for operations managers next quarter

Step 3: code the benefit themes

Read every main-benefit answer from the core's very disappointed users. Tag each one with a short theme label, then merge labels that mean the same thing. Three themes held up in this example: fewer handoffs, a single status view and faster reporting. These become the words for the homepage and onboarding.

Step 4: choose three fixes

The team kept only the somewhat disappointed requests that matched the core's benefits and dropped the rest. It chose three fixes for the next quarter:

1. Cut setup time with a template preloaded for operations teams.
2. Build the ticketing integration, since it feeds the single status view.
3. Add shareable report links, which extends faster reporting.

Each fix ties to a benefit the core already values. The team will re-run the survey on fresh core users after the release and compare the segment score against this wave.

How the survey compares with retention, and what to do this week

The survey and the retention curve answer different questions. The survey records what users say they would feel if the product went away. The curve records what the same users did.

His reasons are practical. The surveys are easy to deploy and require the least amount of data, but they also have the highest chance for generating a false positive. The retention curve has the opposite trade. In Balfour's words, it takes time to collect retention data, and B2B SaaS companies take the longest because they are usually measured on monthly churn.

Mixpanel makes the case for usage data from the other side. It holds that retention is the best indicator of product-market fit, and that survey methods can't help you optimize PMF or experiment with different hypotheses.

Dimension	PMF survey	Cohort retention curve
What it records	Stated reaction to losing the product	Share of a cohort still active over time
Speed to a result	Fast, little data needed	Slow, needs time to accumulate
Main weakness	Can produce a false positive	Long wait, longest for B2B SaaS
How we use it (our reading)	Find the core segment and its reasons	Confirm that segment keeps using the product

Balfour's test for the curve is simple: plot the share active over time by cohort, and IF it flattens off at some point, you have probably found product market fit for some market or audience. Our guide to reading retention curves covers the plotting, and our monthly churn benchmarks give context for subscription products.

The plan for this week

1. Pick the sample. Use the filtered, recently active users described earlier, and note their signup cohort.
2. Send the four questions. Use the template above, with no incentive.
3. Score with the margin of error. Report the very disappointed share as a range, not a single number.
4. Check the retention curve of the same cohort before you scale. If the core segment's curve flattens, increase spend on that segment. If it keeps falling, treat the survey score as unconfirmed.

Decision rule (our method)
Survey core segment at or above threshold + same cohort's curve flattens  -> scale to that segment
Survey core segment at or above threshold + curve still falling           -> fix activation, re-test
Survey below threshold in every segment                                    -> keep building, re-survey
Default cadence: re-run the survey every quarter on fresh users

We treat the quarterly re-run as the default. Balfour's reason applies here: As your market moves, your product needs to move with it.

What this page does not cover

This page does not cover how to plot retention curves step by step, churn benchmarks by industry, or a comparison of survey tools. It also does not cover how to test willingness to pay. It is not financial advice, so decisions about growth spend remain yours to make.