Small Experiments
Getting Your Membership to Try New Things
Summary
The concept of experimentation seems risky. When associations are looking for new ways of delivering value, why not look for the sure bet? This is the irony. Learning how, and getting good, at experimentation creates surprise and unforeseen openings. It allows us to try new things without expending enormous resources and it is an entree to create focused value for our members.
Mostly, however, experimentation is the best way to learn from, and get to know, our members. The outcome is deep understanding of member problems, which is the easiest route to durable community building and empathizing and building relationships with stakeholders hungry for new value.
This article will make the case for experimentation and present an example for how it can work for you and your association. I lead the product community, a product development learning community designed specifically for associations. Let’s compare ideas and build something great.
Why Experiment?
“Experiment relentlessly, then scale what works. Don’t assume you know what will work. Try a ton of different ideas, campaigns, and strategies, and double down on what actually moves the needle. When you find something that resonates, turn it into your strategy.”
Marc Benioff
The traditional (and most common way) to create new member value is based on hunches, untested trends, random volunteer requests, or simply because another organization has found success with a product or program. People don’t want more stuff. They do, however, want solutions so they can deepen connection, solve their problems, and advance their career or industry.
The quickest route to engaging your members meaningfully is to try lots of small experiments to see what resonates. This is how best-in-class product development works as it helps us identify what members are struggling with (especially when they have a hard time identifying or articulating this struggle).
In this way, small experiments are intentionally limited tests designed to validate specific assumptions, features, or concepts with minimal resources and risk. They focus on learning quickly before committing to fully developing a new product, program, or offering. I’ve written lots of articles on learning from your members.
Fifteen Ways to Learn from Your Members. A guide for striving associations.
How to Define an Ideal Member. Developing and using ideal member profiles to create deep connection.
Deep Understanding. A framework to identify and respond to member problems.
Together, these articles reflect the importance satisfying member needs through new programming, offerings, or products. To do so well, we need to build relationships, understand what makes them tick, visit their workplaces, grasp what it means to walk in their shoes, and get a sense of what they struggle with and what opportunities they see day-to-day. We can then start to see patterns of engagement that give us insight and direction.
Best Practices
“Every new opportunity carries uncertainty. But that doesn’t mean it’s a gamble.”
Marc Sniukas
Most associations take a traditional approach to product development and member outreach. Member outreach is often satisfied through surveys, focus groups, or data collection. We also learn a lot about members through volunteers.
Product development tends to be dominated by a few cash cows that command the lion’s share of revenue (think events, membership, or training). We build around these legacy programs (sapping resources and staff capacity) making it hard to try new things. I argue we can break out of this by learning, and getting good at, running experiments.
Small experiments are powerful tools for learning about your member community as you refine your approach with minimal risk. They let you test ideas quickly before committing significant resources, gather real-world data rather than relying on assumptions, and create a culture of continuous improvement. When you run small experiments, you can discover what truly resonates rather than through guesswork. This reduces the risk of large-scale failures and helps build evidence-based confidence in your decisions.
Best Practices for Running Small Experiments
Start with a clear question or hypothesis you want to test
Keep the scope focused and manageable
Define specific, measurable success metrics before you begin
Test with a representative but limited sample of your community
Set a firm timeframe with a definite end date
Document everything, including unexpected results
Be prepared to end experiments that aren't working
Share results openly with your team, even negative ones
Implement winning ideas quickly while learning from failures
Leverage insights to enhance future experiments
These practical steps help ensure your experiments deliver meaningful insights efficiently, allowing you to better serve your member community through evidence-based improvements. Let’s now look at a simple example of how it works in practice.
How to Run a Small Experiment
“It’s impossible to reach the sky if we limit ourselves to climbing a forty-foot ladder.”
Stefan H. Thomke
Here is how to run a simple experiment for a community platform. Say an association sees gaps in how members engage with each other. Members express a need to get to know, and work with, other members outside standard offerings like annual events. Instead of jumping in and buying a new community platform, they decide to run an experiment. These criteria guide their journey:
Narrow scope with clear learning objectives
Quick to implement (days or weeks, not months)
Low resource investment
Designed to provide actionable insights
Often deployed to a limited audience
Instead of building the entire platform, they create a simple weekly email newsletter with discussion prompts and a link to a basic forum thread. Here is how we would go about implementing the experiment:
Select 50 diverse members as participants
Send 4 weekly emails with different discussion topics
Track open rates, click-throughs, and participation
Conduct brief follow-up interviews with 5-10 participants
The most successful experiments are designed for usable outcomes. Here is some insight on how how this might work:
Which topics generate the most engagement
Members' preferred frequency of communication
Whether members value peer discussions
What obstacles prevent participation
This small experiment would take just a month to complete and provide critical insights before investing in a full community platform development. The association could then run additional targeted experiments based on these learnings.
Forever Connection
“Prototype your life. Try stuff instead of making grand plans.”
Kevin Kelly
Small bets pay off in ways that big bets never can. Associations are notoriously risk averse so getting in the game of trying things by using experimentation holds great promise. It’s how we can keep fresh ideas churning and it’s how we can strengthen our connection and learn from our members.
Mostly, experimentation is how we can resist old habits and get durably and meaningfully in the innovation game. In this way, getting good at experimentation is getting good at building an indispensable community.
I lead the product community; we are a learning community because we believe great relationships help us create the value our members want. Remember, product-led growth fuels connection. Join the product community and flip your destiny.
About the Author
James Young is founder and chief learning officer of the product community®. Jim is an engaging trainer and leading thinker in the worlds of associations, learning communities, and product development. Prior to starting the product community®, Jim served as Chief Learning Officer at both the American College of Chest Physicians and the Society of College and University Planning. Please contact me for a conversation: james@productcommunity.us