Rosie asked me to share this story that I shared on Twitter originally, and who am I to say no to the woman herself? :)
I've built up a programming and experimentation team at Reddit that delivers statistically significant, actionable results from experiments run without any engineering help.
Here's how we do it:
1. Remember the human.
We try to bring the Community team's deep understanding of humans and Reddit to bear to give us unique insight into what might work.
2. Work with our community.
Many of our experiments rely on partnership from moderators. Some of our programs are also scaled through bringing on moderators as paid contractors!
3. Build the most minimal viable product apparatus.
A whole lot of what we do is use our bulk messenger tool to message chunks of users. Will this be what a final product looks like? Unlikely - but it'll get us initial results a lot faster than a software team.
4. Put our effort into the messaging.
How you message, what you say, and how responsive you are can be incredibly impactful.
5. Value insights as much as results.
Most experiments fail, but a quality experiment will give you valuable insights even if it fails.
6. Try to act as a force multiplier.
To the MVP point above, we're rarely building the final product - just paving the way for other teams.
Bonus: Exploit bugs!
I found an experiment blocked this week but a team member figured out a way to use a bug to make it work. :P Whatever it takes to get that initial result!
--
With this approach we've brought some of the earliest and most insightful results to the company around efforts like active communities, internationalization, and more. You can do it too! Just add a dash of SQL knowledge, a huge chunk of empathy, and then apply the above.
