OPEN BUBBLES | Thoughts on startups from an entrepreneur, tree hugger, and geek in stilettos.
  • HOME
  • ABOUT
  • STARTUP
    • Analytics
    • Being a Woman Tech Founder
    • Minimum Viable Product (MVP)
    • Personalization
    • Public Relations
    • Social Enterprise
  • CONTACT
Data and Cohort Reports are Dope
Analytics, Startups 0

How to Build a Cohort Report

By Adriana Herrera @adriana_herrera · On May 18, 2014

PROCESS FOR BUILDING A COHORT REPORT
In many ways, the process for pulling out cohort data is very similar to the way data is pulled for FashioningChange’s Daily Reports.  The difference for cohort reports is that for every value we’re interested in, we want to look at it for only a subset group of users. The cohort reporting tools we’ve established start from a particular week, pull out the cohort of users that signed up during that week, and then go through each week after that point calculating the metrics for that group of users only.  The tools then go back to the beginning, move forward one week, pull out the new cohort, and repeat.

For example, looking at the behavior of customer retention, the cohort report is fairly simple. Our systems record each day that a user visits, so all that the cohort report needs to do is look for each cohort and week and which users within the cohort visited during that week.The process is pretty straightforward once you’ve done it a few times. And, as with Daily Reports you don’t need to be technical to establish your own cohort reports because tools like KissMetrics automate this process for you.

INCLUDING A/B TESTS IN COHORT REPORTS
An important element of any A/B test is to determine the metric you’re interested in.  This metric can be as simple and direct like a click-through rate on a particular link or button. What a cohort report allows you to do is investigate the long term impacts of that metrics on a variable like retention rate after 6 weeks (funzies and awesome :) ).  The details of implementing an A/B test vary with the experiment that is being run. Some tests are very complicated and require deep changes in the structure of the application, while others are as simple as testing a different set of wording or color and can be done without any technical changes using a tool like Optimizely.  What makes setting up an A/B test within a cohort report unique is that in order to be able to test the long term impact within a cohort the group any particular user is in needs to be recorded. We set ourselves up to do this by including the experiment group within a hidden set of ‘preferences’ which we store in association with each user. This allows us to generate cohorts based not only on time of signup, but also experimental group.

STATISTICAL RELEVANCE
I have a bad habit of saying “data doesn’t lie.” I say it’s a bad habit because if an experiment doesn’t have statistically relevant results the data can throw even the smartest of people in the wrong direction. We will typically run an experiment until we either see relevance, or we have several thousand data points with no observed difference, at which point we conclude the experiment has no impact. Whatever the metric you’re measuring you need to make sure you compare between your experiment groups using a statistical measure of relevance.  We typically use a chi-squared test and look for a p-value less than 0.05, which translates to a 95% chance that observed differences are not due to random chance.  The more data you have, the smaller the difference can be and still be determined to be real.

What tools do you like to use to build cohort reports? Share them below.

**Special thanks to FashioningChange Co-Founder and CTO, Kevin Ball, for sharing insights to put together this blog post.**

a/b testcohort reportDatastartupstatistical relevance
Share Tweet

Adriana Herrera

Adriana Herrera is a tech entrepreneur, tree hugger, and geek in stilettos. She spends her time building EpicHint.com, surfing, and cooking with family.

You Might Also Like

  • Woman with finger against lips asking for quiet Startups

    Does success in the Silicon Valley come from silence?

  • Developing and Using Cohort Reports | Supporting Product Decisions with Data Analytics

    What is a Cohort Report?

  • Things I Think As a Woman Tech Founder Being a Woman Tech Founder

    Thoughts That Go Through a Woman Startup Founder’s Mind

No Comments

Leave a reply Cancel reply

Subscribe By E-mail

About

Adriana Herrera
Hi my name is Adriana Herrera. I'm an entrepreneur. I've founded companies in tech and outside of tech. I taught myself to code while drinking wine one summer. I wear stilettos and have a closet full of dresses I love! I've written columns in publications like The New York Times. I was named a Top 20 Latino Innovator by NBC/ NBC Latino. Yup, I'm Mexican. I'm very proud of my heritage and the amazing cooking skills passed down to me by my mother and abuelita.

I love reading blogs. I've found that most of the blogs I read are written by men so I thought I'd start my own, Open Bubbles. I write about what I know...building a startup, PR, marketing, data, and developing product. I like conversation and new friends. So feel free to comment, 'Connect', and follow.

Thanks for stopping by!

With Aloha,
Adriana Herrera
adriana@openbubbles.com

Learn about the 'Open Bubbles' name here.

Subscribe & Follow

Follow @@adriana_herrera
Follow on Instagram
Follow adrianaherrera
'FOLLOW' if you support, and want to see more, women in tech.
  • HOME
  • ABOUT
  • STARTUP
    • Analytics
    • Being a Woman Tech Founder
    • Minimum Viable Product (MVP)
    • Personalization
    • Public Relations
    • Social Enterprise
  • CONTACT

ABOUT

Open Bubbles is a site dedicated to information on startups, PR, marketing, product development, data, and being a Mexican woman entrepreneur.

Click here to learn about the entrepreneur, tree hugger, and geek in stilettos behind Open Bubbles.

Categories

  • Analytics (6)
  • Being a Woman Tech Founder (2)
  • Email Marketing (1)
  • Human Rights (2)
  • Minimum Viable Product (MVP) (4)
  • Personalization (1)
  • Politics (2)
  • Public Relations (2)
  • Social Enterprise (1)
  • Startups (21)

Latest Posts

  • Using, or not using, #BlackLivesMatter, #BrownLivesMatter, and #AllLivesMatter

    Using, or not using, #BlackLivesMatter, #BrownLivesMatter, and #AllLivesMatter

    June 3, 2020
  • Public phone key pad

    Migrant Detainees Charged $8.00/ Minute to Use the Phone. Donate to Give Them a Voice.

    June 22, 2018

Search the Site

© 2013 Open Bubbles. All rights reserved.