Say you make a quick poll on Twitter and you only get 7 answers, because you’re not even internet-famous:

Sad, but we have to make the best of it.

When I created that poll, I imagined that the vast majority of people would agree with the statement.

Can I still hold that belief, even after seeing the handful of “disagree” votes? Could it be that, if we got more participants, the proportion of “disagree” answers would actually go down?

Let’s clarify the hypothesis: “The true proportion of people that disagree with the statemement is small (less than 5%).”

Can we prove or disprove this hypothesis, given the modest number of votes?

If we assume that the subset of people that voted in the poll are a representative sample of all my Twitter followers — a big if — then the answer is yes.

We can calculate the probability of an imaginary next vote. Here’s what that looks like at the 95% confidence level:

  • agrees with the statement: between 0% and 63%
  • disagrees with the statement: between 22% and 92%
  • rarely takes notes: between 0% and 49%

Or, in graphical form:

poll confidence intervals

Those are very wide intervals, but they still tell us something. If we were able to get each of my followers to vote in the poll, we would end up with at least 22% of them picking “disagree”. 22% is bigger than 5%, so the hypothesis is disproven.

Science: using math to prove yourself wrong.

You can check out my calculations in this gist. For more details on statistical inference, you could start with this Wikipedia page.