I’ve got some data:

It’s basically a small N study, where people are tested for something, then are given some help, and then tested again.

So I have something like this:


before | after
yes 3 | 9
no 13 | 7


and I’m trying to work out what statistical test I need to use.

The closest I can get is Fisher’s Exact Test, but I think I can’t use that because the before/after results aren’t independent?

Any statisticians here?

Joe

Aha, I think I have *just* enough data to use McNemar’s test.

Although what the hell significance means in this small an N study I don’t know.

Joe

or maybe I can’t. This is doing my head in.

chi squared?

chi squared?

Not for things with this small a number of samples, and not where the two tests are correlated (it’s a before-after study).

Joe

Guessing you can’t test for structural break due to only having two data points?

Is it attribute data or variable data ?

Is it attribute data or variable data ?

It’s basically –

People are first given a problem to solve with no help.

There is a tactic that makes this class of problem easier to solve, but to a lot of people it seems like cheating.

After a first failure (all except one of them failed) We give them a hint, saying that they can do anything they like to solve the problem, then next we see if they use the easy tactic or not.

It’s about rules and expectations and stuff.

It is pretty obvious when you’re running this that it has a significant effect, but I’d really like to have some number to back it up.

Joe

There seems to me to be no Statistical in the data.

A simple Ratio / Percentage is the answer

There seems to me to be no Statistical in the data.

Huh? There must be tons of trials like this – test, intervention, test. Surely they do some kind of statistical test for it?

Obviously it isn’t a properly controlled experiment – so I can’t claim that this was definitely the thing that made the difference occur, but surely there is some way to test whether something has changed?

Joe

Aha –

Got it. I write it out in terms of before/after changes like this:


before
y n
a
f y 3 6
t
e n 0 7
r

There are 6 positive changes, and 0 negative changes.

I can apply a simple sign test to it, giving a significance of P<0.031

We are talking stupidly silly small N here though, but I guess that is the nature of small scale pilot studies.

Ta da!

Joe