[Person & Situation] The Problem of Effect Size
Note: This is my attempt to read the book Person and Situation. I am but an interpreter of these ideas.
In statistics, an effect size is a quantitative measure of the magnitude of a phenomenon.
So there are three criteria to define an Effect size, that would affect how we categorize an effect as important or trivial.
Statistical Criteria of Size
Effect size has little to do with statistical significance. An effect of any size is likely to have statistical significance that it didn’t happen by chance. A more sensible definition for effect size is that it should be judged according to the variability of the measure in question. A quarter standard deviation difference between two means is considered small, a standard deviation difference between two means is big.
Pragmatic Criteria of Size
The objection to statistical criteria is that sometimes we don’t care if an effect is qualified as Big by its statistical significance, and sometimes we care even if the effect is qualified as Small by the same way.
For example, if you’re dying of a rare disease that normally makes you die within 40 hours with a standard deviation of 4 hours. You found a medicine that can prolong your life an extra 1.5 standard deviations, which is about 6 hours. If you find out that this medicine costs 10,000$ then your interest would already disappear. On the other hand, if you’re a medical researcher then you’d be jumping at this medicine because it holds the clue to the cure of this disease.
Another example would be that some personality tests only predict 10 percent of the variance in some important outcome can be very “cost-effective” in predicting people who are likely to be extreme in some dimensions.
Therefore, a utilitarian consideration affects our judgement about the effect size. Effects are big or small relatively to wether they’re sufficient to accomplish objectives with reference to how much we care about those objectives.
Expectation Criteria of Size
Effect sizes are big if it forces a revision on our prior beliefs and the underlying theories that produced the prediction. It is worth that in this context, statistically small effects can sometimes force us to re-think well established theory provided we have very good reasons to expect no difference at all.
In the end
In the end, it depends on context we use to evaluate an effect. For example, I had to re-evaluate my self-worth when I failed to understand a concept. That’s pretty trivial to most people, but the effect was quiet astounding with regard to my then-current hypothesis that I was better than most people.
Of course, again, the understanding of the concept doesn’t necessarily validate that hypothesis either, because there are a lot of people who can understand that concept.
So the moral of the story is: When you observe an effect, use the above three criteria to judge it to see if there’s any level of analysis that resonates to you.