Comparing Cronbach’s Alpha and McDonald’s Omega Reliability

For an IQ test, reliability is what makes a score interpretable rather than just a number. Cronbach’s alpha is the most popular measure of reliability in the psychological literature (McNeish, 2018; Warne et al., 2012). It is not hard to see why. Obtaining the data needed to calculate Cronbach’s alpha only requires one test administration (unlike all other classical reliability statistics, which require multiple test administrations), and the statistic is reported in every statistical software program that psychologists use.

But there are problems with Cronbach’s alpha, and these have been known for decades. Cronbach’s alpha assumes that (1) the scale is unidimensional, (2) the items are on a continuous and normally distributed scale, (3) the items are equally good at measuring the construct, and (4) error from the items are uncorrelated (McNeish, 2018). These assumptions range from the easy to meet (e.g., unidimensionality), to the wildly unrealistic (e.g., that items measure the construct equally well). When these assumptions are violated, Cronbach’s alpha underestimates reliability.

I never worried too much about the underestimate of reliability from Cronbach’s alpha. After all, if Cronbach’s alpha showed that my score was high enough, then what was the problem? If it’s an underestimate and still shows that my test scores are reliable, then there really isn’t anything to worry about.

Although there may be little practical problem, a biased estimator (like Cronbach’s alpha) is still producing inaccurate results. That is why many methodologists prefer other measures of reliability. One that has gained popularity recently is called McDonald’s omega. This is a measure of reliability that does not have the assumptions of Cronbach’s alpha. In fact, it explicitly permits violations of the assumptions of Cronbach’s alpha (McDonald, 1999).

While preparing data for the launch of the Reasoning and Intelligence Online Test (RIOT), I had the chance to compare values for Cronbach’s alpha and McDonald’s omega. For each of the non-speeded subtests on the RIOT, I calculated both alpha and omega. The results are in the table below.

Comparing the two right columns, it is clear that Cronbach’s alpha was always the lower reliability value, consistent with the claims from methodologists that alpha underestimates reliability. On average, the increase of .031; as a percentage, the average increase is 4.5%. The Matrix Reasoning subtest had the smallest increase (.008, or 1.0%), while the Visual Reversal subtest had the largest increase (.082, or 13.0%).

Most of these increases in reliability are not very large. This indicates that for most subtests, the violations of the assumptions of Cronbach’s alpha were not severe. For the RIOT, alpha does not greatly underestimate reliability for most subtests.

There are a few reasons for this. First, the items on most of these subtests are in the same format. For example, every Object Rotation subtest shows a target image and then asks the examinee to identify which of the five images below depicts the same object. You can see an example from the RIOT below. All of the Object Rotation items are similar: one target image, five options, same grey background, etc. There is not a lot of room to violate the unidimensionality assumption on a subtest because the items are all so similar.

An example item from the RIOT: Which of the 5 options at the bottom is the same as the image at the top?

In the “psych” package in R (Revelle, 2025), the dimensionality of the items is reported when the package calculates McDonald’s omega. For almost all of the RIOT subtests, the output showed that there were slight violations of undimensionality. In addition to one overriding factor for each subtest, there was at least one other minor factor (usually two). These minor factors were usually difficulty factors: one in which easy items loaded onto a minor factor, and another in which difficult items loaded onto a minor factor. This violation of strict unidimensionality is not concerning.

Another reason why RIOT subtest items do not severely violate the assumptions of Cronbach’s alpha is that they are good sets of items. Every subtest consists of items that survived multiple tryouts. All of the items were scrutinized closely, and any that were not functioning well were thrown out. Even though I never calculated reliability statistics for items in the tryout phases, by eliminating items that functioned poorly (e.g., with low item discrimination values, or items that were excessively difficult or easy), I was also unwittingly helping the remaining items meet the assumptions of Cronbach’s alpha.

So, the take-home message of this blog post is that Cronbach’s alpha is, indeed, an underestimate of reliability. But when subtests consist of well constructed items, the underestimate from using Cronbach’s alpha instead of McDonald’s omega is usually pretty small — about 4.5%.

McDonald, R. P. (1999). Test theory: A unified treatment. Lawrence Erlbaum Associates.

McNeish, D. (2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23(3), 412-433. https://doi.org/10.1037/met0000144

Revelle, W. (2025). psych: Procedures for psychological, psychometric, and personality research, version 2.5.3. Northwestern University. https://cran.r-project.org/package=psych

Warne, R. T., Lazo, M., Ramos, T., & Ritter, N. (2012). Statistical methods used in gifted education journals, 2006-2010. Gifted Child Quarterly, 56(3), 134-149. https://doi.org/10.1177/0016986212444122