Data Analysis Question: When dealing with significance levels, should I use p < .001 or p < .05? Also, should it change between tests or stay consistent throughout my analyses? (Matt, Chicago, IL)

In statistics, the level at which one seeks to find a significant p-value is known as "alpha". The most common levels of alpha are p < .05, p < .01, and p < .001. The decision about which to use is a difficult one, and is somewhat subjective. Essentially, alpha is the degree of chance that a researcher is willing to accept that the inferences they take from any given analysis are made in error. In other words, if I choose to use an alpha of .05, I'm accepting that there is an approximately 5% chance that I'll make assumptions from my results that may be an inaccurate representation of the population I am seeking to analyze. An alpha of .05 is the most commonly used alpha, although lower values of alpha (such as p < .001) are considered more conservative. There is no right or wrong answer about which to choose, although it is typically encouraged to keep the alpha statistic consistent across each analysis conducted in any given project, and to make the decision about which to use prior to running your analysis.

ANOVA, MANOVA, and MANCOVA Question: What is the difference between ANOVA, MANOVA, and MANCOVA? (Eric, Lafayette, IN)

The difference can definitely be confusing. There are differences on a few different levels. First, an ANOVA is different from both a MANOVA and MANCOVA because an ANOVA has only one dependent variable, while both a MANOVA and MANCOVA have multiple dependent variables. An ANOVA typically compares a continuous (a.k.a interval or scale variable) between multiple independent groups of responses (usually 3 or more groups).

By contrast, both a MANOVA and MANCOVA have multiple dependent variables, but there are differences between the two as well. The difference between a MANOVA and MANCOVA lies in the number of independent variables. A MANOVA, like an ANOVA, has only one independent variable (which is typically a categorical variable that represents independent groups) and compares multiple dependent variables between independent groups.  A MANCOVA is a similar concept to MANOVA, except it allow for multiple independent variables (a.k.a. covariates).

In a MANCOVA, one is able to examine  multiple dependent variables for differences between independent groups, while controlling for other variables that may also be related to the DV. These covariates may be either categorical or continuous. I hope this helps, Eric! Please keep the questions coming!

Regression Analysis Question: What is the difference between a mediator and a moderator? (anonymous)

Great Question! The difference is basically parallel to the difference between explaining something and changing something. In statistics, a mediator is a variable that explains the relationship between two other variables. For example, age may be hypothetically related to having a higher income, but that may be explained by age's association with work experience, which may itself be related to a higher income. If work experience accounts for a significant portion of the variability in income that is explained by age, then work experience is a mediator of that relationship (there are different kinds of mediators also, such as full or partial, but we won't get into that here).

While a mediator explains, a moderator changes. When the strength of the relationship between two variables is dependent on the value of a third variable, that variable is called a moderator. With respect to moderators, the easiest example is often gender. For example, let's pretend that age was associated with liking of ice cream, but that was only true for boys, while girl's liking of ice cream did not tend to vary with age. In this case, one's gender determines whether their age is related to their liking of ice cream. In an entire sample of people, a researcher might say that knowing an individual's gender would CHANGE the extent that they are able to predict one's liking of ice cream from their age.

This is a topic that is commonly confused, so much so, that I made a video about it (which also includes information about suppressors)! Check out my video about mediators, moderators, and suppressors HERE! Thanks for the question and keep them coming!