The skewed regression
In this blog post, I will focus on a pre-print article (available here) that purports to show that hours of video game usage is associated with higher subjective well-being. This manuscript was recently featured in this BBC article (despite its not yet being peer reviewed), with a headline suggesting that video games are "good for well-being".
This generalization has a number of interesting aspects (including the fact that the game was specifically a "social" type game involving interactions with other players, and the fact that data were collected in August 2020, 5 months into the COVID-19 pandemic), but these are not the focus of this post. Additionally, my goal here is not to pick on the authors of this particular study, but rather to explore two issues it raises in further depth than I have attempted before.
These issues are:
- Large sample sizes will produce significant results even when effect sizes are minuscule (and largely uninterpretable)
- Skewed distributions can produce a result that is biased towards data points in the heavy tail
The analysis in question, based on ~6000 participants, is portrayed in Figure 3 of the pre-print. Hours of games played in a two-week window is plotted on the X axis, and subjective well-being is plotted on the Y axis. The immediate concern is that the data points show a strong positive skew: sparse data points corresponding to people with extremely high gameplay hours (50+) appear to be driving the upward slope of the regression line.
This is concerning — and what's more concerning is that the regression itself, while significant, has an \(R^2=0.01\), indicating that only 1% of the variance in the data is shared between these two variables. This is scant evidence on which to conclude the observed relationship is generalizable to the general population. Another concern we see is a floor effect for gaming hours: many participants have zero hours, indicating that they do not play video games at all, and should likely be removed from this analysis.
Simulating the data
Matlab code for what follows is available here.
A great way to delve into these sorts of situations is to simulate your data. This allows you to control the factors you are interested in — such as skewness and effect size — and see how these influence the resulting statistics. A simple way to simulate the skewed distribution from Figure 3 (specifically, the red graph showing results for "Animal Crossing") is shown below. Here I've generated two sets of random Gaussian variables: one to simulate the vast proportion of participants who played between 0 and 20 hours per week (blue), and another to simulate the players with more extreme levels of gameplay (green).
This bears at least some resemblance to the data of Figure 3. Importantly, this simulation approach allows us to vary the sparsity of the green data points as a percentage of the blue ones — here I've chosen 5%. This translates closely to the proportion of "extreme" values — if we define this as \(X>20\) above, we have roughly 4.5% extreme values.
We can also define the offset between the mean of our green and blue distributions. This offset will influence the correlation between our two variables.
It should be clear where this is going: we should be able to see whether, and to what extent, the relatively few extreme data points will influence our resulting statistics. This simulation will also allow us to see the influence of applying a data transformation, such as a logarithmic function, on these statistics.
Extreme values disproportionately influence regression results
The image above shows some simulated data, with about 4.5% of the data points being defined as "extreme", where \(X>20\) hours (shaded areas are 95% confidence intervals). The grey plots at left show the distribution and corresponding regression model when our extreme distribution is not offset from the main one — as expected, there is no association evident. The red plots show a positive offset of 1.5. While these are not the actual data points of the study in question, they look similar enough — and the positive offset produces a similar statistical result of \(R^2=0.01\).
This plot tells us two things:
- A shift in only 4.5% of the extreme data points resulted in a change from nonsignificant to significant, and produced a result similar to that reported in the article
- A log transformation of the data, to remove the substantial positive skew, reduces this effect
Amazingly, the regression models for the log-transformed data still produce significant results at the conventional \(\alpha=0.05\) threshold, but with greatly reduced effect size (about a third of that for the skewed data). This shows just how easily one can obtain a "significant" result from a large sample, and how important it is to go beyond the p-value threshold and consider the actual effect size when interpreting such models.
Exploring proportion and offset of skewed data points
Both the number of data points in the heavy tail (as a proportion of total points) and the magnitude of the mean offsets will influence the statistical results. These relationships are explored in the two sets of plots below. Here, I generated 50 random skewed data sets to get a better idea of how these effects generalize, and varied either the proportion of data points in the skewed set, or the magnitude of the mean offset.
The top plot tells us that it is possible to obtain a model with \(p<0.05\) with a very small percentage (around 2.5%) of skewed data points. While this is shifted only slightly by log-transforming the data, this step reduces the \(R^2\) value by about half. This basically shows that significant testing is trivial with such a large data set.
The bottom plot tells us that we can also obtain a significant result with a relatively small shift in the mean of skewed data points (+0.5, equivalent to 0.3 standard deviations) from the mean of the main body of points. Again, the log transformation reduces this effect.
Determining the degree of influence of skewed data points
Finally, we can estimate the degree to which each data point influences the regression result by comparing the result to that of a model where that data point is removed. I've done this in two ways: by computing the change in effect size, \( \Delta R^2\), and by computing Cook's distance, which is obtained from the equation:
\[ D_i=\frac{\sum_j^n{(\hat{y_j} - \hat{y_{j(i)}})^2}}{ps^2} \]
Here, \(\hat{y_{j(i)}}\) refers to the predicted value of data point \(j\) when \(i\) is removed from the model, \(p=1\) (the number of predictor variables), and \( s=\frac{\mathbf{e}^T \mathbf{e}}{n-p}\), where \(\mathbf{e}\) is the residual error.
These plots show very clearly the disproportionate influence of extreme data points on the statistical outcomes of the regression analysis: participants with 50 or more hours will have a substantially greater influence on the result than those with fewer hours. This influence is drastically reduced when we apply a log transformation.
Summing up
So how do we distill the above considerations? We can see that the heavy tails of skewed distributions can disproportionately influence the regression outcome, but is this necessarily a problem? Arguably, these data points represent the more extreme end of the scale, and because they are apparently more sparse, why shouldn't they have more influence on the regression line?
However, if we are attempting to infer about the general population, it is unlikely that biasing the result towards a relatively few number of outliers allows us to support this inference. There may (and likely are) attributes unique to these individuals that differentially affect subjective well-being compared to individuals with less extreme levels of gameplay (what conditions are necessary to support 25+ hours per week, for example?). It is also possible that this disproportionate influence could be masking effects in the general population.
The above also highlights the utility of an appropriate data transformation, like a log transformation, for reducing the bias imposed by the skewness of a distribution. Log transforms both reduced the apparent effect size and squashed the disproportionate influence of extreme data points, by ensuring the distribution was more uniform across the range of the predictor variable.
Possibly the most important take-home here, though, is the importance of emphasizing the size of the effect, rather than the statistical significance of the p-value at some arbitrary threshold like \( \alpha=0.05 \). An \(R^2\) value of 0.01 represents a very small effect size: basically, regardless of the regression coefficient, this means that the variance of gameplay time explains only 1% of the variance of subjective well-being. That this effect is actually likely inflated due to the skewness only exacerbates this issue.
To conclude on a positive note, however: the findings are still intriguing, and highlight some interesting questions, such as: what is special about individuals who play video games for 25+ hours per week? Are these individuals using the game as a surrogate for social interaction? Do they substantially differ in socioeconomic status? Is this effect a result of the current COVID-19 lockdown, i.e., by providing a way to stay engaged and connected despite being largely confined to one's home?
The authors consider a number of additional factors in order to better elucidate the observed patterns, are careful to acknowledge the small effect size, and emphasize that they "cannot claim that game time causally affects well-being" based on the presented evidence. More generally, the use of large data sets such as this, particularly with a larger variety of game types, may help us understand how this activity influences mental health, whether and under what conditions it constitutes a form of addiction, and how public policy might be designed to deal with it.
