Average value paradox - What is this called?

Simple example to gain intuition:

• Let $A$ be an indicator whether an individual purchased an item in category A.
• Let $B$ be an indicator whether an individual purchased an item in category B.
• Let $X = A + B$ be the number of items purchased.

\begin{array}{ccc} \text{Person} & A & B \\ i & 1 & 0 \\ ii & 0 & 1 \\ iii & 1 & 1 \end{array}

The set of individuals where $A$ is true overlaps the set of individuals where $B$ is true. They are NOT disjoint sets.

Then $\operatorname{E}[X] \approx 1.33$ while $\operatorname{E}[X \mid A] = 1.5$ and $\operatorname{E}[X \mid B] = 1.5$

The statement that would be true is:

$$ P(A)\operatorname{E}[X\mid A] + P(B)\operatorname{E}[X\mid B] - P(AB)\operatorname{E}[X\mid AB] = \operatorname{E}[X]$$

$$ \frac{2}{3}1.5 + \frac{2}{3}1.5 - \frac{1}{3}2 = 1.3333$$

You can't simply compute $P(A)\operatorname{E}[X\mid A] + P(B)\operatorname{E}[X\mid B] $ because sets $A$ and $B$ overlap, the expression double counts the person who purchases both item $A$ and $B$!

Name for illusion/paradox?

I'd argue it's related to the majority illusion paradox in social networks.

You may have a single dude who networks/friends everyone. That person may be one out of a million overall, but he'll be one of each persons's $k$ friends.

Similarly, you have 1 out of 3 here purchasing both categories A and B. But within either category A or B, 1 out of the 2 purchasers is the super purchaser.

Extreme case:

Let's create $n$ sets of lotto tickets. Every set $S_i$ includes two tickets: a losing ticket $i$ and the jackpot winning ticket.

The average winnings in every set $S_i$ is then $\frac{J}{2}$ where $J$ is the jackpot. The average of each category is WAY above the average winnings per ticket overall $\frac{J}{n+1}$.

It's the same conceptual dynamic as the sales case. Every set $S_i$ includes the jackpot ticket in the same way that every category A, B, or C includes the heavy purchasers.

My bottom line point would be that intuition based upon disjoint sets, a full partition of the sample space does not carry over to a series of overlapping sets. If you condition on overlapping categories, every category can be above average.

If you partition the sample space and condition on disjoint sets, then categories have to average out to the overall mean, but that's not true for overlapping sets.

I would call this the family size paradox or something similar

Suppose, for a simple example, everybody had one partner and a Poisson-distributed number of children with parameter $2$:

• The average number of children per person would be $2$
• The average number of children per person with children would be $\frac{2}{1-e^{-2}} \approx 2.313$
• The average sibling group size for each individual (counting their brothers and sisters and themselves) would be $3$

Real demographic and survey numbers produce different numbers but similar patterns

The apparent paradox is that the average size of individuals' sibling groups is larger than the average number of children per family; with stable population dynamics, people tend to have fewer children on average than their parents did

The explanation is whether the average is being taken over parents and families or over siblings: there are different weightings being applied to large families. In your example there is a difference between weighting by individuals or by purchases; your conditional averages are pushed up by fact you condition on a particular purchase being made.

The other answers are overthinking what's going on. Suppose there is one product and two customers. One bought the product (once) and one didn't. The average number of products bought is 0.5, but if you look only at the customer who bought the product, the average rises to 1.

This doesn't seem like a paradox or counterintuitive to me; conditioning on buying a product will generally raise the average number of products bought.

Is this not merely the "average of averages" confusion (e.g. previous stackexchange question) in disguise? Your temptation appears to be that the subsample averages should end up averaging to the population average, but this will rarely happen.

In the classical "average of averages", someone finds the average of N mutually exclusive subsets, and then is flabbergasted that these values do not average to the population average. The only way this average of averages works out is if your non-overlapping subsets have the same size. Otherwise, you need to take a weighted average.

Your problem is made more complex than this traditional average of averages confusion by having overlapping subsets, but it appears to me to just be this classic mistake with a twist. With overlapping subsets, it is even harder to end up with subsample averages that average to the population average.

In your example, since users who appear in multiple subsamples (and therefore have bought many things) will increase these averages. Basically you're counting each big-spender multiple times, while the frugal people that only buy one item are only encountered once, so you're biased to larger values. This is why your particular subsets have above average values, but I think this is still just the "average of averages" problem.

You can also construct all kinds of other subsets from your data where the subsample averages take on different values. For example, let's take subsets somewhat similar to your subsets. If you take the subset of people who did not buy A, you get 7/5=1.4 items on average. With the subset that did not buy B, you also get 1.4 items on average. Those who did not buy C, bought 1.5 items on average. These are all below the population average of 1.6 items/customer. Given the right dataset and the right collection of subsets, you could end up with overlapping subsets whose averages average to the population average; however, this would be uncommon in normal applications.

Is it just me, or does the word average now seem weird after so many repetitions... Hope my answer was helpful, and sorry if I ruined the word average for you!