A number of introductory probability resources allow the codomain $A$ of a random variable $X:\Omega\to A$ to be an arbitrary set. They then proceed to define expectation by
$$\mathbb E[X] =\sum_{a\in A}\mathbb P(X = a)\cdot a.$$
This certainly seems natural if $A = \{1,2\ldots,n\}$ or similar, and can of course be extended to the infinite and continuous cases.
However, what if $A$ carries a structure that is incompatible with standard arithmetic on $\mathbb R$? Here's an example: suppose $A = \{1,2,3\}$, where each element represents a primary color:
$$ 1 = R, \quad 2 = Y, \quad 3 = B $$
Further, let's say that $X$ is distributed uniformly across its domain. Then by definition $\mathbb E[X] = \frac 13 + \frac 23 + \frac 33 = 2$. But this is surely wrong: if I pick a random color from these three, it will not on average be yellow! The primary colors do not lie on a line, they lie on a circle!
As an alternative, we can take $A = \{R,Y,B\}$ directly and then $$\mathbb E[X] = \frac 13 R+\frac 13 Y + \frac 13 B.$$ This feels slightly better: it is reasonable to suppose $A\simeq \mathbb Z_3$ and then $E[X]$ is an element in $\mathbb R[\mathbb Z_3]$. But this still bothers me, because this ring representation a) still has all the information about the distribution of $X$ and b) doesn't provide a good way to compare expectations between R.V.s with the same codomain.
I can also imagine using a linear algebraic structure, but I'm not sure it would be much better.
So my question is: what is the implicit algebraic structure in R.V.s' codomains? Must $A$ always embed into $\mathbb R$? How do probablists talk about situations in which this embedding isn't natural?