The argument is essentially as follows. Suppose that on average $\overline{H}$ individuals live in a civilization before it becomes posthuman; then a posthuman ancestor-simulation will on average contain $\overline{H}$ individuals. Say that on average each posthuman civilizations runs $\overline{N}$ ancestor-simulations. Then each of them will on average run ancestor-simulations containing $\overline{N}\overline{H}$ individuals: $\overline{H}$ individuals in each of $\overline{N}$ ancestor-simulations.
Now suppose that $f_P$ is the fraction of all human civilizations that survive to reach the posthuman stage. If we start with a grand total of $M$ human civilizations, then $f_PM$ of them will survive to the posthuman stage. On average each of those $f_PM$ posthuman civilizations will produce $\overline{N}\overline{H}$ simulated individuals, for a grand total of $f_PM\overline{N}\overline{H}$ simulated individuals. There are also $M\overline{H}$ real individuals: an average of $\overline{H}$ of them for each of $M$ human civilizations. Thus, the total number of individuals, real and simulated, is $f_PM\overline{N}\overline{H}+M\overline{H} = M\overline{H}(f_P\overline{N}+1)$. The fraction of these who are simulated is therefore $f_{sim}=\frac{f_PM\overline{N}\overline{H}}{M\overline{H}(f_P\overline{N}+1)}=\frac{f_P\overline{N}}{f_P\overline{N}+1}\tag{1}$ after cancelling $M\overline{H}$.
Perhaps not all posthuman civilizations run ancestor-simulations; let $f_I$ be the fraction of them that do run a significant number, and let $\overline{N_I}$ be the average number of ancestor-simulations run by the these civilizations. The difference between $\overline{N}$ and $\overline{N_I}$ is similar to the difference between average lifespan and average lifespan of those who survive infancy: if many children die in infancy, the former will be much lower than the latter. Similarly, if $f_I$ is much smaller than $1$, $\overline{N}$ will be much smaller than $\overline{N_I}$.
In fact, $\overline{N}=f_I\overline{N_I}$. To see this, recall that we have $f_PM$ posthuman civilizations, so we have $f_If_PM$ posthuman civilizations that run ancestor-simulations. We assume that the remaining $M-f_If_PM$ posthuman civilizations run a negligible number of ancestor-simulations, so altogether we have $f_If_PM\overline{N_I}$ ancestor-simulations: $\overline{N_I}$ on average for each of $f_If_PM$ posthuman civilizations. Divide this total by $f_PM$, the number of posthuman civilizations, to get $\overline{N}$, the average number of ancestor-simulations per posthuman civilization: $\overline{N} = \frac{f_If_PM\overline{N_I}}{f_PM}=f_I\overline{N_I}\;.$
Substituting this value of $\overline{N}$ into $(1)$, we get $f_{sim}=\frac{f_Pf_I\overline{N_I}}{f_Pf_I\overline{N_I}+1}\;.\tag{2}$
The author then claims that $\overline{N_I}$, the average number of ancestor-simulations run by each posthuman civilization that runs them in any significant numbers, must be extremely large, since such civilizations will have immense computing power. Thus, either $f_Pf_I$ is very small, so as to compensate for the very large factor $\overline{N_I}$, or $f_Pf_I\overline{N_I}$ is large in comparison with $1$, and the fraction in $(2)$ has the form $f_{sim}=\frac{\text{very large number}}{\text{same very large number}+1},$ in which case $f_{sim}$ is very nearly $1$: think of a fraction like $\frac{1000}{1001}$. This is proposition (3) at the end of Section IV.
The alternative, as I said, is that $f_Pf_I$ is very small, so that $f_Pf_I\overline{N_I}$ isn’t a very large number (even though $\overline{N_I}$ is). That can be true only if one or both of the fractions $f_P$ and $f_I$ are very small $-$ practically $0$, in fact, since $\overline{N_I}$ is ‘extremely large’. These are propositions (1) and (2) at the end of Section IV.