First of all: $\rho$ is usually used for a Galois representation; in the context, I'm pretty sure it's supposed to be the Galois representation given by the Tate module of an elliptic curve over $\mathbf{Q}$, which is a continuous homomorphism $\operatorname{Gal}(\overline{\mathbf{Q}} / \mathbf{Q}) \to \operatorname{GL}_2(\mathbf{Z}_p)$ for some prime $p$. $\operatorname{Sym}^2 \rho$ is the symmetric square Galois representation into $\operatorname{GL}_3(\mathbf{Z}_p)$.
In the $H^1$ bit: $\mathbf{Q}_\Sigma$ is the maximal extension of $\mathbf{Q}$ unramified outside a finite set $\Sigma$ of primes, and $H^1(\mathbf{Q}_\Sigma / \mathbf{Q}, -)$ is shorthand for the group cohomology $H^1(\operatorname{Gal}(\mathbf{Q}_\Sigma / \mathbf{Q}), -)$ (more strictly, continuous gropu cohomology, which respects the Krull topology on the Galois group). We can plug in for the "$-$" any Galois representation factoring through the canonical map $\operatorname{Gal}(\overline{\mathbf{Q}} / \mathbf{Q}) \to \operatorname{Gal}(\mathbf{Q}_\Sigma / \mathbf{Q}),$ i.e. any Galois representation unramified outside $\Sigma$; the Galois representation attached to an elliptic curve (or its symmetric square) satisfies this as long as $\Sigma$ contains $p$ and all primes dividing the discriminant of the elliptic curve.
As for $\mathbf{Q}_p / \mathbf{Z}_p$: this is the "Prufer group", an infinite torsion group isomorphic to the direct limit of cyclic groups of order $p^n$ over all $n$. It's useful here because taking homs into $\mathbf{Q}_p / \mathbf{Z}_p$ gives a well-behaved duality theory for topological modules (one form of Pontryagin duality).
The only thing I haven't explained yet is the subscript $f$ in $H^1_f(...)$. This is perhaps the most delicate thing here: it's the "finite part" of the Galois cohomology group $H^1(...)$, a certain canonical submodule defined in a famous 1990 paper of Bloch and Kato. $H^1_f(\mathbf{Q}_\Sigma / \mathbf{Q}, \rho)$ is closely related to the Selmer group of the elliptic curve, so the $H^1_f$ functor is a sort of "Selmer group of an arbitrary Galois representation". The size of the Bloch--Kato Selmer group of $\operatorname{Sym}^2 \rho$ is important here because it determines how the deformation theory of $\rho$ behaves.
A good place to learn more about these things would be the book by Cornell, Silverman and Stevens, which user33240 has already linked to.