If a category $\mathcal{C}$ has all small products (that is, any family indexed by a "small" set has a product), and has equalizers, then every small limit in $\mathcal{C}$ exists, and in particular every inversely directed system of objects of $\mathcal{C}$ has an inverse limit; the construction is essentially the one from $\mathcal{S}et$ (and other categories that have a forgetful functor with an adjoint).
Explicitly, here is Proposition 7.6.6 from George Bergman's An Invitation to General Algebra and Universal Constructions:
Proposition. Let $\mathbf{C}$ be a category and $\mathbf{D}$ a small category, and let $\alpha$ be an infinite cardinal such that $\mathbf{D}$ has fewer than $\alpha$ objects and fewer than $\alpha$ morphisms. Then, if $\mathbf{C}$ has products of all families of fewer than $\alpha$ objects, and has equalizers, then every functor $\mathbf{D}\to\mathbf{C}$ has a limit.
(You can think of inverse limits as the categorical limit of a functor from the category of an inversely directed set, viewed as a category, to $\mathbf{C}$.)
The construction mimics the one from $\mathcal{S}et$: let $F\colon\mathbf{D}\to\mathbf{C}$ be a functor.
P = \prod_{X\in\mathrm{Ob}(\mathbf{D})} F(X)\qquad\text{and}\qquad P' = \prod_{\stackrel{X,Y\in\mathrm{Ob}(\mathbf{D})}{f\in\mathbf{D}(X,Y)}} F(Y). Let $\pi_X$ be the projection morphisms from $P$ to $F(X)$, and $p_{X,Y,f}$ the projection morphisms from P' to $F(Y)$.
Define a,b\colon P\to P' as follows: since they are maps into a product, we can specify the value of the compositions $p_{X,Y,f}a$ and $p_{X,Y,f}b$, which will uniquely determine $a$ and $b$. First, $a$ is the map that satisfies $p_{X,Y,f}a = \pi_Y$ (that is, project $P$ onto $F(Y)$, then map $F(Y)$ to $F(Y)$ via the identity; do this for each triple $(X,Y,f)$; this gives a map from $P$ to $P'$). Second, $b$ is the map that satisfies $p_{X,Y,f}b = F(f)\pi_X$ (that is, project $P$ onto $F(X)$, then map from $F(X)$ to $F(Y)$ via $F(f)$).
Let $k\colon L\to P$ be the equalizer of $a$ and $b$. The universal property of $L$ and $k$ is equivalent to the property that the maps $\pi_Xk\colon L\to F(x)$ give commuting diagrams with the morphisms $F(f)$ and are universal. The object $L$ and the morphisms $\pi_Xk$ satisfy the universal property characterizing $\lim\limits_{\leftarrow} F$, so $L$ is a limit for $F$. $\Box$
(There is a dual result for colimits, when the category $\mathbf{C}$ has all sufficiently large coproducts and has coequalizers).
In fact, this proposition captures exactly how much structure you need to be able to carry out the construction from $\mathcal{S}et$/$\mathcal{G}roup$/etc in an arbitrary category: you need to be able to construct the product (which for arbitrary categories need not exist), and you must be able to specify the "compatibility conditions" in some way, which turns out to be equivalent to the existence of a suitable equalizer. If there is no equalizer, you may not be able to "specify" the appropriate "subobject" of the product.
So... you may be able to carry out the relevant construction even in categories that are not concrete (the objects need not be sets, and the arrows need not be set-theoretic functions between the objects), if you have sufficient structure. Or you may have a concrete category (such as "finite groups"), but where there isn't sufficient structure to carry out the construction.
The reason that you don't need to check the "relevant algebraic structure" for many categories like $\mathrm{G}roup$, $\mathrm{R}ing$, etc., is that these categories come equipped with an underlying set functor $\mathcal{C}\to\mathcal{S}et$ that has a left adjoint (the "free object" functor). Left adjoints respect colimits and right adjoints respect limits, which means that the underlying set of an inverse limit has to be the inverse limit of the underlying sets; so the construction is essentially forced as to what set it "should" be.
In general, if we let $\Delta\colon\mathbf{C}\to\mathbf{C}^{\mathbf{D}}$ be the diagonal functor that takes an object $X$ to the functor $\Delta(X)\colon\mathbf{D}\to\mathbf{C}$ that maps every object to $X$ and every arrow to $\mathrm{id}_X$; and takes each arrow $f\colon X\to Y$ in $\mathbf{C}$ to the natural transformation $\Delta(f)\colon\Delta(X)\to\Delta(Y)$ with value $f$ at all objects of $\mathbf{D}$, then a limit of a functor $F\colon\mathbf{D}\to\mathbf{C}$ is equivalent to an object $L$ that represents the contravariant functor $\mathbf{C}^{\mathbf{D}}(\Delta(-),F)\colon\mathbf{C}^{\rm op}\to\mathcal{S}et$. When every functor $\mathbf{D}\to \mathbf{C}$ has a limit, the $\lim\limits_{\leftarrow}$ construction is a right adjoint of the functor $\Delta\colon \mathbf{C}\to\mathbf{C}^{\mathbf{D}}$.