A Lie group is also a differentiable manifold; in particular we can define its tangent space at the identity. Intuitively, the tangent space is the set of directions $v$ such that if you start at the identity in $G$ and move infinitesimally in direction $v$, you stay in $G$.
Let's say that $G$ is defined by the vanishing of some differentiable function $f$: so $G = \{ X: f(X)=0\}$. The tangent space consists of matrices $M$ such that $f(I + \epsilon M) =0$ to first order in $\epsilon$: using the Taylor expansion and neglecting higher terms, what we want is that $df_I(M)=0$ where $df_I$ is the total derivative at the identity. Elements of the kernel of the total derivative are exactly those vectors which can be realised as $\gamma'(0)$ for some differentiable $\gamma: (-1,1)\to G$: the usual definition of tangent space is the equivalence classes of such functions $\gamma$ under the relation $\gamma_1\sim \gamma_2$ iff $\gamma_1'(0)=\gamma_2'(0)$.
The Lie algebra associated to $G$ has as its underlying vector space the tangent space at the identity, so it really is obtained by differentiating and setting the total derivative equal to zero.
Looking at your $O(n)$ example, the "$f$" is $f(X)=XX^t-I$. To find the total derivative at the identity, we ought to have $f(I+\epsilon M) = I + \epsilon df_I(M) +$ higher order terms. To order one in epsilon we have $f(I+\epsilon M)= \epsilon (M + M^t)$, so the total derivative at the identity is $M+M^t$. The vanishing of this is exactly the condition you gave for being in the Lie algebra of $O(n)$.