$C$ is open
First, make $x_4$ the subject of the inequality in the definition of $C$ (so as to make use of preimage).
\begin{array}{rrl}
& x_1^2 + x_2^2 + x_3^2 <& \ln(1+x_4^2) \\
\iff& \exp(x_1^2 + x_2^2 + x_3^2) <& 1+x_4^2 \\
\iff& \exp(x_1^2 + x_2^2 + x_3^2) - 1 <& x_4^2 \\
\iff& x_4 >& \sqrt{\exp(x_1^2 + x_2^2 + x_3^2) - 1} \text{ or } \\
&x_4 <& -\sqrt{\exp(x_1^2 + x_2^2 + x_3^2) - 1} \tag1\label1
\end{array}
Let $f_1:\Bbb R^3 \to \Bbb R$ be defined as $(x_1,x_2,x_3)\mapsto-\sqrt{\exp(x_1^2 + x_2^2 + x_3^2) - 1}$ and
$f_2:\Bbb R^3 \to \Bbb R$ be defined as $(x_1,x_2,x_3)\mapsto\sqrt{\exp(x_1^2 + x_2^2 + x_3^2) - 1}$.
They are well-defined since $x_1^2 + x_2^2 + x_3^2 \ge 0$, so $\exp(x_1^2 + x_2^2 + x_3^2) \ge 1$. This allows us to take square root.
Next, observe that $f_1$ and $f_2$ are continuous on $\Bbb R^3$ since they're composition of elementary functions.
Rewrite the definition of $C$ using these two continuous functions to see the openness.
\begin{align}
C =& \{( x_1, x_2, x_3, x_4) \in \mathbb{R}^4 \mid x_1^2 + x_2^2 + x_3^2 < \ln(1+x_4^2)\} \\
\stackrel{\eqref{1}}=& \{( x_1, x_2, x_3, x_4) \in \mathbb{R}^4 \mid x_4 > \sqrt{\exp(x_1^2 + x_2^2 + x_3^2) - 1} \text{ or } \\ & x_4 < -\sqrt{\exp(x_1^2 + x_2^2 + x_3^2) - 1}\} \\
=& \{( x_1, x_2, x_3, x_4) \in \mathbb{R}^4 \mid x_4 > f_2(x_1, x_2, x_3) \text{ or } x_4 < f_1(x_1, x_2, x_3)\} \\
=& \{( x_1, x_2, x_3, x_4) \in \mathbb{R}^4 \mid x_4 < f_1(x_1, x_2, x_3) \} \cup \{( x_1, x_2, x_3, x_4) \in \mathbb{R}^4 \mid x_4 > f_2(x_1, x_2, x_3) \} \\
=& \bigcup_{x_4 \in \Bbb R} \{( x_1, x_2, x_3) \in \mathbb{R}^3 \mid x_4 < f_1(x_1, x_2, x_3) \} \cup \{( x_1, x_2, x_3) \in \mathbb{R}^3 \mid x_4 > f_2(x_1, x_2, x_3) \} \\
=& \bigcup_{x_4 \in \Bbb R} f_1^{-1}((-\infty,x_4)) \cup f_2^{-1}((x_4,+\infty))
\end{align}
Since $f_1$ and $f_2$ are continuous, for each $x_4 \in \Bbb R$, the first and the second sets in the last equality are open. Hence $C$ is open.
$C$ is not closed
Expand @DougM's idea. Consider $c_n = (0,0,0,\frac1n) \in \Bbb R^4, n \in \Bbb N$. The points $c_n$ are in $C$ since $0^2+0^2+0^2 = 0 < \ln\left(1+\frac{1}{n^2}\right)$. $(c_n)$ converges to $(0,0,0,0)$ so $(0,0,0,0) \in \overline{C}$, but $(0,0,0,0) \notin C$ since $\ln(1+0^2) = 0$, so the strictly inequality in the definition of $C$ doesn't hold.