This isn't true as stated. For example, consider $Z = X = \{x = y, z = 0\} \cup \{xz = 1\}$ in $\mathbb{A}^3$ and $Y = \mathbb{A}^2$ with the morphism $f:X \to Y$ given by $(x,y,z) \mapsto (y,z)$. Each fiber is just a singleton so it satisfies your hypothesis but $X$ is not irreducible. However, if we impose some mild assumptions, then this is true. We can choose one of two assumptions:
- $f$ is a closed map.
- The irreducible components of $Z$ are the same dimension and $f$ is surjective.
A very similar thing is proved here. The result is as follows:
Suppose $f:X \to Y$ is surjective, proper morphism to an irreducible variety $Y$ such that the fibers are irreducible and of the same dimension. Then $X$ is irreducible.
The argument in that proof can be modified to obtain your result. First, as implied in the comments, we can restrict to our closed subset $Z$ and replace $Y$ with $\overline{f(Z)}$. Then we have reduced the problem to a dominant morphism $f:X \to Y$ of varieties with $Y$ irreducible and satisfying the conditions on the fibers.
Let $Z_i$ be the irreducible components of $X$ and $X_y = f^{-1}(y)$ for any $y \in Y$. By the same argument as in the linked question, but replacing $f(Z_i)$ with $\overline{f(Z_i)}$ since we aren't assuming $f$ is closed, we can conclude that each $Z_i$ is the union of fibers $X_y$ that intersect it.
- If we assume that $f$ is closed, then by the same argument in the linked question, some component $Z_0$ satisfies $f(Z_0) = Y$ so $Z_0$ contains every nonempty fiber and thus $Z_0 = X$ is irreducible.
- If we assume each of the irreducible components are the same dimension and $f$ is surjective, then since $Y$ is irreducible and $f$ is dominant, there exists some component $Z_0$ such that $\overline{f(Z_0)} = Y$. Then letting $d$ be the dimension of the fibers, we know that $d = \dim Z_i - \dim \overline{f(Z_i)}$ as in the link. Now, since each component is the union of fibers that intersect it, we know that $Z_0$ is the only dominant component. Thus, $f(Z_i) \subset Y \setminus f(Z_0)$ for all $i \neq 0$. But $Y \setminus f(Z_0)$ has dimension lower than $f(Z_0)$. In particular, $\dim Y > \dim \overline{f(Z_i)}$ for $i \neq 0$. By the equality above, $\dim Z_0 - \dim Y = \dim Z_i - \dim \overline{f(Z_i)}$. Putting these two things together, we get $\dim Z_0 > \dim Z_i$ for $i \neq 0$, contradicting the fact that all the irreducible components are the same dimension, thus $Z_0 = X$ and $X$ is irreducible.