Continuity is a property that a function between topological spaces may or may not satisfy, and it would make no sense without fixed topologies on the spaces under consideration. If $X$ and $Y$ are topological spaces (implicitly meaning that there is a fixed topology on each under consideration), then a function $f:X\to Y$ is continuous if (by definition) for each open set $U\subseteq Y$, the set $f^{-1}(U)=\{x\in X: f(x)\in U\}$ is open in $X$. Continuity can also be defined in terms of nets or filters, and in the first countable case in terms of sequences. A good textbook on general topology, such as those by Munkres or Kelley, is probably the best way to learn more about topology and continuous functions in general.
In $B(H)$, the space of bounded linear operators on a Hilbert space $H$, you are considering 3 topologies: norm (uniform), SOT (strong operator topology), and WOT (weak operator topology). The norm topology is the topology of uniform convergence on bounded sets. The SOT is the topology of pointwise convergence in the norm of $H$. The WOT is the topology of pointwise weak convergence in $H$. A net $(x_i)_i$ in $H$ converges weakly to $x\in H$ if and only if for each $y\in H$, $(\langle x_i,y\rangle)_i$ converges to $\langle x,y\rangle$. A net $(T_i)_i$ in $B(H)$ converges in the WOT to $T\in B(H)$ if and only if for each $x\in H$, the net $(T_ix)_i$ converges weakly to $Tx$.
I recommend Chapters 12 to 14 of Halmos's A Hilbert space problem book for building your intuition with these. There are also a number of good texts on operator theory or operator algebras with a treatment of these topologies, including Kadison and Ringrose's Fundamentals of the theory of operator algebras and J.B. Conway's A course in operator theory.
The norm topology is a little easier to work with in some ways. With this topology $B(H)$ is a metric space, so in particular it is first countable. With either the SOT or WOT, $B(H)$ is a locally convex topological vector space, defined in terms of families of seminorms. Therefore you may also benefit from studying locally convex spaces in general, for example using a good textbook on functional analysis like Rudin's Functional analysis or J.B. Conway's A course in functional analysis.