Ok, let's start from the connected sum of knots in $S^3. I have seen some confusion in the comments: the connected sum of two knots is well defined, doesn't metter if the arc of the connected sum is knotted. The reason why this is true is the content of Figure 1 and is proved in many textbooks (e.g. Cromwell's book "Knots and links").
About the connected sum in arbitrary dimension: the connected sum $A \# B$ does not depend from the chosen arc if at least one of the two submanifolds (say $B$) is $\textit{local}$ (meaning that is contained in a ball disjoint from $A$). The proof of this fact is pictorially described in Figure 2 and can be made rigorous by integrating a suitable vector field supported in a tubular neighbourhood of the arc used to perform the connected sum (the red one in the figure).
Notice that if $A$ and $B$ are both non-local then the result of the connected sum $A \# B$ can a priori depend on the chosen arc. Here a low-dimensional example.
Let $K, J \subset S^1 \times S^2$ be the two knots in the Kirby diagram of Figure 3 (a). Denote by $K \#_n J$ be the connected sum of $K$ and $J$ performed along the arc described in Figure 3 (b) (in the box there are $2n$ positive crossings). I claim that $K \#_0J$ and $K\#_1 J$ are not isotopic. This is because $K\#_0J \subset S^3$ is local (obvious, just picture it) while $K \#_1 J$ is not. To see that $K'=K \#_1J$ is not local we argue by contradiction: surgery on a local knot always produce a connected sum while for a suitable surgery along $K'\subset S^1 \times S^2$ we get something that is not. To prove this last statement you need some Kirby calculus: $K'$ and the zero framed unknot in the picture are linked as a Whitehead link and if we put surgery coefficient $-1$ on $K'$ we get (after blowing down $K'$) a diagram for the $0$-surgery on the trefoil knot (an irreducible Seifert manifold).
$\textbf{EDIT:}$ of course, the result of the connected sum of two knots $K_1, K_2 \subset S^3$ dramatically depends on the chosen arc if $K_1$ and $K_2$ are somehow linked. For example if we take $K_1 \cup K_2$ as the Hopf link and we perform connected sum along the blue arc of Figure 4 we get the unknot while if we do it along the red one we get the (untwisted) Whitehead double of the trefoil knot.
The naive definition of connected sum we are discussing here is not the one usually adopted. Here a non problematic one (compare with the definition of Murasugi sum)
$\textbf{Definition.}$ Suppose that $(M,A)$ and $(N,B)$ with $M$ and $N$ smooth $n$-manifolds and $A$ and $B$ embedded $k$-submanifolds are given. Choose balls $U \subset M $ and $V \subset N$ such that
$$ (M\cap U, A \cap U)=(N \cap V, B \cap V)= (\mathbb{R}^n, \mathbb{R}^k) \ .$$
Define $ A \# B= (A - U) \cup_\partial (B-V) \subset M \# N$.
Btw given two eventually linked submanifolds $A, B \subset M$ one can pick an arc $\gamma$ connecting $A$ and $B$ in $M$ and use it to perform embedded surgery. Some people call this operation $\textit{piping}$. As result of the piping operation we get an embedding $A \# B \hookrightarrow M$ whose isotopy type a priori depends on $\gamma$ and (as already many of you pointed out in the comments) there are obvious sufficient (homotopic) conditions that one can impose in order to guarantee uniqueness.
