I will be working off the formula
$$(x || y) = x * b^{\lfloor\log_b(y)\rfloor + 1} + y$$
You can see that this should be the case because with $b = 10$, $y \in [10,99]$ should multiply $x$ by $100$.
In the following, I will assume $b = 10$ and write $\log(x)$ for $\log_{10}(x)$. (The proof generalizes immediately to any $b$). The proof does not require case analysis, only applying a couple elementary properties of the floor function.
The more interesting point is perhaps about whether concatenation is 'unmathematical.' I'm not sure what you mean by 'unmathematical'. In general we want to be able to define concatenation of words for arbitrary symbolic systems (alphabets). I suppose you (or this youtube poster) are taking issue with the dependency on indexing (and knowing the length of the second argument) in the usual definition of concatenation.
We have by definition of the operator $||$
$\big(x || y \big) = 10x * 10^{\lfloor\log(y)\rfloor} + y$
$\big(y || z\big) = 10y * 10^{\lfloor\log(z)\rfloor} + z$
So $\big(x || y\big) || z =$ $$10*\big(x || y\big)*10^{\lfloor\log(z)\rfloor} + z =$$
$$(100x * 10^{\lfloor\log(y)\rfloor} + 10y)*10^{\lfloor\log(z)\rfloor} + z = $$
$$100x*10^{\lfloor\log(y)\rfloor + \lfloor\log(z)\rfloor} + 10y*10^{\lfloor\log(z)\rfloor} + z$$
Meanwhile $x||\big(y||z\big) = $
$$x || \big(10y * 10^{\lfloor\log(z)\rfloor} + z\big) =$$
$$10x * 10^{\lfloor \log (10y * 10^{\lfloor\log(z)\rfloor}) \rfloor} + 10y * 10^{\lfloor\log(z)\rfloor} + z = $$
$$10x * 10^{\lfloor 1 + \log(y) + \lfloor\log(z)\rfloor \rfloor} + 10y * 10^{\lfloor\log(z)\rfloor} + z$$
Consider that $\lfloor 1 + a + \lfloor b \rfloor \rfloor = 1 + \lfloor b \rfloor + \lfloor a \rfloor$, since $1 + \lfloor b \rfloor$ is an integer.
Applying this to the last line in the expansion of $x||\big(y||z\big)$ shows $x||\big(y||z\big) = $
$$10x * 10^{1 + \lfloor \log(y) \rfloor + \lfloor\log(z)\rfloor} + 10y * 10^{\lfloor\log(z)\rfloor} + z = $$
$$100x * 10^{\lfloor \log(y) \rfloor + \lfloor\log(z)\rfloor} + 10y * 10^{\lfloor\log(z)\rfloor} + z$$.
This shows that $x||\big(y||z\big) = \big(x||y\big)||z$ as desired, and we have proved that your formal "concatenation" operator satisfies associativity!
Now, have we actually mathematically captured concatenation?
The only problem to get around is zeros. $x || 0$ is not well-defined for us, and $0 || y$ returns $y$, so it isn't really concatenation in a string sense. Moreover if a number contains $0$ in it's middle, then treating it as a word we see that is the concatenation of $x||y$ where $y$ has a leading zero, and the formula breaks. The problems are that our formula deals with numbers, but concatenation deals with strings. In the former, leading zeros don't matter, but in the latter, they do.
This isn't ultimately a problem though. We can use this formula to define concatenation for any finite symbolic system.
Let $||$ be defined as above on any base $b$ number system with the explicit definition $x || 0 = x$ for all $x$.
Given an alphabet $A$ consisting of $n$ characters, we define concatenation on two words $u,v$ of $A$ as follows:
$$u || v = \phi^{-1}\big(\phi(u) || \phi(v)\big)$$
where $\phi$ is any bijection $\phi: A \rightarrow \{1,\ldots,n\}$ extended to act on words of $A$ elementwise, with its image on the empty word explicitly defined to be $0$. Note that $\phi$ is then a bijection between words of $A$ and numbers base $n+1$ which either are zero or have no zeros as digits.
So the operator of concatenation on words of any alphabet can be defined quite rigorously without making reference to the lengths or elements of the words. We only require for this that the alphabet be finite, because our formula for concatenation in a base $b$ number system breaks down for $b$ infinite.