Is it possible to have a continuous function $f$ s.t. $$f(x,y,1)= x+y \\f(x,y,2)=xy\\f(x,y,3)=x^y\ \\ f(x,y,-1)=x-y \\ f(x,y,-2)=\frac{x}{y}\\ f(x,y,-3)=\sqrt[y]x$$
etc? so we can have things like $f(x,y,1.5), f(x,y,5)$ defined?
Are there such generalisation already studies in some branch of mathematics?
You can definitely have such a function. Just define a continuous $g(t)$ with the following properties:
$g(t)=1$ for $t=0$
$g(t)=0$ for $t\notin (1,1)$ (the open interval from $-1$ to $1$, not including $-1,1$).
Then, if you have an arbitrary set of $f_i(x,y)$, $i\in\mathbf{Z}$ (in your case, $f_1(x,y)=x+y$, $~~f_2(x,y)=xy$, and so on) and you want a continuous $f(x,y,t)$ that is equal to $f_i(x,y)$ for $t=i$, you can easily obtain it as:
$f(x,y,t)=\sum_i g(t-i) f_i(x,y)$
Note that there are many many ways to choose a continuous $g(t)$ with the properties above ($g(t)$ should be $1$ in $0$, and $0$ at all points "sufficiently far" from $0$). The additional properties you choose for $g(t)$ influence those of the resulting $f(x,y,t)$: for example, if you choose $g(t)$ as an even function such that $g(t)+g(1-t)=1$ (note that this still leaves you a lot of flexibility), your $f(x,y,t)$ for $t$ between two integers $i,i+1$ will be a weighted average of $f_{i}(x,y)$ and $f_{i+1}(x,y)$.
This is essentially what you do in computer graphics when "morphing smoothly" one image (which you can view as a function mapping pixel coordinates into colour) into another. More in general, it's a very common way of doing things in signal processing.
There are something called hyperoperators which can be found at https://en.wikipedia.org/wiki/Hyperoperation . If you look at the page for https://en.wikipedia.org/wiki/Tetration#Inverse_operations , there are two main "inverses" for tetration which they call the super-root and the super-logarithm. So there is definitely some things out there to take care of the case of $f(a,b,n)$ where $n \in \mathbb{Z}$. However, I'm not aware of anything that generalizes to the case where $n \in \mathbb{Q}$.
However, when it comes to hyperoperations and their inverses, there are still a few issues to consider. One is the issue of associativity; notice that defining exponentiation as repeated multiplication is fine since $(a \cdot a) \cdot a = a \cdot (a \cdot a)$. However, when we want to create an operator based on repeated exponentiation, then we have to make a choice since it is not the case that $(a^a)^a = a^{(a^a)}.$ The definition on Wikipedia defines the hyperoperations using right-associativity.
Another issue that exists is commutativity. Since $a+b = b+a$ and $ab = ba$, then there isn't much ambiguity that the inverse operation to both is subtraction and division respectively. But consider $a^b$. If we want to get $a$ by itself, then we take a root: $\sqrt[b]{a^b} = a$ but if we want to get $b$ by itself, we need to take a logarithm: $\log_a(a^b) = b$. So you would have to make a decision as to what the inverse operation would be for anything "greater than" exponentiation.