For differentiability I used $df = Ah+Bk+h\phi+k\psi$. My professor told me to take $h = p\cos\theta$ and $k = p\sin\theta$ and then use for arbitrary $\theta = \tan^{-1}\dfrac{h}{k}$. that implies $p$ tends to $0$ as $(h,k)$ tends to $(0,0)$.
I didn't get what my professor suggested and how to proceed further and conclude.
I will use the following theorem:
Theorem (differentiability implies derivability). Let $f:A\subseteq\Bbb R^2\to\Bbb R$ a scalar field differentiable at $(x,y)=(x_0,y_0)\in A$. Then exists $f'\big((x_0,y_0);\hat v\big)$ for all $\hat v\in\mathbb R^2$.
Proof. If $f(x,y)$ is differentiable at $(x,y)=(x_0,y_0)$ then there are two numbers $A$ and $B$, and an infinitesimal $\alpha(x,y)\to0$ when $(x,y)\to(x_0,y_0)$ such that $$f(x,y)=f(x_0,y_0)+A(x-x_0)+B(y-y_0)+\sqrt{(x-x_0)^2+(y-y_0)^2}\alpha(x,y).$$ We want the limit $$\lim_{h\to0}\frac{f\big((x_0,y_0)+h\hat v\big)-f(x_0,y_0)}h$$ exists for all $\hat v\in\mathbb R^2$, where $x=x_0+hv_1$ and $y=y_0+hv_2$.
So $$\begin{align}&\dfrac{f(x_0+hv_1,y_0+hv_2)-f(x_0,y_0)}h\\=&\dfrac{A(hv_1)+B(hv_2)+\sqrt{h^2v_1^2+h^2v_2^2}\alpha(x_0+hv_1,y_0+hv_2)}h\\=&Av_1+Bv_2+\frac{|h|}h\cdot1\cdot\underbrace{\alpha(x_0+hv_1,y_0+hv_2)}_{\to0\text{ when }h\to0}\\\Longrightarrow&f'\big((x_0,y_0);\hat v\big)\\=&\displaystyle\lim_{h\to0}\frac{f(x_0+hv_1,y_0+hv_2)-f(x_0,y_0)}h\\=&Av_1+Bv_2+\underbrace{\displaystyle\lim_{h\to0}\frac{|h|}h\alpha(x_0+hv_1,y_0+hv_2)}_0\\\therefore&\exists f'\big((x_0,y_0);\hat v\big)=Av_1+Bv_2\quad\text{for all }A\text{ and }B,\text{ and for all }\hat v\in\Bbb R^2.\qquad\Box\end{align}$$
Remark. The opposite statement is also valid: if the directional derivative is not of the form $Av_1+Bv_2$, for some numbers $A$ and $B$, and $\hat v=(v_1,v_2)\in\Bbb R^2$, then $f(x,y)$ is not differentiable at $(x,y)=(x_0,y_0)$.
In this case,
$$\begin{align} &\lim_{h\to0}\frac{f\big((0,0)+h(v_1,v_2)\big)-f(0,0)}{h}\\ =&\lim_{h\to0}\frac{f(v_1h,v_2h)-0}h\\ =&\lim_{h\to0}\frac{\frac{(v_1h)^3-(v_2h)^3}{(v_2h)^2+(v_2h)^2}}h\\ =&\lim_{h\to0}\frac{v_1^3h^3-v_2^3h^3}{v_1^2h^2+v_2^2h^2}\cdot\frac1h\\ =&\lim_{h\to0}\frac{h^3(v_1^3-v_2^3)}{h^2\underbrace{(v_1^2+v_2^2)}_1}\cdot\frac1h\\ =&\lim_{h\to0}\frac{h^3(v_1^3-v_2^3)}{h^3}\\ =&\lim_{h\to0}(v_1^3-v_2^3)\\ =&v_1^3-v_2^3.\\ \therefore&\exists f'\big((0,0);\hat v\big)=v_1^3-v_2^3.\end{align}$$
As we can see, $f(x,y)$ has directional derivatives for all $\hat v\in\Bbb R^2$, but there are no numbers $A$ and $B$ such that $Av_1+Bv_2$ (the components should not be raised to the third power).
Thus, and by virtue of the opposite statement of the theorem, $$\boxed{f(x,y)\text{ is not differentiable at }(x,y)=(0,0)}.$$
So first let us make a correction: the function of interest is $f(x,y)=\begin{cases} \frac{x^3-y^3}{x^2+y^2} & (x,y) \neq (0,0) \\ 0 & (x,y)=(0,0) \end{cases}$.
With that in mind, if $f$ were differentiable at the origin, then its derivative would be characterized by its vector of partial derivatives at the origin, whose components are $\lim_{x \to 0} \frac{\frac{x^3}{x^2}}{x}=1$ and $\lim_{y \to 0} \frac{\frac{-y^3}{y^2}}{y}=-1$. So the question now reduces to checking whether $f(x,y)=x-y+o(\| (x,y) \|)$ as $(x,y) \to (0,0)$.
For that purpose, polar coordinates is convenient: we have $f(r,\theta)=\frac{r^3\cos(\theta)^3-r^3\sin(\theta)^3}{r^2}=r\cos(\theta)^3-r\sin(\theta)^3$. Is this equal to $r\cos(\theta)-r\sin(\theta)+o(r)$ as $r \to 0$ for all $\theta$?
For $$ f(x,y)= \begin{cases} \dfrac{x^3-y^3}{x^2+y^2}~~~(x,y)\neq(0,0)\\ 0~~~~~~~~~~~~~~~~(x,y)=(0,0) \end{cases} $$ beside your way we have \begin{eqnarray*} && f_x=\frac{x(x^3+3xy^2+2y^3)}{(x^2+y^2)^2} \\ && f_y=\frac{-y(y^3+3yx^2-2x^3)}{(x^2+y^2)^2} \end{eqnarray*} These show that partial derivatives are not continuous in the origin.