If $C$ is the right angle, $c=5$ cm.
We know from the Law of Sines, $\frac c {\sin C}=2R\implies R=\frac 5 2$
$A+B=\pi-C=\pi-\frac \pi 2=\frac \pi 2 $
Using this, $r=\frac{a+b-c}2=\frac{2R(\sin A+\sin B-\sin C)}2=\frac 52(\sin A+\cos A-1)=\frac 52(\sqrt2 \cos(A-\frac{\pi}4)-1)$
$r$ will be maximum if $\cos(A-\frac{\pi}4)=1,A=2m\pi+\frac{\pi}4$ where $m$ is any integer.
But, $0 So, $A=\frac{\pi}4$
So, $B=\frac \pi 2-A=\frac{\pi}4=A$
Form this, the in-radius $r=4R\sin \frac A 2 \sin \frac B 2 \sin \frac C 2$
$r=4\frac 5 2\sin \frac A 2 \sin \frac B 2 \frac 1{\sqrt 2}$ $=\frac5{\sqrt2}2\sin \frac A 2 \sin \frac B 2$ $=\frac5{\sqrt2}\left(\cos(\frac{A-B}2)-\cos(\frac{A+B}2)\right)$
$=\frac5{\sqrt2}\left(\cos(\frac{A-B}2)-\frac 1{\sqrt2}\right)$
This will be maximum if $\cos(\frac{A-B}2)=1$ or
if $\frac{A-B}2=2n\pi$ where $n$ is any integer.
$\implies A=B+4n\pi,$
But $0 So, the difference of $A,B$ can not be $>\frac \pi 2$
So, $n=0,A=B\implies a=2R\sin A=2R\sin B=b$
$r_{max}=\frac{5(\sqrt 2-1)}2$