It appears you are using the wrong angular units: $2\;\tan^{-1}\left(\frac{5}{100}\right)=5.7248$ degrees $=0.099917$ radians.
The formula you cite above is valid for a flat object perpendicular to the line of sight. If your object is a sphere, the angular diameter is given by $2\;\sin^{-1}\left(\frac{5}{100}\right)=5.7320$ degrees $=0.100042$ radians.
Usually, the angular size is referred to as the apparent size. Perhaps you want to find the actual size of the object which has the same apparent size but lies at a different distance. In that case, as joriki says, just multiply the actual distance by $\frac{10}{100}$ to get the actual diameter. This is a result of the "similar triangles" rule used in geometry proofs.
Update: In a comment to joriki's answer, the questioner clarified that what they want is to know how the apparent size varies with distance. 
The formulae for the angular size comes the diagram above:
for the flat object: $\displaystyle\tan\left(\frac{\alpha}{2}\right)=\frac{D/2}{r}$; for the spherical object: $\displaystyle\sin\left(\frac{\alpha}{2}\right)=\frac{D/2}{r}$