This is an exercise in elementary symmetric polynomials. Write $s_1=a+b$ and $s_2=ab$. From the equation $ x^2-x-5=(x-a)(x-b)=x^2-(a+b)x+ab=x^2-s_1x+s_2 $ we can read that $s_1=1$ and $s_2=-5$.
The number that you wanted to know is $ (a^2+4b-1)(b^2+4a-1)=a^2b^2+4(a^3+b^3)+16ab-4(a+b)-(a^2+b^2)+1. $ We need to express the quantities in parenthesis in terms of $s_1$ and $s_2$. This is not too difficult, because from $ s_1^2=a^2+2ab+b^2=(a^2+b^2)+2s_2 $ we get $a^2+b^2=s_1^2-2s_2=11$. Similarly from $ s_1^3=a^3+3a^2b+3ab^2+b^3=(a^3+b^3)+3ab(a+b) $ we get that $a^3+b^3=s_1^3-3s_1s_2=16.$
Putting all this together gives $ (a^2+4b-1)(b^2+4a-1)=25+64-80-4-11+1=-5. $
What makes this tick is that $(a^2+4b-1)(b^2+4a-1)$ is symmetric in the unknowns $a$ and $b$. IOW if you swap the values of $a$ and $b$ nothing will change. Such polynomial functions can always be written in terms of the elementary symmetric polynomials $s_1$ and $s_2$. This result can be generalized to several unknowns. The buzzword "elementary/basic symmetric polynomials" should give you enough material. The power sums such as $a^3+b^3$ are a well-known special case. The buzzword "Newton's identities" helps you there.