If the co-ordinates of the hexagon are $(x_i,y_i)$ then the area of the larger hexagon is $\frac{| x_1 y_2 - x_2 y_1 + x_2 y_3 - x_3 y_2 + x_3 y_4 - x_4 y_3 + x_4 y_5 - x_5 y_4 + x_5 y_6 - x_6 y_5 + x_6 y_1 - x_1 y_6 |}{2}.$
The area of the smaller hexagon can be calculated similarly but replacing $x_i$ by $\frac{x_{i-1}+x_{i+1}}{2}$ and $y_i$ by $\frac{y_{i-1}+y_{i+1}}{2}$ making the obvious adjustments for indices above 6 or below 1.
Dividing the area of the smaller hexagon by that of the larger hexagon and tidying up will give the answer you want of $\frac14$.
This presupposes that the smaller hexagon is simple enough for this to be meaningful. There is probably a simpler counterexample, but starting with the points $(20,20),(20,72),(164,76),(126,24)(88,8),(42,6)$ the larger hexagon is convex, while the smaller hexagon is self-intersecting.