2
$\begingroup$

Is this always the case and why is this the case? I would really like to find out more about it. To get a better sense of it.

Zhen said in one of the comments:

For the linear case it's a question of linear algebra. For the differentiable case, it's the implicit function theorem. I'm not sure what machinery we invoke for the general case...

Can someone elaborate on why is it a question of linear algebra for linear equations. And each of the others? Or at least, reference an outside source which could help me grasp the matter a little bit better?

  • 0
    No, it is not always the case that if there are $n$ equations there must be $n$ unknowns. For example, here is a set of 2 equations in 1 unknown: $x+1=2$ and $x+2=4$. And here is a set of 2 equations in 3 unknowns: $x+y+z=10$, $x-y+z=20$. (It is a different matter that if you have more equations than unknowns then the system may be overdetermined and have no solutions, and if there are more unknowns than equations the system may be overdetermined and have many solutions, but a set of equations can have any number of unknowns in it.)2011-09-23

1 Answers 1

3

Ok, so a little introduction how to find parameters for the model. Suppose that you have a model $ f(x,k) $ where $k_1,k_2,k_3,...,k_n$ are unknown parameters. You would like to make some measurements $(x_i,f(x_i,k))_{i=1}^m$ to find all these parameters. Just an example: $ f(x,k) = k_1x+k_2. $ Here it is necessary and sufficient to make exactly two measurements at two different points $x_1,x_2$ to find both $k_1$ and $k_2$. With measurements I mean that given a single input $x_1$ you measure the output $f(x_1,k)$.

How does this procedure work? Suppose, you made inputs $x_1 = 0,x_2 = 1$ and obtain $f(0,k) = 0$ and $f(1,k)=1$. Then you have a system $ \begin{cases} k_1\cdot0+k_2 = 0, \\ k_1\cdot1+k_2 = 1 \end{cases} $ and as a result you have $k_1 = 1,k_2 = 0$. So, you solved the system of linear equations. This is the main idea: to find paramers, you make measurements and solve the system of equations.

Equations on paramteres only depend on how $f$ depends on $k$, and does not depend on how $f$ depends on $x$. If $f$ is linear in $k$, then you have to have exactly $n$ measurements unless you unluckily make some measurements which give you no new information, that will cause the degeneracy of system of linear equations.

In the general case, the number of measurements can vary. E.g. you may obtain an equation $ k_1^2+k_2^2 = 0 $ which has a unique solution and does not ask for more measurements.

In other cases even $n$ measurements are not enough, this is what Implicit Function Theorem about. But usually (so, this is an informal statement) you need $n$ equations on $n$ variables to find all of them.

Finally, it's worth to mention that in real life your model is never perfect as well as the date form the measurements. So, given $2$ parameters you may obtain $100$ measurements which never can be satisfied by any pair of parameters. So to obtain the values of parameters, you give an estimation based, say on Least Squares method.

  • 0
    @TheChaz: Yes, I've just tried to give an observation how does this problem arise.2011-09-23