Let 
be a function where 
are the independent variables and 
are the p parameters lying in the domain A. If A is not the whole
space 
, the problem is said to be constrained.
If a situation can be observed by a set of events
, i.e. a
set of couples representing the measured dependant and
variables, it is possible to deduce the value of the
parameters of your model 
corresponding to that situation.
As the measurements are generally given with some error, it is
impossible to get the exact value of the parameters but only an
estimation of them. Estimating is in some sense finding the most likely
value of the parameters. Much more events than parameters are in general
necessary.
In a linear problem, if the errors on the observations have a gaussian distribution, the ``Maximum Likelihood Principle'' gives you the ``best estimate'' of the parameters as the solution of the so-called ``Least Squares Minimization'' that follows:
with
The expected variance of the so--computed estimator is minimum among all approximation methods and is therefore called in statistics an ``efficient estimator''.
The quantities
are named the residuals and 
the weight of the 
observation that can be, for instance, the inverse of the computed
variance of the observation.
If 
depends linearly on each parameter 
, the problem is
also known as a Linear Regression and is solved in MIDAS by the command
REGRESSION. This chapter deals with 
which
have a non-linear dependance in a.
Let us now introduce some mathematical notations.
Let 
and 
be respectively the gradient and the Hessian
matrix of the function 
. They can be expressed by
where 
is the residuals vector
the Jacobian matrix of 
i.e.
and 
is
with 
, the Hessian matrix of 
.
In the rest of the chapter, all the functions are supposed to be differentiable if they are applied the derivation operator even when this condition is not necessary for the convergence of the algorithm.
A certain number of numerical methods have been developed to solve the
non--linear least squares problem, four have so far been implemented in
MIDAS. A complete description of these algorithms can be found in [1]
and [3], the present document will only give a basic introduction.