The Newton-Raphson Method.

This is the simplest one. The necessary condition for the function $\chi^2(a)$to have an extremum is that the partial derivatives vanish i.e.

$$\sum_{i}~r^{(i)}~{\partial {r^{(i)}} \over \partial{a_j}} = 0 \quad (j=1,\ldots,p)~~$$

or, equivalently,

J(a)^T r(a) =0   .

This is usually a system of non-linear equations that, numerically, can be solved using the Newton-Raphson's method also called in the one-dimensional case the tangents method. The Taylor development of the function limited to the first order is taken around some initial guesses of the parameters. The resulting linear system

$$J(a^{(k)})^T~J(a^{(k)})~\Delta a^{(k)}~=~-J(a^{(k)})~r(a^{(k)})$$

gives thus a correction to the solution and

$$a^{(k+1)}~=~a^{(k)}~+~\gamma~\Delta a^{(k)}$$

is taken as the new approximation of the optimum. The relaxation factor $\gamma$ is a parameter of the method. The convergence of the process towards the solution of the non-linear minimization problem has been proven for locally convex $\chi^2(a)$ or under other assumptions impossible to detail here. These conditions are not generally fulfilled in real problems. Moreover, the algorithm ignores the second order conditions and therefore, may end on a saddle point or never converge. Two different relaxation factors may lead to different solutions or one may give convergence and the other one not. No general rule can be given for the choice of a good relaxation factor.