HEAD | PREVIOUS |

Elliptic Problems and Iterative Matrix Solution

$\nabla .(D\nabla \mathit{\psi})=-s.$ | $(6.1)$ |

$\mathbf{B}\mathit{\psi}=-\mathbf{s}.$ | $(6.2)$ |

$\mathit{\psi}=-{\mathbf{B}}^{-1}\mathbf{s}.$ | $(6.3)$ |

$\frac{\partial \mathit{\psi}}{\partial t}=D{\nabla}^{2}\mathit{\psi}+s.$ | $(6.4)$ |

${\mathit{\psi}}_{j}^{(n+1)}-{\mathit{\psi}}_{j}^{(n)}=\frac{D\mathit{\Delta}t}{\mathit{\Delta}{x}^{2}}({\mathit{\psi}}_{j+1}^{(n)}-2{\mathit{\psi}}_{j}^{(n)}+{\mathit{\psi}}_{j-1}^{(n)}+{s}_{j}^{(n)}\mathit{\Delta}{x}^{2}/D).$ | $(6.5)$ |

${\mathit{\psi}}_{j}^{(n+1)}-{\mathit{\psi}}_{j}^{(n)}=\frac{1}{2}({\mathit{\psi}}_{j+1}^{(n)}-2{\mathit{\psi}}_{j}^{(n)}+{\mathit{\psi}}_{j-1}^{(n)}+\frac{{s}_{j}^{(n)}}{D}\mathit{\Delta}{x}^{2}).$ | $(6.6)$ |

${\mathit{\psi}}_{j}^{(n+1)}-{\mathit{\psi}}_{j}^{(n)}=(\sum _{q}{a}_{q}{\mathit{\psi}}_{q}^{(n)}/\sum _{q}{a}_{q})-{\mathit{\psi}}_{j}^{(n)}+\frac{{s}_{j}^{(n)}\mathit{\Delta}t}{\sum _{q}{a}_{q}},$ | $(6.7)$ |

Figure 6.1: (a) One dimensional Gauss Seidel odd-even iteration produces
successive solutions for each half-step that form a web that
progresses upwards toward the solution. (b) In two dimensions the
alternate squares to be updated are all the red (lighter shaded), then all the black.

Consider a Fourier mode${}^{37}$ of wave number ${k}_{x}=p\mathit{\pi}/L$, where $p$ is the
integer mode number and $L$ is the length of the domain
(one-dimensional for simplicity) at whose ends Dirichlet boundary
conditions are assumed. Its half-update through eq. (6.6) (ignoring the source term), gives rise to an
amplification factor
$A={\mathit{\psi}}_{j}^{(n+1)}/{\mathit{\psi}}_{j}^{(n)}=\frac{1}{2}({e}^{\mathit{ip}\mathit{\pi}\mathit{\Delta}x/L}+{e}^{-\mathit{ip}\mathit{\pi}{\mathit{\Delta}}_{x}/L})=\mathrm{cos}(p\mathit{\pi}\mathit{\Delta}x/L)$ | $(6.8)$ |

$A\approx 1-\frac{1}{2}{\left(\frac{\mathit{\pi}\mathit{\Delta}x}{L}\right)}^{2}=1-\frac{1}{2}{\left(\frac{\mathit{\pi}}{{N}_{j}}\right)}^{2}.$ | $(6.9)$ |

$m=\frac{1}{2}\mathrm{ln}(1/F)/\mathrm{ln}A\approx \mathrm{ln}F{\left(\frac{{N}_{j}}{\mathit{\pi}}\right)}^{2}.$ | $(6.10)$ |

${\mathit{\psi}}_{j}^{(n+1)}-{\mathit{\psi}}_{j}^{(n)}=\mathit{\omega}[\hspace{1em}\left(\sum _{q}{a}_{q}{\mathit{\psi}}_{q}^{(r)}/\sum _{q}{a}_{q}\right)-{\mathit{\psi}}_{j}^{(n)}+\frac{{s}_{j}^{(n)}\mathit{\Delta}t}{\sum _{q}{a}_{q}}],$ | $(6.11)$ |

- There is an optimal value of $\mathit{\omega}$ somewhere between 1 and 2, where SOR converges fastest.
- If ${A}_{J}$ is the amplification factor for the corresponding Jacobi iteration [$\mathrm{cos}(\mathit{\pi}\mathit{\Delta}x/L)$ for a uniform problem] then the optimal value is $\mathit{\omega}={\mathit{\omega}}_{b}=2/(1+\sqrt{1-{A}_{J}^{2}})$.
- For this optimal $\mathit{\omega}$ the amplification factor for the SOR
is ${A}_{\mathit{SOR}}={\mathit{\omega}}_{b}-1=({A}_{J}{\mathit{\omega}}_{b}/2{)}^{2}$.
For the uniform case and large ${N}_{j}$, these imply

${\mathit{\omega}}_{b}\approx \frac{2}{1+\mathit{\pi}/{N}_{j}},\mathrm{\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}}\text{and}\mathrm{\hspace{1em}\hspace{1em}\hspace{1em}}{A}_{\mathit{SOR}}\approx 1-\frac{2\mathit{\pi}}{{N}_{j}}.$ $(6.12)$

Figure 6.2: Number of iterations required to converge a SOR solution of
Poisson's equation with uniform source on a mesh of length
${N}_{j}=32$. It is declared converged when the maximum $\mathit{\psi}$-change
in a step is less than ${10}^{-6}{\mathit{\psi}}_{\mathrm{max}}$. The minimum
number of iterations is found to be 63 at $\mathit{\omega}=1.85$. This
should be compared with theoretical values of
$\mathrm{ln}({10}^{6})({N}_{j}/2\mathit{\pi})=70$ at
$\mathit{\omega}=2/(1+\mathit{\pi}/{N}_{j})=1.821$.

There are other iterative matrix solution techniques, associated with
the name Krylov. Like the SOR solution technique, they
use just multiplication by the matrix, not inversion. That is a big
advantage for very sparse matrices arising in PDE solving. They go by
names like "Conjugate Gradient", "Newton
Krylov" and "GMRES". In some
situations they converge faster than SOR, and they don't require
careful adjustment of a relaxation parameter. However, they have their
own tuning problems associated with "preconditioning". These topics
are introduced very briefly in the final chapter.
${\nabla}^{2}\mathit{\phi}=-s=\mathrm{exp}(\mathit{\phi})-1.$ | $(6.13)$ |

- Linearize it about the current estimate of the solution.
- Solve or advance the linear problem.
- Repeat from 1, until converged on the nonlinear solution.

$s(\mathit{\phi})=s({\mathit{\phi}}^{(n)})+\frac{\partial s}{\partial \mathit{\phi}}\stackrel{~}{\mathit{\phi}}+\frac{1}{2!}\frac{{\partial}^{2}s}{\partial {\mathit{\phi}}^{2}}{\stackrel{~}{\mathit{\phi}}}^{2}\dots ,\mathrm{\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}}\text{where}\mathrm{\hspace{1em}\hspace{1em}\hspace{1em}}\stackrel{~}{\mathit{\phi}}=(\mathit{\phi}-{\mathit{\phi}}^{(n)}).$ | $(6.14)$ |

${\nabla}^{2}\stackrel{~}{\mathit{\phi}}-\mathrm{exp}({\mathit{\phi}}^{(n)})\stackrel{~}{\mathit{\phi}}=\mathrm{exp}({\mathit{\phi}}^{(n)})-1-{\nabla}^{2}{\mathit{\phi}}^{(n)}.$ | $(6.15)$ |

${\nabla}^{2}\stackrel{~}{\mathit{\phi}}+{\frac{\partial s}{\partial \mathit{\phi}}|}_{{\mathit{\phi}}^{(n)}}\stackrel{~}{\mathit{\phi}}=-s({\mathit{\phi}}^{(n)})-{\nabla}^{2}{\mathit{\phi}}^{(n)}.$ | $(6.16)$ |

$\frac{{\partial}^{2}\mathit{\phi}}{\partial {x}^{2}}+\frac{{\partial}^{2}\mathit{\phi}}{\partial {y}^{2}}=s(x,y)$ | $(6.17)$ |

Suppose the final solution of the system is denoted ${\mathit{\phi}}_{s}$. We can define a new dependent variable $\mathit{\psi}=\mathit{\phi}-{\mathit{\phi}}_{s}$, which is the error between some approximation of the solution ($\mathit{\phi}$) and the actual solution. Of course, while we are in the process of finding the solution, we don't know how to derive $\mathit{\psi}$ from $\mathit{\phi}$, because we don't yet know what ${\mathit{\phi}}_{s}$ is. That fact does not affect the following arguments. Substituting for $\mathit{\phi}=\mathit{\psi}+{\mathit{\phi}}_{s}$ in the differential equation and using the fact that ${\mathit{\phi}}_{s}$ exactly satisfies it and the boundary conditions, we immediately deduce that $\mathit{\psi}$ satisfies the homogeneous differential equation

$\frac{{\partial}^{2}\mathit{\psi}}{\partial {x}^{2}}+\frac{{\partial}^{2}\mathit{\psi}}{\partial {y}^{2}}=0,$ | $(6.18)$ |

${\mathit{\psi}}_{j,k}^{(n+1)}-{\mathit{\psi}}_{j,k}^{(n)}=\frac{1}{2}(\frac{{\mathit{\psi}}_{j+1,k}^{(n)}+{\mathit{\psi}}_{j-1,k}^{(n)}}{\mathit{\Delta}{x}^{2}}+\frac{{\mathit{\psi}}_{j,k+1}^{(n)}+{\mathit{\psi}}_{j,k-1}^{(n)}}{\mathit{\Delta}{y}^{2}}){(\frac{1}{\mathit{\Delta}{x}^{2}}+\frac{1}{\mathit{\Delta}{y}^{2}})}^{-1}-{\mathit{\psi}}_{j,k}^{(n)}.$ | $(6.19)$ |

${A}_{J}\equiv {\mathit{\psi}}_{j,k}^{(n+1)}/{\mathit{\psi}}_{j,k}^{(n)}=(\frac{\mathrm{cos}(p\mathit{\pi}/{N}_{x})}{\mathit{\Delta}{x}^{2}}+\frac{\mathrm{cos}(q\mathit{\pi}/{N}_{y})}{\mathit{\Delta}{y}^{2}}){(\frac{1}{\mathit{\Delta}{x}^{2}}+\frac{1}{\mathit{\Delta}{y}^{2}})}^{-1}.$ | $(6.20)$ |

$\begin{array}{ccc}\multicolumn{1}{c}{{A}_{J}}& \approx \hfill & 1-(\frac{{\mathit{\pi}}^{2}}{2{N}_{x}^{2}\mathit{\Delta}{x}^{2}}+\frac{{\mathit{\pi}}^{2}}{2{N}_{y}^{2}\mathit{\Delta}{y}^{2}}){(\frac{1}{\mathit{\Delta}{x}^{2}}+\frac{1}{\mathit{\Delta}{y}^{2}})}^{-1}\hfill \\ \multicolumn{1}{c}{}& =\hfill & 1-\frac{1}{2}[{\left(\frac{\mathit{\pi}}{{N}_{x}}\right)}^{2}\frac{\mathit{\Delta}{y}^{2}}{\mathit{\Delta}{x}^{2}+\mathit{\Delta}{y}^{2}}+{\left(\frac{\mathit{\pi}}{{N}_{y}}\right)}^{2}\frac{\mathit{\Delta}{x}^{2}}{\mathit{\Delta}{x}^{2}+\mathit{\Delta}{y}^{2}}]\hfill \\ \multicolumn{1}{c}{}& =\hfill & 1-\frac{1}{2}{\left(\frac{\mathit{\pi}}{{N}_{x}{N}_{y}}\right)}^{2}\frac{{N}_{y}^{2}\mathit{\Delta}{y}^{2}+{N}_{x}^{2}\mathit{\Delta}{x}^{2}}{\mathit{\Delta}{x}^{2}+\mathit{\Delta}{y}^{2}}.\hfill & \hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}(6.21)\end{array}$ |

For brevity in the rest of our equations, let's define a number to represent the second term

$M\equiv {N}_{x}{N}_{y}/\sqrt{\frac{{N}_{y}^{2}\mathit{\Delta}{y}^{2}+{N}_{x}^{2}\mathit{\Delta}{x}^{2}}{\mathit{\Delta}{x}^{2}+\mathit{\Delta}{y}^{2}}}$ | $(6.22)$ |

$\begin{array}{cccc}\multicolumn{1}{c}{{\mathit{\omega}}_{b}}& =\hfill & \frac{2}{1+\sqrt{(1+{A}_{J})(1-{A}_{J})}}\approx \frac{2}{1+\frac{\mathit{\pi}}{M}}.\hfill & \hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}(6.23)\end{array}$ |

The resulting amplification factor for SOR iteration using this ${\mathit{\omega}}_{b}$ is

${A}_{\mathit{SOR}}={\mathit{\omega}}_{b}-1\approx 1-\frac{2\mathit{\pi}}{M},$ | $(6.24)$ |

$m=-\mathrm{ln}F/\mathrm{ln}{A}_{\mathit{SOR}}\approx M\frac{\mathrm{ln}F}{2\mathit{\pi}}.$ | $(6.25)$ |

${\mathit{\psi}}^{(n+1)}-{\mathit{\psi}}^{(n)}=\mathit{\omega}[(-\mathbf{I}+\mathbf{U}){\mathit{\psi}}^{(n)}+\mathbf{L}{\mathit{\psi}}^{(n+1)}].$ |

Collecting $n$ terms together

$(\mathbf{I}-\mathit{\omega}\mathbf{L}){\mathit{\psi}}^{(n+1)}=[(1-\mathit{\omega})\mathbf{I}+\mathit{\omega}\mathbf{U}]{\mathit{\psi}}^{(n)},$ |

which can be written

${\mathit{\psi}}^{(n+1)}=(\mathbf{I}-\mathit{\omega}\mathbf{L}{)}^{-1}[(1-\mathit{\omega})\mathbf{I}+\mathit{\omega}\mathbf{U}]{\mathit{\psi}}^{(n)}=\mathbf{H}{\mathit{\psi}}^{(n)}.$ |

The eigenvalues of the advancing matrix $\mathbf{H}$ are the "amplification factors" for the true modes of the system. They are the solutions, $\mathit{\lambda}$, of $det(\mathbf{H}-\mathit{\lambda}\mathbf{I})=0$. But

$\mathbf{H}-\mathit{\lambda}\mathbf{I}=(\mathbf{I}-\mathit{\omega}\mathbf{L}{)}^{-1}\{(1-\mathit{\omega})\mathbf{I}+\mathit{\omega}\mathbf{U}-\mathit{\lambda}(\mathbf{I}-\mathit{\omega}\mathbf{L})\}$ |

So

$det\{\mathit{\lambda}\mathit{\omega}\mathbf{L}+(1-\mathit{\lambda}-\mathit{\omega})\mathbf{I}+\mathit{\omega}\mathbf{U}\}=0.$ |

Now the determinant of any matrix ${\mathit{\alpha}}^{-1}\mathbf{L}-\mathbf{D}+\mathit{\alpha}\mathbf{U}$, where $\mathbf{L}$ and $\mathbf{U}$ are lower and upper triangular parts and $\mathbf{D}$ is the diagonal, is independent of $\mathit{\alpha}$. One can see this by noticing that any term in the expansion of the determinant has equal numbers of elements from $\mathbf{U}$ as it has from $\mathbf{L}$; so the $\mathit{\alpha}$ factors cancel out. As implied by our notation, we can arrange the nodes in an appropriate order such that all the even coefficients are in the upper triangle and the odd coefficients in the lower triangle part of the matrix. This would be achieved by the simple expedient of putting all the even positions first. Actually we don't need to do the rearrangement. We just need to know it could be done. In that case, we can balance the upper and lower parts of the determinantal equation, multiplying the $\mathbf{L}$ term by ${\mathit{\lambda}}^{-1/2}$ and the $\mathbf{U}$ term by ${\mathit{\lambda}}^{1/2}$ to make it:

$det\{{\mathit{\lambda}}^{1/2}\mathit{\omega}\mathbf{L}+(1-\mathit{\lambda}-\mathit{\omega})\mathbf{I}+{\mathit{\lambda}}^{1/2}\mathit{\omega}\mathbf{U}\}=0,$ |

i.e.

$det\{-(\mathit{\lambda}+\mathit{\omega}-1){\mathit{\omega}}^{-1}{\mathit{\lambda}}^{-1/2}\mathbf{I}+\mathbf{L}+\mathbf{U}\}=0.$ |

Now notice that the eigenvalues $\mathit{\mu}$ for the Jacobi iteration matrix, $\mathbf{L}+\mathbf{U}$, satisfy $det(-\mathit{\mu}\mathbf{I}+\mathbf{L}+\mathbf{U})=0$, which is exactly the same equation with the identification $(\mathit{\lambda}+\mathit{\omega}-1){\mathit{\omega}}^{-1}{\mathit{\lambda}}^{-1/2}=\mathit{\mu}$. There's a direct mapping between eigenvalues of the Jacobi iteration and of the SOR iteration. The relationship can be considered a quadratic equation for $\mathit{\lambda}$, given $\mathit{\mu}$ and $\mathit{\omega}$

${\mathit{\lambda}}^{2}+(2\mathit{\omega}-2-{\mathit{\omega}}^{2}{\mathit{\mu}}^{2})\mathit{\lambda}+(\mathit{\omega}-1{)}^{2}=0.$ |

The optimum $\mathit{\omega}$ gives the smallest magnitude of the larger $\mathit{\lambda}$ solution. It occurs when the $\mathit{\lambda}$ roots coincide, i.e. when $(\mathit{\omega}-1-{\mathit{\omega}}^{2}{\mathit{\mu}}^{2}/2{)}^{2}=(\mathit{\omega}-1{)}^{2}$ whose solution is

$\mathit{\omega}={\mathit{\omega}}_{b}=\frac{2}{1+\sqrt{1-{\mathit{\mu}}^{2}}}.$ |

The corresponding eigenvalue is $\mathit{\lambda}={\mathit{\omega}}_{b}-1$. For $\mathit{\omega}>{\mathit{\omega}}_{b}$, the roots for $\mathit{\lambda}$ are complex with magnitude $\mathit{\omega}-1$. Therefore SOR is stable only for $\mathit{\omega}<2$, and the convergence rate degrades linearly to zero between ${\mathit{\omega}}_{b}$ and 2.

1. Start with your code that solved the diffusion equation explicitly. Adjust it to always take timesteps at the stability limit $\mathit{\Delta}t=\mathit{\Delta}{x}^{2}/2D$, so that:

${\mathit{\psi}}_{j}^{(n+1)}-{\mathit{\psi}}_{j}^{(n)}=(\frac{1}{2}{\mathit{\psi}}_{j+1}^{(n)}-{\mathit{\psi}}_{j}^{(n)}+\frac{1}{2}{\mathit{\psi}}_{j-1}^{(n)}+\frac{{s}_{j}^{(n)}}{2D}\mathit{\Delta}{x}^{2}).$ |

Now it is a Jacobi iterator for solving the steady-state elliptic matrix equation. Implement a convergence test that finds the maximum absolute

$\frac{{d}^{2}\mathit{\psi}}{{\mathit{dx}}^{2}}=1$ |

on the domain $x=[-1,1]$ with boundary conditions $\mathit{\psi}=0$, with a total of ${N}_{x}$ equally-spaced nodes. Find how many iterations it takes to converge, starting from an initial state $\mathit{\psi}=0$, when (a) ${N}_{x}=10$ (b) ${N}_{x}=30$ (c) ${N}_{x}=100$ Compare the number of iterations you require with the analytic estimate in the notes. How good is the estimate? Now we want to check how accurate the solution really is. (d) Solve the equation analytically, and find the value of $\mathit{\psi}$ at $x=0$, $\mathit{\psi}(0)$. (e) For the three ${N}_{x}$ values, find the relative error${}^{41}$ in $\mathit{\psi}(0)$. (f) Is the actual relative error the same as the convergence test value ${10}^{-5}$? Why?

2. Optional and not for credit. Turn your iterator into a SOR solver by splitting the iteration matrices up into red and black (odd and even) advancing parts. Each part-iterator then uses the latest values of $\mathit{\psi}$, that has just been updated by the other part-iterator. Also provide yourself an over-relaxation parameter $\mathit{\omega}$. Explore how fast the iterations converge as a function of ${N}_{x}$ and $\mathit{\omega}$.

Note. Although in Octave/Matlab it is convenient to implement the matrix multiplications of the advance using a literal multiplication by a big sparse matrix, one does not do that in practice. There are far more efficient ways of doing the multiplication, that avoid all the irrelevant multiplications by zero.

HEAD | NEXT |