LU decomposition

Every nonsingular matrix $$A$$ can be factored as $$A=PLU$$ where $$P$$ is a permutation matrix, $$L$$ unit lower triangular and $$U$$ is an upper triangular matrix. The diagonal values of $$U$$ are the pivots of Gauss elimination. The cost of this factorization is $$(2/3)n^3$$

LU decomposition is unique (assuming that $$A$$ is unique and $$L$$ has 1s on the diagonal), which is a useful fact for theoretical purposes.

Note that not every matrix has $$LU$$ decomposition. A matrix has $$LU$$ decomposition iff it's nonsingular and all of its pivots are nonzero. Another way to characterize the existence of $$LU$$ factors is the following: Each leading principal submatrix of the matrix is nonsingular. But fortunately, it is possible to obtain the $$PLU$$ decomposition as above for any nonsingular matrix.

How to compute LU factors?
Fortunately, every matrix does have a $$PLU$$ decomposition where $$P$$ is a permutation matrix. In fact it is rather easy to obtain PLU decomposition for any matrix: It suffices to do Gauss elimination with Type III operations (see page144-145 ). Both operations are actually rather simple to program. The $$LU$$ factors here are obtained straightforwardly as the result of the Type III operations. The coefficients of Type III operations that are used to annihilate the values of $$A$$ at each iteration form the $$L$$ matrix, and the upper triangular matrix that remains as the result of the Matrix elimination becomes the $$U$$ matrix. The row interchange that is carried out for partial pivoting gives the matrix $$P$$.

How to solve $$Ax=b$$
Once $$LU$$ decomposition is obtained, solving the system $$Ax=b$$ is very simple thanks to the upper and lower triangular matrices. That is, one first solves $$Ly=b$$ with forward substitution and then $$Ux=y$$ with backward substitution (see page145-146). If there is a permutation matrix $$P$$ (i.e., we need to solve $$PAx=b$$, we can simply shuffle the rows of $$b$$ into $$\tilde{b}$$ instead of incorporating the matrix $$P$$ into the solution (see page 146 )

What's the point of LU decomposition?
LU decomposition is widely used to solve systems of the kind of $$Ax=b$$. It is much more efficient to solve these systems with LU decomposition instead of computing $$A^{-1}$$. In fact, even computation of $$A^{-1}$$ is done more efficiently with LU decomposition (by solving $$AX=I$$). LU decomposition is efficient not only in terms of speed but also memory; one can progressively overwrite the matrix $$A$$ and compute $$LU$$ factorization without additional memory.

In addition, $$LU$$ decomposition is unique, which is useful for theoretical proofs. E.g. it can be used to show that $$LDL^T$$ factorization is unique, too (see exercise 3.10.9.b ).