Solving linear equations

In general, solving a system of linear equations $$Ax=b$$ is pretty difficult. However, there are certain subsets of systems that are easy to solve. The idea is to take any system and convert it to this form that's easy to solve. Below are the main sets of systems that are easy to solve (we consider square matrices, $$A \in \mathbb R^{n\times n}$$) :
 * $$A$$ is diagonal ($$x_i = b_i/a_{ii}$$)
 * $$A$$ is lower triangular (start with $$x_1 = b_1/a_{11}$$ and substitute it going forward)
 * $$A$$ is upper triangular (start with $$x_n = b_n/a_{nn}$$ and substitute it going backward)
 * $$A$$ is orthogonal ($$x=A^Tb$$)
 * $$A$$ is permutation matrix

Factorization
The main strategy for solving general $$Ax=b$$ can now be stated as factoring $$A$$ into a favorable form $$A=A_1A_2\dots A_k$$ and then solving recursively by starting with $$A_1z_1 = b$$, then $$A_2 z_2 = z_1$$, until $$A_k z_k = z_{k-1}$$.

The factorizations that facilitate the solving of equations are (below $$L,U$$ denote a lower- and an upper-triangular matrix, respectively):
 * LU factorization, $$A=LU$$. Every nonsingular matrix $$A$$ can be factored as $$A=PLU$$, where $$P$$ is a permutation matrix
 * Cholesky factorization: every positive definite $$A$$ can be factored as $$A = LL^T$$
 * $$LDL^T$$: Every nonsingular symmetric matrix $$A$$ can be factored as $$A=PLDL^TP^T$$ where $$P^T$$ is a permutation matrix.

Equations with structured sub-blocks
If a matrix $$A$$ has as structured form according to which $$Ax =b$$ can be written as

$$\begin{bmatrix}A_{11} & A_{12} \\ A_{21} & A_{22}\end{bmatrix}\begin{bmatrix}x_1 \\ x_2\end{bmatrix} = \begin{bmatrix}b_1 \\ b_2\end{bmatrix} $$

where $$A_{11}$$ is nonsingular, then we can eliminate $$x_{11} = A_{11}^{-1}(b1-A_{21}A_{11}^{-1}b_1)$$, compute $$x_2$$ from here and then use $$x_2$$ to find $$x_1$$.

Structured matrix plus low rank term
If we try to solve a system that can be written as $$(A+BC)x = b$$ where $$Ax=b$$ is easy to solve and $$BC$$ is low-rank, then we can write this system as: $$\begin{bmatrix}A & B \\ C & -I\end{bmatrix}\begin{bmatrix}x \\ y\end{bmatrix} = \begin{bmatrix}b \\ 0\end{bmatrix}$$

now apply block elimination and solve $$(I+CA^{-1}B)y = CA^{-1}b$$, and then solve $$Ax = b-By$$