Moore-Penrose Pseudoinverse

The most popular type of pseudo-inverse of a matrix $$A$$ is the Moore-Penrose pseudoinverse, which is typically denoted as $$A^\dagger$$. The form of this specific matrix depends on the type of matrix.

$$A^\dagger = (A^T A)^{-1} A^T$$. In this case, $$A^\dagger$$ is a left inverse.
 * If $$A$$ is skinny and full rank, we generally refer to:

$$A^\dagger = A^T (AA^T)^{-1}$$. In this case, $$A^\dagger$$ is a right inverse.
 * If $$A$$ is fat and full rank, then we refer to the pseudo-inverse:

$$A^\dagger = V \Sigma^{-1} U^T$$ where $$V, \Sigma, U$$ are obtained from SVD.
 * If $$A\neq 0 $$ (i.e., the most general condition), the typical pseudo-inverse is:

Equivalently, the pseudo-inverse in this most general condition can be represented as:

$$A^\dagger = A'(AA')^-A(A'A)^-A'.$$

Derivation of Moore-Penrose Inverse
Moore-Penrose is a direct consequence of the SVD decomposition. According to SVD, for any matrix, there exists a decomposition in the form

$$A = U \begin{pmatrix} C & 0 \\ 0 & 0 \end{pmatrix} V^T,$$

where $$U, V$$ are orthogonal and $$C$$ is a square and full-rank diagonal matrix. Moore-Penrose pseudoinverse is obtained by simply reversing everything in the equation:

$$A^\dagger = V \begin{pmatrix} C^{-1} & 0 \\ 0 & 0 \end{pmatrix}  U^T$$.

Interpretation
Moore-Penrose is directly linked to the Fundamental Theorem of Linear Algebra, and this can be understood as follows. Let us consider $$A^\dagger$$ and $$A^\dagger$$ as linear operators. The Fundamental Theorem of Linear Algebra says that there is exactly one solution to the (consistent) linear system $$Ax=b$$ such that $$x \in R(A^T)$$. This means that, if we consider the version of the linear operator $$A$$ that is restricted to $$R(A^T)$$, $$A_{/R(A^T)}$$, then the range of this operator is the entire $$R(A)$$, and since we have unique solution such that $$x\in R(A^T)$$, the operator $$A_{/R(A^T)}$$ is invertible. The restricted version $$A^\dagger_{/R(A)}$$ of the Moore-Penrose pseudoinverse is the inverse of the linear operator $$A_{/R(A^T)}$$. This specifies the precise way in which Moore-Penrose is an inverse -- it is the pseudoinverse that characterizes the Fundamental Theorem of Linear Algebra.