EE263 Overview

Below are my notes from the online EE263 class taught by Stephen Boyd. The sections in the summary below refer to the lecture notes rather than the videos. Not all lecture notes are included -- only the ones used in the video. Also, some notes mentioned in the video are omitted, because there is more focus on linear algebra parts rather than the LDSs.

Lecture 1: Overview
Linear dynamical systems (LDSs) are introduced: Equations, terminology, the applications and reasons to study LDSs are provided. They are motivated in comparison to non-linear dynamic systems.

Lecture 2: Linear functions and examples
Linear functions and the matrix multiplication function are introduced. It is shown that every linear function from vector to vector ($$f:\mathbb{R}^n\to \mathbb{R}^m$$) can be written as a matrix multiplication function ($$f(x) = Ax$$) and vice versa.

Interpretations of the matrix multiplication function $$y=Ax$$ are given. Specifically, the interpretation of each matrix element of $$a_{ij}$$ is discussed: $$a_{ij}$$ is the gain factor from $$j$$th input to $$i$$th output. Examples of applications are given about to illustrate the importance of this intuitive explanation.

Block diagram representation is also illustrated.

Linearization is introduced.

Broad categories of application are given to illustrate the usefullness of solving $$y=Ax$$.

Different ways of interpreting matrix multiplication are illustrated both for matrix-vector multiplication and for matrix-matrix multiplication (some are listed in Matrix multiplication).

Lecture 3: Linear algebra review
Basic of linear algebra are revised, such as: vector spaces, subspaces, independence, basis, dimension, nullspace (and its interpretations), range and interpretation of range, onto matrices, inverse matrices, rank and its usage for fast matrix-vector multiplication, inner product, angle and norm, and the Cauchy-Schwartz inequality.

Lecture 4: Orthonormal sets of vectors and QR factorization
orthonormal set of vectors and their geometric properties (i.e., isometric; preserving inner product, norm and angle) are discussed. The concept of orthonormal basis and orthogonal matrices are also introduced. Interpretations of the expension in orthonormal basis, $$x = U U^T x= \sum_{i=1}^n (u_i^T) u_i$$ (where $$U$$ is orthogonal) are given:


 * $$u_i^T x$$ is called the component of $$x$$ in direction $$u_i$$
 * $$a = U^T$$ resolves $$x$$ into the vector of its $$u_i$$ components
 * $$x=Ua$$ reconstitutes $$x$$ from its $$u_i$$ components
 * $$x=Ua$$ is the $$(u_i)$$ expansion of $$x$$

The Gram Schmidt (GS) procedure and its general version (i.e., the one that can be used on dependent vectors too) are given. Applications of the GS procedure are listed:
 * Yielding orthonormal basis for $$\mathcal{R}(A)$$
 * Yielding factorization $$A=BC$$ with full-ranked $$B,C$$ which is useful for fast matrix-vector multiplication (see also Rank)
 * Important: To check if a given vector is in $$R(A)$$
 * Revealing rank
 * Staircase pattern showing which columns of $$A$$ depend on previous ones

The full QR factorization is also discussed. Of note, the section nicely ties in the full QR factorization to the Fundamental Theorem of Linear Algebra.

Lecture 5: Least-squares
Least squares (LS) to approximately solve an overdetermined set of equations (skinny matrix) is introduced. The geometric interpretation of LS (i.e., projecting $$y$$ onto $$\mathcal{R}(A)$$ and orthogonality between projection and residual) is shown. Properties such as BLUE property are discussed. A motivating example from the first lecture (the one where RMS error is lower than quantization error) is shown.

Lecture 6: Least-squares applications
Applications of LS are shown, e.g., polynomial fitting, system identification. An alternative method to LS that is appropriate to growing sets of data (i.e., online LS) is shown too (see also Least squares).

Lecture 7: Regularized least-squares and Gauss-Newton method 7
Regularized LS is shown -- it's nothing but a special case of multi-objective LS (which is also discussed in the same lecture). The Gauss-Newton method for non-linear least squares is also shown.

Lecture 8: Least-norm solution
Least-norm solution: This time we are dealing with an underdetermined set of equations (fat matrix) and there are infinite possible solutions. As its name suggests, least-norm is the solution with the smallest norm. Geometric interpretation: $$x_{ln}$$ is the projection of $$0$$ on solution set $${x : y = Ax}$$ (also, $$x_{ln} \perp \mathcal{N}(A)$$).

The relation between least-norm and regularized LS is discussed. Also, the least-norm solution is derived via Lagrange, too. It is shown that both least-norm and least-squares are special cases of norm minimization with equality constraints.

Lecture 9: Autonomous linear dynamical systems
Autonomous LDSs are discussed, examples are given. Linearization near equilibrium and along trajectory point is discussed.

Lecture 10: Solution via Laplace transform and matrix exponential
Laplace Transform is introduced and its usage for solving autonomous LDS is shown. Characteristic Polynomial is also introduced. Finding the eigenvalues is introduced. Matrix exponential is introduced, and its usage for solving an autonomous LDS is shown. Properties of matrix exponential as a generalization of the exponential are shown. Interpretations of having real or complex eigenvalues are introduced. Stability of a system and its relation to the eigenvalues are shown.

Lecture 11: Eigenvectors and diagonalization
Eigenvalues and left and right eigenvectors are introduced. The scaling interpretation for eigenvectors is given. The relationship between eigenvalues, eigenvectors and the behavior of an LDS is introduced. As an application, the usage of right eignevectors for finding the steady state distribution of a Markov chain is shown.

Diagonalization is introduced, which is very useful for studying systems and showing certain properties of systems (i.e., for proving things). modal form is also introduced, and its usages are discussed (i.e., simplifying expressions and analyzing systems -- see WTPO? in Modal Form). Real Modal Form is shown for to analyze with real matrices systems that have complex eigenvalues.

Lecture 12: Jordan canonical form
Jordan Canonical Form is introduced, which is a generalization of modal form that works for any square matrix (i.e., works for non-diagonalizable matrices too). The Cayley-Hamilton theorem is also introduced -- this is a useful theorem for showing that any power of an $$n \times n$$ matrix is in the span of the set of the first $$n-1$$ powers of the matrix (see WTPO in Cayley-Hamilton theorem).

Lecture 13: Linear dynamical systems with inputs & outputs
Description of an LDS with i/o is given along with terminology of the matrices and vectors that define the system as well as a block diagram. Of note, the "idea of state" is also presented with 4 bullet points (see slide 13-29). The transfer matrix, impulse matrix (or impulse response), step matrix are discussed. It is shown how to use those concepts and convolution to understand the behavior of a system (e.g. see slide 13-9). The discretization of an LDS and the discrete-time transfer function are shown too.

Lecture 15: Symmetric matrices, quadratic forms, matrix norm, and SVD
The title is of the lecture pretty comprehensive. Symmetric matrices are introduced, which are very useful for studying quadratic forms because the eigenvalues of symmetric matrices are real and define upper/lower bounds on quadratic forms. Also, the eigendecomposition of a symmetric matrix is introduced (see Symmetric matrices too), which is a very useful decomposition for showing --proving-- things. The interpretation of the latter decomposition is discussed. Positive (semi)-definite matrices are introduced and used to define matrix inequalities. Ellipsoids are introduced too. The concept of matrix gain in a direction is discussed and used to define matrix norm. SVD is introduced too, and its interpretation is discussed (see SVD).

Lecture 16: SVD Applications
Application of SVD as a general pseudo-inverse is shown. The full SVD is introduced and used for giving an interpretation to the operation of SVD. SVD is discussed in the context of estimation ($$y=Ax$$) too, and it's used to show the optimality of Least-squares from another perspective. The usage of SVD for low-rank matrix approximation is discussed. Condition number and its utility is introduced. The distance of a matrix to singularity is discussed -- a very useful concept for matrices that are technically non-singular but in practice are pretty close to singularity.