Orthogonality

• Definition:
  For vectors x and y in , the 1×1 matrix (also treated as a scalar) is called the scalar product of x and y, and is given by
  
• Other names: inner product ⟨x,y⟩, dot product x⋅y

• Definition:
  For a vector x in , the Euclidean length (norm, magnitude)
  ‖x‖ of x is defined to be
  
• A vector u in \mathbb{R}^n is a unit vector if and only if ‖u‖=1.
• If x is a nonzero vector, then is a unit vector in the direction of x.
• The distance between two vectors x and y is

• Definition:
  Let the angle between the two vectors be 𝜃 (0≤𝜃≤𝜋), then =‖x‖‖y‖ cos⁡𝜃, i.e. if u=x/‖x‖ ,𝐯=y/‖y‖

• Definition:
  Two subspaces 𝑋 and y of are said to be orthogonal if for every x∈𝑋 and every y∈y.
  If 𝑋 and y are orthogonal, we write 𝑋{\bot}y.
  Note: 𝑋 and y are subspaces of the same vector space .
• In .
• In ,

• Definition:
  For a subspace y of \mathbb{R}^n, the set of vectors in that are orthogonal to every vector in y will be denoted . Thus, ={ for every $ y∈y$}
• The set is called the orthogonal complement of y
• is the “biggest” subspace orthogonal to y.

If 𝑋 and y are orthogonal subspaces of , then 𝑋∩y={𝟎}

If y is a subspace of , then is also a subspace of .

• Let A be an m×n matrix.
  • As a linear transformation, . (m-n)
    N(A) is a subspace of (where ).
    The range of A = the column space of A (subspace of ):
    R(A)={ |b = Ax for some }

  ---

  • As a linear transformation, . (n-m)
    N() is a subspace of (where ).
    The column space of , is a subspace of
    ={y =A^T x$ for some }

  ---

The column space of is basically the row space of A, except row vectors “stands up” and become column vectors in .
Thus, if and only if is in the row space of A.

If A is an m×n matrix, then

• 
  ^[1]

If S is a subspace of , then dim + dim =n.
Furthermore, if {} is a basis for S and {} is a basis for , then {} is a basis for .

• Definition:
  If U and 𝑉 are subspaces of a vector space 𝑊 and each 𝐰∈𝑊 can be written uniquely as a sum u + 𝐯, where u∈U and 𝐯∈𝑉, then we say that 𝑊 is a direct sum of U and 𝑉, and we write

If S is a subspace of , then .

If S is a subspace of , then

For a subspace S in :

• For a m×n matrix A:
• Subspaces of . ^[2]
• Subspaces of . ^[3]

• For an m×n matrix A, if the linear system Ax=b does not have a solution , we still want a vector \hat{x} \in \mathbb{R}^n $  such that $A\hatx is closest to b.
• If 𝑟(x)=b−Ax, we seek to minimize ‖𝑟(x)‖, which is the same as minimizing ‖𝑟(x)‖
• A vector x \in \mathbb{R}^n that minimizes ‖𝑟(x)‖=‖b−Ax‖ is called a least squares solution of .
• is the vector in S=R(A), the column space of A, that is closest to b.

• 
  
• Therefore, b−A\hatx ∈N(A^T)
  

Summary

• To solve the least squares problem
• We need to solve
• Least squares solution x minimizes ‖𝑟(x)‖=‖b−Ax‖ and b−Ax is orthogonal to R(A).

If A is an m×n matrix of rank n, the normal equation have a unique solution  is the unique least squares solution of .

If A is an m×n matrix of rank n (i.e. columns are independent),

• The unique least squares solution of Ax=b is \hatx =(A^T A)^{−1} A^T b
• The projection of b onto R(A) is p=A\hatx =A(A^T A)^{−1} A^T b
• Projection is a linear operator, with projection matrix

For ANy vector space 𝑉, there is an abstract definition of inner product ⟨x,y⟩: it must satisfy

• ⟨x,x⟩≥0, with equality if and only if x=𝟎
• ⟨x,y⟩=⟨y,x⟩
• ⟨x+𝛽y,𝐳⟩=⟨x,𝐳⟩+𝛽⟨y,𝐳⟩

• ⟨x,y⟩ in , it means the scalar product .

• Definition:
  A set {} of nonzero vectors is said to be an orthogonal set if ⟨⟩=0 whenever i≠𝑗.

• Definition:
  An orthonormal set of vectors is an orthogonal set of unit vectors.
  you can turn an orthogonal set into an orthonormal set by scaling each vector to have unit length:

If {} is an orthogonal set of nonzero vectors, then is linearly independent. ^[4]

If 𝐵={} is an orthonormal set, then by the theorem above, 𝐵 is a basis for the subspace S=Span()=R(U).
We say that 𝐵 is an orthonormal basis for S.

• Definition:
  An n×n matrix 𝑄 is called an orthogonal matrix if the columns of 𝑄 form an orthonormal basis in .

An n×n matrix 𝑄 is an orthogonal matrix if and only if

• Properties
  Let 𝑄 be an n×n matrix. The following are equivalent:
  • 𝑄 is an orthogonal matrix.
  • The columns of 𝑄 form an orthonormal basis for .
  • , that is, .
  • for all .
  • ‖𝑄x‖=‖x‖ for all .