date: 2023-12-23
title: Orthogonality
status: DONE
author:
  - AllenyGy
tags:
  - Orthogonality
  - Lec3
  - NOTE
  - LinearAlgebra
created: 2023-12-23T02:12
updated: 2024-04-08T22:58
publish: True

Orthogonality

1.1. The Scalar Product in

1.1.1. Scalar Product

Definition:
For vectors x and y in , the 1×1 matrix (also treated as a scalar) is called the scalar product of x and y, and is given by
Other names: inner product ⟨x,y⟩, dot product x⋅y

1.1.2. Length

Definition:
For a vector x in , the Euclidean length (norm, magnitude)
‖x‖ of x is defined to be
A vector u in \mathbb{R}^n is a unit vector if and only if ‖u‖=1.
If x is a nonzero vector, then is a unit vector in the direction of x.
The distance between two vectors x and y is

1.1.3. Angle

Definition:
Let the angle between the two vectors be 𝜃 (0≤𝜃≤𝜋), then =‖x‖‖y‖ cos⁡𝜃, i.e. if u=x/‖x‖ ,𝐯=y/‖y‖

Orthogonal

Definition:
If =0, then we say that x and y are orthogonal, and denote x{\bot}y.

1.1.4. Projection

Projecting vector onto a line

scalar projection: .
vector projection: .
If u is a unit vector, then
The scalar projection of x onto u is
The vector projection of x onto u is

Projection Matrix:

1.2. Orthogonal Subspaces

1.2.1. Orthogonal Subspaces

Definition:
Two subspaces 𝑋 and y of are said to be orthogonal if for every x∈𝑋 and every y∈y.
If 𝑋 and y are orthogonal, we write 𝑋{\bot}y.
Note: 𝑋 and y are subspaces of the same vector space .
In .
In ,

1.2.2. Orthogonal Complement

Definition:
For a subspace y of \mathbb{R}^n, the set of vectors in that are orthogonal to every vector in y will be denoted . Thus, ={ for every $ y∈y$}
The set is called the orthogonal complement of y
is the “biggest” subspace orthogonal to y.

1.2.3. Theorem 3.2.1

If 𝑋 and y are orthogonal subspaces of , then 𝑋∩y={𝟎}

1.2.4. Theorem 3.2.2

If y is a subspace of , then is also a subspace of .

1.2.5. Fundamental subspaces

Let A be an m×n matrix.
- As a linear transformation, . (m-n)
  N(A) is a subspace of (where ).
  The range of A = the column space of A (subspace of ):
  R(A)={ |b = Ax for some }
- As a linear transformation, . (n-m)
  N() is a subspace of (where ).
  The column space of , is a subspace of
  ={y =A^T x$ for some }

The column space of is basically the row space of A, except row vectors “stands up” and become column vectors in .
Thus, if and only if is in the row space of A.

1.2.6. Theorem 3.2.3

If A is an m×n matrix, then

^[1]

1.2.7. Theorem 3.2.4

If S is a subspace of , then dim + dim =n.
Furthermore, if {} is a basis for S and {} is a basis for , then {} is a basis for .

1.2.8. Direct Sum

Definition:
If U and 𝑉 are subspaces of a vector space 𝑊 and each 𝐰∈𝑊 can be written uniquely as a sum u + 𝐯, where u∈U and 𝐯∈𝑉, then we say that 𝑊 is a direct sum of U and 𝑉, and we write

1.2.9. Direct Sum Theorem 3.2.5

If S is a subspace of , then .

1.2.10. Theorem 3.2.6

If S is a subspace of , then

For a subspace S in :

For a m×n matrix A:
Subspaces of . ^[2]
Subspaces of . ^[3]

1.3. Least Squares Problems

1.3.1. Line of best fit

1.3.2. Least Square Solutions

For an m×n matrix A, if the linear system Ax=b does not have a solution , we still want a vector such that is closest to b.
If , we seek to minimize , which is the same as minimizing ‖𝑟(x)‖
A vector that minimizes is called a least squares solution of .
is the vector in , the column space of A, that is closest to b.

Therefore,

Summary

To solve the least squares problem
We need to solve
Least squares solution x minimizes ‖𝑟(x)‖=‖b−Ax‖ and b−Ax is orthogonal to R(A).

1.3.3. Theorem 3.3.1

If A is an m×n matrix of rank n, the normal equation have a unique solution is the unique least squares solution of .

1.3.4. Projection formula

If A is an m×n matrix of rank n (i.e. columns are independent),

The unique least squares solution of Ax=b is $\hatx =(A^T A)^{−1} A^T b$
The projection of b onto R(A) is $p=A\hatx =A(A^T A)^{−1} A^T b$
Projection is a linear operator, with projection matrix

1.4. Ⅴ Orthonormal Sets

1.4.1. Note on Inner Product Spaces

For ANy vector space 𝑉, there is an abstract definition of inner product ⟨x,y⟩: it must satisfy

⟨x,x⟩≥0, with equality if and only if x=𝟎
⟨x,y⟩=⟨y,x⟩
⟨x+𝛽y,𝐳⟩=⟨x,𝐳⟩+𝛽⟨y,𝐳⟩

⟨x,y⟩ in , it means the scalar product .

1.4.2. Orthogonal Set

Definition:
A set {} of nonzero vectors is said to be an orthogonal set if ⟨⟩=0 whenever i≠𝑗.

1.4.3. Orthonormal Set

Definition:
An orthonormal set of vectors is an orthogonal set of unit vectors.
you can turn an orthogonal set into an orthonormal set by scaling each vector to have unit length:

1.4.4. Theorem [Nonzero orthogonal set is linearly independent] 3.5.1

If {} is an orthogonal set of nonzero vectors, then is linearly independent. ^[4]

1.4.5. Orthonormal Basis

If 𝐵={} is an orthonormal set, then by the theorem above, 𝐵 is a basis for the subspace S=Span()=R(U).
We say that 𝐵 is an orthonormal basis for S.

Orthonormal Basis and Coordinates

Theorem 3.5.2

Let {} be an orthonormal basis for 𝑉. If , then .
In other words, for any x∈𝑉 then
^[5]
In other word,

Parseval’s formula
If , then ‖x‖
in other word, ‖x‖

Orthonormal Sets and Projection

The unique least squares solution of Ax=b is $\hatx =(A^T A)^{−1} A^T b$
The projection of b onto R(A) is $p=A\hatx =A(A^T A)^{−1} A^T b$
Projection is a linear operator, with projection matrix

Theorem 3.5.3

If the column vectors of m×n matrix A form an orthonormal set of vectors in , then the projection of onto S=R(A) is $p=A\hatx =AA^Tb$ projection matrix:

Theorem [Projection formula] 3.5.4

If {} is an orthonormal basis for a nonzero subspace S of , then the projection of onto S is

1.4.6. Orthogonal matrix

Definition:
An n×n matrix 𝑄 is called an orthogonal matrix if the columns of 𝑄 form an orthonormal basis in .

1.4.7. Theorem 3.5.5

An n×n matrix 𝑄 is an orthogonal matrix if and only if

Properties
Let 𝑄 be an n×n matrix. The following are equivalent:
- 𝑄 is an orthogonal matrix.
- The columns of 𝑄 form an orthonormal basis for .
- , that is, .
- for all .
- ‖𝑄x‖=‖x‖ for all .

Orthonormal Sets and Least Squares

Least Square
$A^T A\hatx =A^T b⇒\hatx =(A^T A)^{−1} A^T b$

Theorem 3.5.4

If the column vectors of m×n matrix A form an orthonormal set of vectors in , then and the solution to the least squares problem Ax=b is $\hatx =A^T b$

1.5.Summary

Orthogonal set: when i≠𝑗
Vectors in an orthogonal set are linearly independent
Orthonormal set/basis: =𝛿
Orthogonal matrix:
If {} is an orthonormal basis for 𝑉:
- $x=⟨x,u_1 ⟩ u_1+…+⟨x,u_n ⟩ u_n $
- , ‖x‖
If {} is an orthonormal basis for a subspace S=R(U):
Least squares solution to Ux=b is $\hatx =U^T b$
Projection of b onto S is

↩︎
↩︎
↩︎
If =𝟎, need to show .
Taking the inner product of 𝐯_1 on both sides:

but since , ⟨⟩=‖‖>0, so .↩︎
Proof: ↩︎