date: 2024-11-11
title: SVM
status: DONE
author:
  - AllenYGY
tags:
  - NOTE
  - SVM
  - MachineLearing
publish: True

SVM

SVM optimization problem

We can write the constraints as
When we construct the Lagrangian for our optimization problem, we have:
Let’s find the dual form of the problem.
- First minimize with respect to and (for fixed ), to get .
We’ll do this by setting the derivatives of with respect to and to zero:
We have: and . Plugging back into the Lagrangian equation:

Hard SVM

Hyperplane:
Constraint:
Goal: s.t.
Lagrangian:
Partial derivative:
Solution:
Lagrangian becomes: s.t. and
Weight vector:
Bias:

Soft SVM

Hyperplane:
Constraint:
Goal:
Lagrangian:
Partial Derivative:
Solution:
Dual Problem:
s.t.

Weight vector:
Bias:

The reason that ξ disappears: The slack variables disappear in the dual problem because they are implicitly handled through the Lagrange multipliers .
By taking the derivative of the Lagrangian with respect to , we obtain: This relationship ensures that is bounded by .
Consequently, the slack variables do not explicitly appear in the dual formulation. Instead, the dual problem balances maximizing the margin and allowing for misclassification through the constraint on .

Kernel SVM

Hyperplane:
Constraint:
Goal:
Lagrangian (Dual):
s.t.
Weight vector:
Decision Function:
Bias:
Kernel Functions:
Linear:
Polynomial:
Gaussian (RBF):
Sigmoid: