Note-1: Please read this note first. Note-2: While this page has the basic theoretical content, more elaborate explanations/visualizations will be added in sometime. Will notify on Moodle after adding.

Page Author: [Shubodh Sai](mailto:[email protected]?Subject=The%20Notion%20Note-taking%20Project%20|%20Least%20Squares%20Optimization)

Using this Notion page

(click on triangle shaped thing to toggle)

• Read this to know how to use this page the right way.

Resources

• Resources: Books are referred to by their codenames in this page.

Introduction

• Be it Machine Learning or Computer Vision or Robotics problems, a common approach to formulating them is as optimization problems.
  • Formulate an objective function first.
  • Then, one of the most common ways is to pose it as Minimizing $squared$ error.
  • Other ways include Minimizing cross-entropy or Maximizing log-likelihood (Ex: Deep Learning), Maximizing discounted sum of rewards (Ex: Reinforcement Learning).
• We look at Minimizing $squared$ error.

Linear Least Squares

Linear Algebra: The Overdetermined System

$$ A x=b \\\quad A \: is\: m \times n \text{ matrix where } m>n. $$

Note: We will be considering only full rank matrices. There are special cases like a low rank (like enforcing in F matrix computation), unknown rank systems etc which we won't be covering in this lecture.

In practice, it is full rank (noise) and overdetermined system (data). Assumed throughout this lecture.