Generalised Least Squares Is Still A Projection — Just In a Different Geometry

In this video, we develop the geometric interpretation of Generalised Least Squares (GLS).

In ordinary least squares, the solution is given by an orthogonal projection of the observation vector onto the model space. GLS turns out to be exactly the same idea — but in a different geometry.

We show that GLS solves a least squares problem not in the standard Euclidean space, but in a space equipped with a weighted inner product defined by the inverse covariance matrix (\Omega^{-1}):
[
\langle u, v \rangle_{\Omega^{-1}} = u^\top \Omega^{-1} v
]

This leads to a key result

GLS is the orthogonal projection of onto the image of the design matrix
but with orthogonality defined in the weighted geometry.

We cover:

* the weighted inner product and induced norm
* how the GLS objective corresponds to this geometry
* the projection theorem in the weighted space
* the orthogonality condition for the residual
* why GLS is still a projection, just with a different notion of angle

We also connect this to whitening:

After transforming the system using (\Omega^{-1/2}), the problem becomes a standard Euclidean projection. This shows that:

GLS = Euclidean projection in transformed coordinates
GLS = weighted projection in original coordinates

This provides a unifying interpretation:

GLS is not a different algorithm, it is the same projection principle, expressed in a geometry adapted to the noise.

This video is part of the Parametric Regression series:

If you're interested in a mathematically rigorous, first-principles approach to machine learning, consider subscribing.

#MachineLearning #Statistics #GLS #LinearRegression #Geometry #DataScience #Mathematics #StatisticalLearning

Видео Generalised Least Squares Is Still A Projection — Just In a Different Geometry канала ML & AI: Foundations & Methods