The Principle of Maximum Entropy
The machine learning consultancy: https://truetheta.io
Join my email list to get educational and useful articles (and nothing else!): https://mailchi.mp/truetheta/true-theta-email-list
Want to work together? See here: https://truetheta.io/about/#want-to-work-together
What's the safest distribution to pick in the absence of information? What about in the case where you have some, though only partial, information? The Principle of Maximum Entropy answers these questions well and as a result, is a frequent guiding rule for selecting distributions in the wild.
SOCIAL MEDIA
LinkedIn : https://www.linkedin.com/in/dj-rich-90b91753/
Twitter : https://twitter.com/DuaneJRich
Enjoy learning this way? Want me to make more videos? Consider supporting me on Patreon: https://www.patreon.com/MutualInformation
Sources
Chapters 11-12 of [2] were primary sources - this is where I ironed out most of my intuition on this subject. Chapter 12 of [1] was helpful for understanding the relationship between the maximum entropy criteria and the form of the distribution that meets it. [3] was useful for a high level perspective and [4] was helpful for determining the list of maximum entropy distribution.
Also, thank you to Dr. Hanspeter Schmid of the University of Applied Sciences and Arts, Northwestern Switzerland. He helped me interpret some of the more technical details of [2] and prevented me from attaching an incorrect intuition to the continuous case - much appreciated!
[1] T. M. Cover and J. A. Thomas. Elements of Information Theory. 2nd edition. John Wiley, 2006.
[2] E. T. Jaynes. Probability theory: the logic of science. Cambridge university press, 2003.
[3] Principle of Maximum Entropy, Wikipedia, https://en.wikipedia.org/wiki/Principle_of_maximum_entropy
[4] Maximum Entropy Distribution, Wikipedia, https://en.wikipedia.org/wiki/Maximum_entropy_probability_distribution
Timestamps :
0:00 Intro
00:41 Guessing a Distribution and Maximum Entropy
04:16 Adding Information
06:40 An Example
08:00 The Continuous Case
10:26 The Shaky Continuous Foundation
Видео The Principle of Maximum Entropy канала Mutual Information
Join my email list to get educational and useful articles (and nothing else!): https://mailchi.mp/truetheta/true-theta-email-list
Want to work together? See here: https://truetheta.io/about/#want-to-work-together
What's the safest distribution to pick in the absence of information? What about in the case where you have some, though only partial, information? The Principle of Maximum Entropy answers these questions well and as a result, is a frequent guiding rule for selecting distributions in the wild.
SOCIAL MEDIA
LinkedIn : https://www.linkedin.com/in/dj-rich-90b91753/
Twitter : https://twitter.com/DuaneJRich
Enjoy learning this way? Want me to make more videos? Consider supporting me on Patreon: https://www.patreon.com/MutualInformation
Sources
Chapters 11-12 of [2] were primary sources - this is where I ironed out most of my intuition on this subject. Chapter 12 of [1] was helpful for understanding the relationship between the maximum entropy criteria and the form of the distribution that meets it. [3] was useful for a high level perspective and [4] was helpful for determining the list of maximum entropy distribution.
Also, thank you to Dr. Hanspeter Schmid of the University of Applied Sciences and Arts, Northwestern Switzerland. He helped me interpret some of the more technical details of [2] and prevented me from attaching an incorrect intuition to the continuous case - much appreciated!
[1] T. M. Cover and J. A. Thomas. Elements of Information Theory. 2nd edition. John Wiley, 2006.
[2] E. T. Jaynes. Probability theory: the logic of science. Cambridge university press, 2003.
[3] Principle of Maximum Entropy, Wikipedia, https://en.wikipedia.org/wiki/Principle_of_maximum_entropy
[4] Maximum Entropy Distribution, Wikipedia, https://en.wikipedia.org/wiki/Maximum_entropy_probability_distribution
Timestamps :
0:00 Intro
00:41 Guessing a Distribution and Maximum Entropy
04:16 Adding Information
06:40 An Example
08:00 The Continuous Case
10:26 The Shaky Continuous Foundation
Видео The Principle of Maximum Entropy канала Mutual Information
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
Linear Regression in 12 minutes
How to Learn Probability Distributions
Jensen's Inequality
Importance Sampling
The Bernstein Basis
The Exponential Family (Part 2)
Factor Analysis and Probabilistic PCA
Policy Gradient Methods | Reinforcement Learning Part 6
Why Do Tree Based-Models Outperform Neural Nets on Tabular Data?
The Exponential Family (Part 1)
Function Approximation | Reinforcement Learning Part 5
What happens at the Boundary of Computation?
Reinforcement Learning, by the Book
The Bias Variance Trade-Off
Gaussian Processes
Why Do Neural Networks Love the Softmax?
Temporal Difference Learning (including Q-Learning) | Reinforcement Learning Part 4
The Kelly Criterion
Bellman Equations, Dynamic Programming, Generalized Policy Iteration | Reinforcement Learning Part 2
The Fisher Information
Monte Carlo And Off-Policy Methods | Reinforcement Learning Part 3