What is information, mathematically ?
Answer given by Shannon’s Information Theory
Information is created when you reduce uncertainty
- So, can we quantify information ?
- If X is a discrete random variable, with a range R = {x1, x2, …}, and pi = P{X = xi}, then:
- ?i >=1 ( - pi log pi ) = a measure of “information” provided by an observation of X.
- This is called the “entropy” function.
- The entropy function also happens to be a measure of the “uncertainty” or “randomness” in X.