Laplace's definition.
Probability is the ratio of favourable outcomes to possible outcomes.
\[\P{} A = \frac{N_A}{N}\]
Theoretical statistics
The lecture notes are the primary material. The book is supplementary material and not required.
Lecture notes
The notes are located link.
1 Introduction
Interpretation of \(\P (A)\)
Classical
Frequentist
Subjectivist
Propensity
Deterministic
Axiomatic
2 Preliminary material
A gathering of concepts that should be familiar.
2.1 Set theory and operations
Set element
Set
\(\mathbb{N}\)
\(\mathbb{C}\)
Subset
Proper subset
Cardinality
Countability
Empty set
2.1
Natural numbers
2.1
Integers
2.1
Rationals
2.1
Reals
2.1
Complex numbers
2.1
Finite set
2.1
Set of sets
2.1
Set of \(n\times m\) matrices
2.1
Set union
2.1.1
Set intersection
2.1.1
Set complement in Ω
2.1.1
Set difference
2.1.1
Set symmetric difference
2.1.1
Cartesian product
2.1.1
Power set
2.1.1
Example
2.2
Irrationals
2.3
De Morgan's law for sets
2.1.1
Proof
2.2 Sequences of sets
Set sequence
Set sequence union
Set sequence intersection
Countable set sequence
Union or intersection of countable set sequence
Increasing set sequence
2.2.1
Decreasing set sequence
2.2.1
Increasing set sequence limit
2.2.1
Decreasing set sequence limit
2.2.1
Limit index arithmetic
Increasing & decreasing
2.4
Limit superior of a set sequence
Limit inferior of a set sequence
Limit of a set sequence
Exercise
2.1
Exercise
2.2
Exercise
2.3
Exercise
2.4
2.3 Convergence of functions
Pointwise convergence
2.3.1
Example
2.5
Example
2.6
Uniform convergence
2.3.2
Example
2.7
Example
2.8
Uniform convergence properties
2.3.1
Corollary
2.3.2
\(L^1\) · Lebesgue integrable
\(L^1\) convergence · Lebesgue convergence
2.3.3
Example
2.9
Corollary
2.3.3
Proof
3 A primer on measure and integration
Measure theory: measures, spaces and integration.
Measure idea
Weakness of Riemann integral idea
Lebesgue integration idea
Probability and measure idea
3.1 Measures and measure spaces
σ-algebra
3.1.1
Sigma algebra idea
Measurable set
3.1.2
Trivial σ-algebra
3.1
Power set
3.2
σ-algebra generated by \(F \subset{} E\)
3.3
Sub-σ-algebra
3.4
σ-algebra coarseness idea
σ-algebra generated by subsets
3.1.3
Intersection of σ-algebras
Borel σ-algebra on \(\mathbb{R}\)
3.5
Borel σ-algebra on \(\mathbb{R}\)-subset
3.6
Measurable space
3.1.4
Measure
3.1.5
σ-additivity
3.1.5
Finite additivity
3.1.1
Monotonicity
3.1.1
σ-subadditivity
3.1.1
Proposition
3.1.1
Proof
Continuity of measures
3.1.2
Proof
Characterization of measures
3.1.3
Remark
Finite measure
3.1.6
σ-finite measure
3.1.7
Remark
Trivial measure
3.7
Borel measures
3.8
Lebesgue measure
3.9
Lebesgue-Stieltjes measures
3.10
Dirac measure
3.11
Counting measures
3.12
Angular measure
3.13
Probability measures
3.14
Measure space
3.1.8
Example
3.15
Probability space
3.15
Negligible set
3.1.9
Complete measure
3.1.10
Example
3.16
Complete measure space
3.1.11
Remark
Example
3.17
\(\mathcal{L} (\R )\) · Lebesgue σ-algebra
3.17
The Cantor set
3.18
Product measure
3.1.12
Remark
Remark
Lebesgue measure in \(\mathbb{R}^d\)
3.19
Example
3.19
3.2 Measurable functions
Measurable
3.2.1
Set indicator function
3.20
Example
3.21
Measurability via generating set
3.2.1
Example
3.22
Borel measurable
3.2.2
Corollary
3.2.2
σ-algebra generated by a function
3.2.3
Measurability preserving operations
3.2.3
Measurable composition is measurable
3.2.4
Proof
Almost everywhere
3.2.4
Almost surely
3.2.5
Push-forward measure
3.2.6
3.3 Integration with respect to a measure
Simple function
3.3.1
Integral of simple function
3.3.2
Approximation by simple functions
3.3.1
Remark
Contour partition function
Approximation by contours
Approx by contours
Integral of non-negative measurable function
3.3.3
µ-integrable
3.3.4
Integral of measurable function
3.3.4
Theorem
3.3.2
Proof
Integrable if finite
3.3.3
Proof
Theorem
3.3.4
Proof
Remark
Integration wrt. pushforward measure
3.3.5
Corollary
3.3.6
3.4 Lebesgue's convergence theorems
When \(f_n \to{} f\), when does \(\int{} f_n d\mu{} \to{} \int{} f d\mu\)?
Proposition
3.4.1
Proof
Lebesgue's monotone convergence theorem
3.4.2
Proof
Fatou's lemma
3.4.3
Proof
Reverse Fatou's lemma
3.4.4
Proof
Fatou-Lebesgue theorem
3.4.5
Example
3.23
Example
3.24
Lebesgue's dominated convergence theorem
3.4.6
Proof
Example
3.25
Example
3.26
Example
3.27
TODO LimSup
Fatou's lemmas
3.5 Riemann integral
Riemann integral
3.6 Lebesgue integral
Lebesgue integral
Lebesgue vs. Riemann
3.6.1
Remark
3.7 Riemann-Stieltjes integral
Riemann-Stieltjes integral
Intuition behind Riemann-Stieltjes I
3.28
Intuition behind Riemann-Stieltjes II
3.29
3.8 Lebesgue-Stieltjes integral
Lebesgue-Stieltjes integral
3.9 Absolute continuity and the Radon-Nikodym theorem
Absolutely continuous measure
3.9.1
Equivalent measures
3.9.1
Example
3.30
Radon-Nikodym
3.9.1
Radon-Nikodym derivative
3.9.1
Remark
3.10 Exercises
Exercise
3.1
Exercise
3.2
Exercise
3.3
Exercise
3.4
Exercise
3.5
Exercise
3.6
Exercise
3.7
Exercise
3.8
Exercise
3.9
Exercise
3.10
Exercise
3.11
Exercise
3.12
Exercise
3.13
Exercise
3.14
Exercise
3.15
Exercise
3.16
Exercise
3.17
Exercise
3.18
Exercise
3.19
Exercise
3.20
Exercise
3.21
Exercise
3.22
Exercise
3.23
Exercise
3.24
Exercise
3.25
Exercise
3.26
Exercise
3.27
Exercise
3.28
4 An axiomatic approach to probability
Axiom
4.1 Sample space and events
Trial · Experiment
4.1.1
Random trial
4.1.1
Bernoulli trial
4.1.2
Sample space
4.1.3
Example
4.1
Power set
Example
4.2
Event
Example
4.3
Exercise
4.1
Exercise
4.2
4.2 σ-algebras of events
σ-algebra of events
Remark
Trivial σ-algebra
4.4
Power set
4.5
σ-algebra generated by event
4.6
Borel σ-algebra on \(\R\)
4.7
Borel σ-algebra on \(\R^d\)
4.8
Examples
4.9
σ-algebra generated by a set
4.2.2
Remark
Borel σ-algebra
4.2.1
Proof
Remark
Remark
Exercise
4.3
Exercise
4.4
Exercise
4.5
Exercise
4.6
Exercise
4.7
Exercise
4.8
Exercise
4.9
Exercise
4.10
4.3 Probability measure
Probability measure
4.3.1
σ-additivity
4.3.1
Finite additivity
4.3.1
Remark
Remark
Finite additivity
4.3.1
Proof
Corollary
4.3.2
Proof
σ-additivity equivalences
4.3.3
Proof
Exercise
4.11
4.4 Conditional probability and independence
Conditional probability
Independence · Mutual independence
4.4.1
Pairwise independence
4.4.1
Remark
Complement independences
4.4.1
Proof
Dependent & independent examples
4.10
Conditional probability
4.4.2
Theorem
4.4.2
Conditional probability measure
4.4.2
Proof
Chain rule
4.4.3
Proof
Partition
4.4.3
Finite partition
4.4.3
Countable partition
4.4.3
Law of total probability
4.4.4
Proof
Bayes' theorem
4.4.5
Proof
4.5 The Borel-Cantelli lemmas
Event sequences
Borel-Cantelli idea
First Borel-Cantelli lemma
4.5.1
Proof
Second Borel-Cantelli lemma
4.5.2
Proof
Exercise
4.12
Exercise
4.13
Exercise
4.14
Exercise
4.15
Exercise
4.16
Exercise
4.17
Exercise
4.18
Exercise
4.19
Exercise
4.20
Exercise
4.21
Exercise
4.22
Exercise
4.23
Exercise
4.24
Exercise
4.25
Exercise
4.26
Exercise
4.27
5 Random variables
Random variable motivation
5.1 Random variables
Random variable · Real random variable
5.1.1
Random variable name
Probability measure pushforward
Random variables
5.1
Not random variables
5.2
Generalized random variable
5.2
Random vector
5.1.2
Stochastic process
5.1.3
Exercise
5.1
5.2 Law and distribution of a random variable
Example
5.3
Law
5.2.1
Distribution · CDF · Cumulative distribution function
5.2.2
Law & CDF equivalence
Distribution properties
5.2.1
Proof
Random variable via CDF
Exercise
5.2
5.3 Types of random variables
Discrete random variable
5.3.1
Continuous random variable
5.3.1
Mixed random variable
5.3.1
Absolutely continuous random variable
5.3.2
Absolutely continuous function
Density function
5.3.2
Integral of density
CDF Lebesgue derivative
Non-absolutely continuous function
Examples of different distribution functions
5.4
5.4 Transformation of random variables
Change of variables in Lebesgue integral
5.4.1
Random variable under diffeomorphism
5.4.2
Proof
Corollary
5.4.3
Proof
Remark
Exercise
5.3
Exercise
5.4
Exercise
5.5
Exercise
5.6
Exercise
5.7
Exercise
5.8
5.5 Expectation
Expectation
5.5.1
Remark
Example
5.5
Example
5.6
Example
5.7
Law of the unconscious statistician
5.5.1
Proof
Remark
Remark
Exercise
5.9
Exercise
5.10
Exercise
5.11
Exercise
5.12
Exercise
5.13
Exercise
5.14
Exercise
5.15
Exercise
5.16
Exercise
5.17
5.6 Independence of random variables
Independent of sigma algebras
5.6.1
Independence of two sigma algebras
5.6.2
Remark
Independence of random variables
5.6.3
Independence of random variables
5.6.1
Proof
Corollary
5.6.2
Exercise
5.18
Exercise
5.19
Exercise
5.20
Exercise
5.21
Assignment
Probability is the frequency with which the outcome occurs when the experiment is repeated.
\[\P{} A = \Lim{n \to{} \infty} f_n A\]
Also known as Bayesian.
\[\P{} \Sqm{A}{F}\]
No experiment is the same; the generating conditions change.
An experiment is predetermined based on the environment, and has probability 0 or 1.
\[\Vb{A}\]
\[\hash A\]
Finite countable, infinite countable, infinite uncountable
Can be constructed by the Peano axioms.
The ratios of the integers.
Can be constructed as the limits of sequences of rationals.
\[B \setminus{} A = A^\complement{} \setminus{} B\]
Union but intersection removed, \(A \triangle{} B\)
\(\mathcal{P} A\), \(2^A\). Cardinality \(2^{\hash A}\) when \(A\) is finite.
\(\R{} \setminus{} \Q\)
Defined as the union or intersection "limit" of a set sequence. Can similarly be defined for Cartesian products.
\(A_n \subset{} A_{n+1}\)
\(A_n \supset{} A_{n+1}\)
\(A_n \uparrow{} A\) means \(\cup A_n = A\) for increasing \(A_n\).
\(A_n \downarrow{} A\) means \(\cap A_n = A\) for decreasing \(A_n\).
These limits can be started from any finite index.
\(\underset{n}{\operatorname{LimSup}} A_n = \bigcap\bigcup\)
Uniform convergence on compact domain is \(L^1\).
Intuitively a measure assigns a number to things like line segments, areas and volumes.
Weak to functions with unlimited oscillation.
The idea behind the Lebesgue integral is that it is invariant under a change on a set with measure 0.
Probability theory can be seen as a special case of measure theory: Probability function is a measure. Expectation (given a probability function) is an integral wrt. a measure.
σ-algebra of \(\mathcal{E}\) over \(E\)
Let \(\mathcal{E} \subseteq{} \PowsP{} E\). Then \(\mathcal{E}\) is a σ-algebra if it satisfies:
- 1) \(\mathcal{E}\) is nonempty.
- 2) \(\mathcal{E}\) contains the total set \(E\).
- 3) \(\mathcal{E}\) is closed under countable union.
- 4) \(\mathcal{E}\) is closed under complement.
2) replaceable by \(\emptyset\) thanks to 4) through complement.
3) replaceable by countable intersection thanks to complement and de Morgan's law.
Formalizes the idea that pieces of measurable sets each contribe a set "volume".
An element of a sigma algebra.
\(\Br{\emptyset , E}\); The smallest σ-algebra over \(E\).
\(\PowsP{E}\); The largest σ-algebra over \(E\).
\(\sigma (F) = \Br{\emptyset , F, F^c, E}\)
The bigger (or finer) a σ-algebra is, the more sets it is possible to measure.
The smallest sigma algebra containing the subsets. Always exists.
The intersection of σ-algebras over \(E\) is a σ-algebra.
\(\mathcal{B} \mathbb{R}\), Generated by all \([a..b]\).
Collection of all intersections of the interval with all sigma algebra elements.
\(E\) with σ-algebra \(\mathcal{E}\) is a measurable space.
Measure of empty is zero and σ-additivity
Measure of union of partition pieces equals sum of measures of partition peaces.
If not disjoint, the equation is less than or equal.
\(\mathcal{M}_{\Omega , A, \P}\), the measure space of events over outcomes with a probability measure.
A negligible set is a subset of a measurable set with measure zero.
\(f\) is \(\mathcal{E} \to{} \mathcal{F}\)-measurable if \(\forall B \in{} \mathcal{F} f^{-1}B \in{} \mathcal{E}\).
"Every measurable set in \(\mathcal{F}\) was measurable in \(\mathcal{E}\)."
Let \(\mathcal{G}\) generate \(\mathcal{F}\).
Then \(f\) is measurable iff \(\underset{G \in{} \mathcal{G}}\forall{} f^{-1}G \in{} \mathcal{E}\)
"We just need to check if the generating set was measurable."
\(f : \MeasurableSpace{E}{\mathcal{E}} \to{} \R\) is Borel measurable if \(f : \MeasurableSpace{E}{\mathcal{E}} \to{} \MeasurableSpace\R{\mathcal{B} \R}\) is measurable.
\(f : E \to{} \R\) is Borel measurable iff the preimage of every \((\infty ..a]\) is measurable.
Let \(f : E \to{} F\) and \(\mathcal{F}\) a σ-algebra.
\(\sigma (f)\) is the σ-algebra generated by \(f\) on \(E\) which makes \(f\) measurable.
The σ-algebra generated by the preimages of the measurable sets.
Linear combination: \(af + bg\)
Multiplication: \(fg\)
Division: \(\frac{f}{g}\) (\(g \ne{} 0\))
TODO
TODO
Let \(E, F, G\) be measurable spaces, and \(f, g\) measurable where \(\operatorname{Img} f \subseteq{} \operatorname{Dmn} g\).
Then \(g \circ{} f\) is measurable.
A property \(P(x)\) holds almost everywhere if it is only false on a negligible set.
Almost everywhere in the context of a probability space.
A property \(P(x)\) holds almost surely if it is only false within an impossible event.
Let \(E = \MeasureSpace{E}{\mathcal{E}}\mu\), \(F = \MeasurableSpace{F}{\mathcal{F}}\) and \(f : E \to{} F\) a measurable map.
\(\mu^* = \mu{} \circ{} f^{-1}\) is the pushforward measure via \(f\) (Sometimes denoted \(f\hash\mu\)).
Intuitively: each number is multiplied by the "size" of its partition. Then they are summed together.
A non-negative measurable function can be written as the pointwise limit of an increasing sequence of simple functions.
Let \(E = \MeasureSpace{E}{\mathcal{E}}\mu\) and \(f : E \to{} \R\) a non-negative measurable function.
\(\exists\) a sequence of simple functions \(f_\underline{n}\)
A simple function defined on \(n\) contours of \(f\).
\[f_n(x) = \begin{cases} \frac{1-1}{2^n}&: f(x) \in{} [ \frac{1-1}{2^n} .. \frac{1}{2^n} )\\\frac{2-1}{2^n}&: f(x) \in{} [ \frac{2-1}{2^n} .. \frac{2}{2^n} )\\\vdots\\\frac{n2^n-1}{2^n}&: f(x) \in{} [ \frac{n2^n-1}{2^n} .. \frac{n2^n}{2^n} )\\n&: f(x) \geq{} n\\ \end{cases}\]
Let
- The contours of a function \(f\) determines a \(n\)-partition of \(E\).
- The contour values and the partitions determine a simple function that approximates \(f\).
- As \(n \to{} \infty\), the simple function converges pointwise to \(f\).
- Simultaneously, the integral of the simple function converges to a value, which we define to be the integral of \(f\).
\(µ\)-integrable function on \(A\)
A measurable function \(f\) is \(µ\)-integrable if \(\int_E f^+ dµ < \infty\) and \(\int_E f^- dµ < \infty\).
Integral of measurable function with respect to \(\mu\)
Let \(f : E \to{} \R\) be a measurable function. \(f = f^+ - f^-\).
\[\int_E f d\mu{} = \int_E f^+ d\mu{} - \int_E f^- d\mu\]
Let \(f, g\) be \(\mu\)-integrable functions.
- \[\Vb{\int_E d\mu} \leq{} \int_E \Vb{f} d\mu\]
- d
d
Let \(f_n\) be an increasing non-negative sequence of measurable functions and \(f = \LimSup{} f_n\).
\[\int_E f_n d\mu{} \to{} \int_E f d\mu\]
As \(n \to{} \infty\), a sequence converges, diverges to \(\pm\infty\) or oscillates.
We can look at \(\LimInf\) and \(\LimSup\) to determine this.
Fatou's lemmas are relevant to a function sequence \(f_n\) that oscillates as \(n \to{} \infty\).
It is possible that
Note that if \(f_n\) converges, this cannot happen because \(\LimSup{} = \Lim{} = \LimInf\) and we get the monotone convergence theorem instead.
Lebesgue-Stieltjes integral of \(f\) on \(A\) with respect to Lebesgue-Stieltjes measure \(\mu_g\)
\(\nu\) is absolutely continuous with respect to \(\mu\), \(\nu{} \ll{} \mu\)
\[\mu{} \ll{} \nu{} \iff{} \Pa{\nu (A) = 0 \impliedby{} \mu (A) = 0}\]
"\(\nu\) measures at least the zeroes of \(\mu\)."
\(\mu\) and \(\nu\) are equivalent if \(\mu{} \ll{} \nu\) and \(\nu{} \ll{} \mu\).
Ran
The Radon-Nikodym derivative is the function \(f\) in the Radon-Nikodym theorem.
Notation: \(f = \frac{d\nu}{d\mu}\).
σ-additivity is equivalent to the probability of increasing and decreasing set sequences converging to the probability.
...
In a probability space, a sequence of measurable sets corresponds to a sequence of events.
\(\LimSup{} A_n\) can be seen as the collection of outcomes that occur in infinitely many of the \(A_n\). It is also the event that infinitely many of the \(A_n\) occur.
\(\LimSup{} A_n\) can be seen as the collection of outcomes that occur in every \(A_n\) except a finite subset. It can also be seen as the event that infinitely many of the events in \(A_n\) occur.
Tells us about the probability of the event \(\LimSup{} A_n\), given the probabilities of \(A_n\).
The first Borel-Cantelli lemma says that
The Borel-Cantelli lemmas relates the probability of \(F_i\) to the probability of this \(\LimSup{} F_i\).
...
Let \(A_\underline{n}\) be a sequence of events and \(\sum_n^{1:}\infty\\ \P{} A_n\) is finite.
\[\P{} \LimSup{} A_n = 0\]
A random variable is a map \(Ω \to{} \R\) from a probability space to \(\R\) where \({ω \in{} Ω : X(ω) \le{} x}\) is an event. "The collection of outcomes assigned to \(x\) or below is an event."
Alternatively, \(X^{-1}(-\infty ..x]\). Recall that this is a generator of all the Borel sets.
A random variable is in reality a function. However, it is very common to omit any mentions of \(Ω\) and \(ω\) and treat them as implicit. Doing so, \(X = X(ω)\) looks like a variable.
The fact that \(X^{-1}(-\infty ..x]\) is an event means that \(X\) is a measurable map. This gives us a pushforward measure of \(\P\) on \(\R\).
A random variable but with a range \(\R{} \cup{} \{-\infty , \infty \}\).
A random vector of dimension \(n\) is a function \(X : Ω \to{} \R^n\) with \(X(ω) = [X_1(ω), \dots , X_n(ω)]\).
A stochastic process is a collection \(X_\underline{n}\) of random variables.
The law of a random variable \(X\) is the pushforward measure of \(\P\) that acts on Borel sets.
\(μ_X = \P{} \circ{} X^{-1}\).
The distribution of a random variable is the function \(\R{} \to{} [0..1]\) of a real number \(x\) that evaluates the law measure for the Borel set \((-\infty ..x]\).
The distribution of a random variable is the function \(F_X\) defined by \(F_X (x) = μ_X (-\infty ..x] = \P [ X \le{} x ]\)
The law and the CDF of a random variable are equivalent. Given one, the other can be constructed.
The distribution function \(F_X\) of \(X\) satisfies:
- \(\Lim{x \to{} -\infty} F_X (x) = 0\) and \(\Lim{x \to{} \infty} F_X (x) = 1\)
- \(F_X\) is an increasing function (not necessarily strictly).
- \(F_X\) is right-continuous. (because of the \(\le\))
A function satisfying the CDF properties can serve as a CDF. This will then induce a law and a probability measure on \(Ω\).
\(F_X\) is a step function, or equivalently, the image of \(F_X\) is countable.
\(X\) is absolutely continuous if \(F_X\) can be written as \(F_X (x) = \int_{-\infty}^x f_X (y) dy\) for a non-negative integrable function \(f_X\).
The density function of \(X\) is the function \(f_X\) if it exists in the definition of absolutely continuous random variable above.
The integral of the density on \(\R\) must be \(1\). This follows from the fact that \(\Lim{x \to{} \infty} F_X (x) = 1\).
By Lebesgue's differentiation theorem, \(F'_X (x) = f_X (x)\) and \(f_X\) is unique almost everywhere.
Let \(Ω = \MeasureSpace{Ω}{\mathcal{A}}\P\) be a probability space and \(X\) a random variable.
The expectation of \(X\) is \(\mathbb{E} [X] = \int_Ω X d\P\).
\[\E{} [g(X)] = \int_\R{} g dF_X\]
The integral is a Riemann-Stieltjes or Lebesgue-Stieltjes integral with respect to the increasing function \(F_X\), which induces a Lebesgue-Stieltjes measure.
When \(F_X\) has a non-zero derivative almost everywhere, then \(\E{} [g(X)] = \int_\R{} g(x) f_X (x) dx\)