Let us calculate the number of parameters for bi-gram hMM given as
Let be the total number of states
and
be the vocabulary size and
be the length of the sequence
- Before directly estimating the number of parameters, let us first try to see what are the different probabilities or rather probability matrix we have.
- Once we know the probability matrix, we can estimate the parameters by its size.
- If you’re thinking how does a probability matrix appear, notice that we have conditional probabilities here,
and
or probability expression involving 2 variables
and
such as
, hence we should have the probability matrix.
- So for
we have a probability matrix where each row
is a state and each column is an output variable
. Hence this matrix is of size
leading to these many parameters.
- Similarly for
, we have
x
matrix and hence same number of parameters.
- From (d) and (e), we have at least
parameters.
- Now think carefully if we there is anything which we missed in our calculation.
- We also have start tokens
and initial state
. In State Probability matrix explained in (e), we have one more row but columns are still N. Therefore number of parameters due to initial state
becomes
- Once you’re convinced, you’ll get to know that total number of parameters in the above hMM model are
- Give a try for a general
for n-gram hMM!