F  Inbreeding coefficient = prob. that two alleles are identical by descent (autozygous)

[Note than alleles can be identical by allelic state without being identical by descent]

F is also the proportion of the population that is inbred (has two alleles identical by descent).

What is the effect of inbreeding on genotype proportions?

In the absence of inbreeding, expected f(AA) = p²
                                                                    f(AB) = 2pq
                                                                    f(BB) = q²
In the presence of inbreeding,

f(AA) = (1-F)(p²)  +  (F)(p)(1)  =  p² - Fp² + Fp  =  p² + Fp(1-p)  =  p² + Fpq
       [A fraction (1-F) of the population is not inbred:
          the expected frequency of AA homozygotes among these is p²
        A fraction (F) of the population is inbred:
          a fraction p of these individuals have one A allele.
          Since they are inbred, the other allele must also be allele A with a probability = 1]
 

f(AB) = (1-F)(2pq) + (F)(0)     =  2pq - 2Fpq      =  2pq (1 - F)
   [A fraction (1-F) of the population is not inbred:
       the expected frequency of AB heterozygotes among these is 2pq.
    A fraction (F) of the population is inbred:
       among these, none can be heterozygotes, since both alleles are identical.]

f(BB) = (1-F)(q²) + (F)(q)(1)  =  q² - Fq² + Fq  =  q² + Fpq
    [Same logic as for f(AA) above, applied to B allele]