I'm currently in the process of writing an application related to poker, and I've been struggling to determine the full formula for a piece of this. I need to determine the number of ways to combine x cards (ignoring suit) where the best 5 card hand made from that combination is not a straight or high card hand. First piece. f(x, y) is r-com with repetition of x possible values for y cards, g(x, y) is r-com without repetition, i is the number of cards in a hand. Here's where I'm at so far:
f(13, i) - g(13, i) - f(13, i-5)
So that's all combinations of cards where at least one card is repeated at least once, but no card is repeated at least more than 4 times. However, I forgot to account for the fact that at 6 cards or more, it's possible to have both a straight and a pair. So I need to subtract those. Here's what I've got so far for that:
9*(f(12, i-5) - g(7, i-5)) + (f(13, i-5) - g(8, i-5))
However, this number also doesn't work for 8 or more cards, as the second part of the formula includes cases where you have both a straight and a full house, four of a kind, or an illegal hand (5 or more of a card). And that's where I'm stuck. Any help would be very much appreciated!