If I understand your problem hypotheses correctly, it is a modification of an M/M/1/K queueing system. What do I mean with this? It is a specific continuous-time Markov process on the state space $\{0,1,\dots,K\}$, whose transition rates are specified by a certain generator matrix $Q$. Being in state $n$ means that there are $n$ packets in the systems and the oldest of them is currently being served/processed.
Why the name M/M/1/K? The first M stands for memoryless arrival times for the packets (thus described by a Poisson process), the second M stands for memoryless service times (i.e. exponentially distributed), the number 1 means that there is only one server handling the packets and the number K is the maximum size of the queue (above which the new arriving packets are simply rejected).
The modification comes from the fact that you have a certain probability $p_1$ of rejecting a new packet, but it is well know that the probabilistic thinning (with probability $p$) of a Poisson process with rate $\lambda$ is again a Poisson process with rate $\lambda p$. Therefore, in our case, the arrival process is Poisson with rate $(1-p_1)\lambda$.
With these assumptions, the transition rates of this process are
$q_{0,1}=(1-p_1)\lambda$,
$q_{n,n+1}=(1-p_1) \lambda$ and $q_{n,n-1}=\mu$ for $1 \leq n \leq K-1$,
$q_{K,K-1}=\mu$.
Usually the diagonal elements of the generator matrix $Q$ are set such that every row sums to zero.
One can then uniformize the process, in order to get a discrete-time Markov chain: this can be done simply by defining its probability transition matrix as $P= Q / q^*+I$, where $q^*:=\max_i |q_{i,i}|$. Otherwise one can simply compute from the generator Q the probabilities of going back or forth, given that a jump has occurred, i.e. the matrix $\hat{P}$, obtained by dividing each row of $Q$ by the opposite of its diagonal element.
The only differences that I see between this model and yours is that you do not assume the whole system to be Markovian (or was this implicit?) and the fact that apparently in your model packets are not always rejected when the queue is full (i.e. $K$), but they are accepted with probability $1-p_K$ (but where do you put them, if the capacity is only $K$?).