Estimating Weibull Distribution Parameters from Censored Data with Variable Censoring Threshold

Question

Goal

Evaluate the shape and scale parameters which describes a given set of Type I censored data points.

Problem

The data was collected in such a manner that the censoring threshold for each censored data point is different. Here is a sample data set to illustrate:

start_time   fail_time       life
          0  11.297102  11.297102
          2         NA         NA
          2   9.934173   7.934173
          2   8.441901   6.441901
          4  16.651268  12.651268
          4  10.277692   6.277692
          4         NA         NA
          5  10.728138   5.728138
          6  14.069520   8.069520
          8  14.630614   6.630614
         13         NA         NA
         13  20.504782   7.504782
         14         NA         NA
         17         NA         NA
         18  19.095108   1.095108
         21  -EXPERIMENT ENDS-

In we want to find a Weibull distribution that describes the behavior of our 'life' variable. Because the experiment ended at time 21, we do not have failure times (and thus 'life' values) for all items.

When calculating the parameters for this distribution, we need to set a censoring threshold for those missing points such that when they're estimated via the Weibull distribution's maximum-likelihood function, they're assumed to be at least as old as they where at the time the experiment ended.

If all the items were started at the same time (t=0) and ran until the experiment ended (t=21), this would be relatively easy. Working in R we would just use this line of code:

cenWbMLE.T1(life, Cx=21)  # part of the extWeibQuant package in R

And it would return parameters for the distribution. But that function only accepts one censoring threshold (Cx).

Is there a way to estimate the weibull parameters of this censored data set in such a way that the different start times are accounted for? If so, how would it be done/coded?

Answer 1

The Weibull PDF is $$f_X(x) = \frac{k}{\lambda} \left(\frac{x}{\lambda}\right)^{k-1} e^{-(x/\lambda)^k}, \quad x > 0$$ and its survival function is $$S_X(x) = \Pr[X > x] = e^{-(x/\lambda)^k}.$$ Then the likelihood of a set of $n$ failure times $\boldsymbol x = (x_1, \ldots, x_n)$ and a set of $m$ observations whose failure times are censored at $\boldsymbol c = (c_1, \ldots, c_m)$ is simply $$\mathcal L(\lambda, k \mid \boldsymbol x, \boldsymbol c) = \prod_{i=1}^n f_X(x_i) \prod_{j=1}^m S_X(c_j).$$ The log-likelihood is $$\ell(\lambda, k \mid \boldsymbol x, \boldsymbol c) = n \log k - nk \log \lambda + (k-1) \sum_{i=1}^n \log x_i - \frac{n \overline {x^k} + m \overline{c^k}}{\lambda^k}$$ where $\overline{x^k} = \frac{1}{n}\sum_{i=1}^n x_i^k$ and similarly for $\overline{c^k}$. Even when there are no censored observations, ML estimation of $k$ generally requires numeric methods. Does ML estimation of $\lambda$ given $k$ in our case have a closed form? We search for critical points: $$\frac{\partial \ell(\lambda \mid k, \boldsymbol x, \boldsymbol c)}{\partial \lambda} = -\frac{nk}{\lambda} + k \frac{n\overline{x^k} + m \overline{c^k}}{\lambda^{k+1}} = 0,$$ or equivalently, $$\lambda^k = \overline{x^k} + \frac{m}{n} \overline{c^k},$$ hence $$\hat\lambda \mid k = \left(\overline{x^k} + \frac{m}{n} \overline{c^k}\right)^{1/k}.$$ This is consistent with the ML estimate when $\boldsymbol c$ is the null set.

For ML estimation of $k$, a closed form solution is not tractable. As to your question regarding whether the data can be reparametrized, I don't know enough about this R package to see if there's another way, but if there is only one time $c$ after which all failure times are censored, then you can't manipulate the data to fit such a likelihood model, since such a likelihood has the form $$(S_X(c))^m \prod_{i=1}^n f_X(x_i).$$ This is too restrictive for your data.