1
$\begingroup$

I am using this weighted sum from the question How to build a function that gives recent years higher weight?

To reduce the value I tried square root of $X,$ then tried the $8$th root, $\sqrt{\sqrt{\sqrt{X}}}$ which is $X^{\frac{1}{8}}$ and get better results.

My question is how can I justify this use? Can I know why I got better results when using the $8$th root or is it just random, depend on data only? And how can I know if there is a better way of doing it?

  • 0
    …then you could claim it made no sense if the assumptions have a meaningful basis in reality. And if you are trying to optimize something, there are of course sub-optimal models.2012-08-06

1 Answers 1

1

There is extensive literature on data transformations, specifically to lower the variance of the residuals contained in the data, fit to a given model.

I'm not certain if it is exactly what you need, but the square root transformation is a well-known variance stabilizer.

From a cursory review of your data, I think you might be able to look into various types of regression analysis, and that would lead you into all sorts of ways to model your data, all of which have rich and rigorous justifications for such transformations.

Even more simply, you could use an exponentially-decaying rate to arrive at your weights.

The point to remember is that with your current methodology, you are just shrinking the values by multiplying them by smaller and smaller numbers. If you wanted a really small number you could use $x^{1/100}$ but you are correctly deducing that there isn't much of a justifiable reason (mathematically) to do so.

Keep experimenting with things...for one, I would try weighting each score by a factor of $x^{\frac1t}$ where t is the number of years ago. For example

Year 1: 10 movies, 5 comedies Year 2: 8 movies, 2 comedies Year 3: 10 movies, 7 comedies

take your sample proportions (Year 1= .5, Year 2=.25 and Year 3=.7) and apply the following weights:

Year 1= $(.5)^{\frac13}$

Year 2= $(.25)^{\frac12}$

Year 3= $(.7)^{\frac11}$

This would down-weight past years in a more "tidy" way.

  • 0
    Most likely your best bet is just to Google "variance stabilizing transformations". A good start would be here: http://www.stat.ncsu.edu/people/bloomfield/courses/st762/md-02-2.pdf ... Also, you can often times use trig functions in transformations, like sin/cos, etc. These are obviously bounded.2012-08-07