0
$\begingroup$

Let $A,B\in\mathbb{R}^{m\times n}$ be matrices with all positive entries. I want to compute the following minimum. $$\min_{\vec{u}\in\mathbb{R}^m,\ \ \vec{v}\in\mathbb{R}^n} \ \ \sum_{i=1}^m \sum_{j=1}^n (u_i v_j A_{ij} - B_{ij})^2.$$

Rephrased in terms of matrix notation and the Frobenius norm $\|\cdot\|_F$, this would be the following. $$\min_{\vec{u}\in\mathbb{R}^m,\ \ \vec{v}\in\mathbb{R}^n} \ \ \| D_{\vec{u}} A D_{\vec{v}} - B\|_F^2,$$ where $D_{\vec{u}}$ is the diagonal matrix with diagonal $\vec{u}$ (and likewise for $D_{\vec{v}}$). Does anyone know how to find this minimum?

  • 0
    A duplicate question on a different stackexchange: http://scicomp.stackexchange.com/questions/1632/weighted-svd-problem – 2017-04-04

3 Answers 3

0

After much searching, I have found that this can be expressed as a weighted low rank approximation problem. We want to approximate the matrix $B\oslash A$ with a rank-1 matrix that minimizes the weighted Frobenius norm, with weight matrix $W$ such that $W_{ij}=A_{ij}^2$.

According to the wikipedia page, there is no analytic solution in terms of the SVD. Instead, it requires an iterative optimization method. One such method that I came up with myself (and many others have also invented) is below.

Repeat the following steps until the desired accuracy is reached (convergence is linear). $$u_i\leftarrow \frac{\sum_j v_j A_{ij} B_{ij}}{\sum_j v_j^2 A_{ij}^2},\qquad u\leftarrow u/\|u\|,\qquad v\leftarrow \frac{\sum_i u_i A_{ij} B_{ij}}{\sum_i u_i^2 A_{ij}^2}.$$

2

Let $X=uv^T$. Then what you are asking for is the least-squares solution to $$A\odot X = B$$ The solution to the unconstrained problem is $$X=B\oslash A$$ where $(\odot, \oslash)$ represent Hadamard multiplication and division.

And the closest rank-1 matrix $\,X_c\,$ can be obtained via the SVD $$\eqalign{ B\oslash A &= USV^T = \sum_{k=1}^{r} \sigma_k u_kv_k^T \cr S &= {\rm Diag}(\sigma_1,\,\sigma_2,\,\ldots\,\sigma_r) \cr &\,\,\,\,\,\,{\sigma_1\geq\sigma_2\geq\ldots\geq\sigma_r\gt 0} \cr\cr X_c &= \sigma_1 u_1v_1^T \cr\cr }$$

Update

After reading noumenon28's counterexample in the comments, it is interesting to note that the direction of the above $X_c$ is okay, but the magnitude is wrong. But finding the correct magnitude is easy.

Solve the scalar sub-problem $$\eqalign{ \min_\lambda \| \lambda u_1v_1^T\odot A-B \|_F^2 \cr }$$ omit the details and jump to the solution $$\eqalign{ \lambda &= \frac{(u_1v_1^T\odot A):B}{(u_1v_1^T\odot A):(u_1v_1^T\odot A)} \cr }$$ where colon denotes the inner (Frobenius) product, i.e. $\,\,A:B={\rm tr}(A^TB)$.

This yields a closest rank-1 solution of $$\eqalign{ X_c = \lambda u_1v_1^T }$$

  • 0
    The problem with that method is that the minimum of $\|uv^T-B\oslash A\|_F^2$ is not the same as the minimum of $\|uv^T\odot A-B\|_F^2$. As a counterexample: $$A=\begin{bmatrix}2&1\\1&2\end{bmatrix},\qquad B=\begin{bmatrix}1&1\\1&1\end{bmatrix}.$$ Then the SVD of $B\oslash A$ is $\sigma_1=1.5$, $u_1=v_1=[-\sqrt{2}/2, -\sqrt{2}/2]^T$, and $\|\sigma_1u_1v_1^T\odot A-B\|_F^2=5/8$. However, if we choose $u=[1,1]^T$ and $v=[3/5, 3/5]^T$, we get $\|uv^T\odot A-B\|_F^2=2/5$, so your method is not optimal. – 2017-03-29
  • 0
    Counterexample to your update: $$A=\begin{bmatrix}1&2\\1&1\end{bmatrix},\qquad B=\begin{bmatrix}1&1\\1&1\end{bmatrix}.$$ Then the SVD of $B\oslash A$ is: $$u_1 = \begin{bmatrix}3+\sqrt{17}\\5+\sqrt{17}\end{bmatrix},\qquad v_1 = \begin{bmatrix}1+\sqrt{17}\\4\end{bmatrix}.$$ Then $\lambda = 0.0218$ and $\|\lambda u_1 v_1^T\odot A-B\|_F^2\approx 0.1427$. However, if we choose $u=[1,\sqrt{2}/2]^T$ and $v=[0.8047,0.5690]^T$ we get $\|u v^T\odot A-B\|_F^2\approx 0.1144$. – 2017-04-04
  • 0
    correction: $u = [1, \sqrt{2}]^T$. – 2017-04-04
0

This is the comment for greg's answer. (I couldn't use the comment function since I'm newcomer and don't have enough reputation to use the function yet.)

I'm doubtful that $u_1v_1^{\top}$ always provides the correct direction. I would like to know why it can be said so.

I implemented greg's method as well as numerical gradient descent method, and observed the residual obtained by the former method often larger than that obtained by the latter when tested with random A and B with positive entries.

For noumenon28's counterexample, greg's method gives optimal (at least the same result as that of the numerical method).

I'm interested whether there are any moderate conditions where greg's method provides optimal. I also tested with symmetric A and B, but still optimal was not obtained.

See the following link for the code. https://github.com/HiroshiKERA/playground/blob/master/hadamard-lstsq-test.py