Temporal Matrix Factorization
[Question A] For any partially observed data matrix 
with the observed index set 
, suppose )
is the orthogonal projection supported on , then the matrix factorization (i.e., 
are the rank-
latent factor matrices) involving temporal modeling on latent temporal factors (i.e., 
) can be formulated as follows,
\|_{F}^{2}+\frac{\lambda}{2}\|\boldsymbol{X}\boldsymbol{\Psi}_0^\top-\boldsymbol{A}(\boldsymbol{I}_d\otimes\boldsymbol{X})\boldsymbol{\Psi}^\top\|_{F}^{2},)
where 
is the coefficient matrix. The symbol 
denotes Kronecker product.
If \times T})
and \times dT})
are known, then how to optimize the variables 
? Do these variables have any closed-form solutions?
Note that
\|_{F}^{2}=\frac{1}{2}\sum_{(i,t)\in\Omega}\left(y_{it}-\boldsymbol{w}_{i}^\top\boldsymbol{x}_{t}\right)^{2},)
where 
is the 
th column of 
, while 
is the 
th column of .
Is it possible to use alternating least squares (ALS) method to solve the optimization problem?
[Note] Let 
be the objective, the first-order partial derivatives are:
- with respect to :
)
- with respect to :
 \\ &+\lambda\left(\boldsymbol{X}\boldsymbol{\Psi}_{0}^\top-\boldsymbol{A}(\boldsymbol{I}_{d}\otimes\boldsymbol{X})\boldsymbol{\Psi}^\top\right)\boldsymbol{\Psi}_0 \\ &-\lambda\sum_{k=1}^{d}\boldsymbol{A}_{k}^\top\left(\boldsymbol{X}\boldsymbol{\Psi}_{0}^\top-\boldsymbol{A}(\boldsymbol{I}_{d}\otimes\boldsymbol{X})\boldsymbol{\Psi}^\top\right)\boldsymbol{\Psi}_{k} \\ \end{aligned})
Let 
, then we have a vectorized closed-form solution to as
=\left(\boldsymbol{S}+\lambda\sum_{k=0}^{d}\sum_{h=0}^{d}(\boldsymbol{\Psi}_{k}\otimes\boldsymbol{A}_{k})^\top(\boldsymbol{\Psi}_{h}\otimes\boldsymbol{A}_{h})\right)^{-1}\text{vec}(\boldsymbol{W}\mathcal{P}_{\Omega}(\boldsymbol{Y})))
where 
, and we have
\boldsymbol{\Psi}^\top\right)\boldsymbol{\Psi}_{k}\right) \\ =&\lambda\sum_{k=0}^{d}\sum_{h=0}^{d}(\boldsymbol{\Psi}_k\otimes\boldsymbol{A}_k)^\top(\boldsymbol{\Psi}_h\otimes\boldsymbol{A}_h)\text{vec}(\boldsymbol{X}) \\ \end{aligned})
In this case, we also define a block matrix
\times (RT)})
with the following sub-matrix
\in\Omega}\boldsymbol{w}_{i}\boldsymbol{w}_{i}^\top\in\mathbb{R}^{R\times R},\forall t\in\{1,\ldots,T\})
- with respect to
:
\boldsymbol{\Psi}^\top)\boldsymbol{\Psi}(\boldsymbol{I}_{d}\otimes\boldsymbol{X})^\top)
where 
.
Here, we have a sequence of matrices:
\times d} & \boldsymbol{I}_{T-d} \end{bmatrix})
\times (d-k)} & \boldsymbol{I}_{T-d} & \boldsymbol{0}_{(T-d)\times k} \end{bmatrix},\forall k\in\{1,\ldots,d\})
[Question B] How do the following two models perform?
\|_{F}^{2}+\frac{\rho}{2}(\|\boldsymbol{W}\|_{*}^{2}+\|\boldsymbol{X}\|_{*}^{2}))
vs.
\|_{F}^{2}+\frac{\rho}{2}(\|\boldsymbol{W}\|_{F}^{2}+\|\boldsymbol{X}\|_{F}^{2}))
Which is better?
References
[Question C] What is the difference between
\|_{F}^{2}+\frac{\lambda}{2}\|\boldsymbol{X}\boldsymbol{\Psi}_0^\top-\boldsymbol{A}(\boldsymbol{I}_d\otimes\boldsymbol{X})\boldsymbol{\Psi}^\top\|_{F}^{2})
and
\|_{F}^{2}+\frac{\lambda}{2}\sum_{t=d+1}^{T}\left\|\boldsymbol{x}_t-\sum_{k=1}^{d}\boldsymbol{A}_{k}\boldsymbol{x}_{t-k}\right\|_{F}^{2})
[Note] For 
, the first-order partial derivative is given by
 \\ &+\lambda\boldsymbol{A}_{0}^\top\sum_{k=0}^{d}\boldsymbol{A}_{k}(\boldsymbol{x}_{t-k}-\boldsymbol{x}_{t-1-k}) \\ &+\lambda(\boldsymbol{A}_{1}-\boldsymbol{A}_{0})^\top\sum_{k=0}^{d}\boldsymbol{A}_{k}(\boldsymbol{x}_{t+1-k}-\boldsymbol{x}_{t-k}) \\ &\cdots \\ &+\lambda(\boldsymbol{A}_{d}-\boldsymbol{A}_{d-1})^\top\sum_{k=0}^{d}\boldsymbol{A}_{k}(\boldsymbol{x}_{t+d-k}-\boldsymbol{x}_{t+d-1-k}) \\ &+\lambda(-\boldsymbol{A}_{d})^\top\sum_{k=0}^{d}\boldsymbol{A}_{k}(\boldsymbol{x}_{t+d+1-k}-\boldsymbol{x}_{t+d-k}) \\ \end{aligned})
