Research
problem:
Bayesian inference with automatic model order selection
Brief Description:
Model order selection is a fundamental problem in signal processing and machine learning. Choosing too small the model size cannot fit the data well, while choosing too large the model size leads to overfitting. Traditionally, a regularization term is added to the loss function to strike a balance between data fitting and the model complexity, with the regularization parameter tuned to obtain the best performance. However, the ``best'' parameter varies widely across datasets and applications, and may not be effective for unseen data.
Sparse Bayesian
learning provides a tuning-free alternative. The idea is that if a suitable
prior distribution is specified, the relative importance of different
components in the signal can be learnt. Together with the learnt noise power
from the data, one can easily tell what is the proper model order. While the
idea is simple, the algorithm derivation is usually not straightforward. In
particular, in Bayesian
statistics, if we want to estimate a certain parameter, other parameters should
be integrated out from the joint posterior distribution. Unfortunately,
most of the time, we cannot perform the integration analytically due to the
complicated nature of the posterior distribution. To handle this problem,
previous Bayesian analysis is mainly based on Monte Carlo statistical methods,
such as Markov chain Monte Carlo (MCMC) and Gibbs sampling, where a large
number of random samples are generated from the joint distributions and
marginalization is approximated by operations on samples. Although sampling
methods can approach the true posteriors when the number of samples approaches
infinity, it is computationally demanding.
Variational inference is another way to
approximate the parameter inference. It seeks a variational distribution that
closely approximates the true posterior distribution. Under the commonly used
mean-field constraint, the variational distribution will be in a form where
marginalization can be easily carried out. This saves the computational
complexity significantly, as each update is in closed-form.
We have applied the variational inference with
automatic model order selection to various signal processing problems. The
first one is the iterative joint doubly-selective channel estimation and data
detection in wireless communication systems, where the model order corresponds
to the channel length and Doppler shift. The second one is the distributed
estimation of system state in power grid. The third one is the subspace
identification for channel estimation in massive MIMO systems, where the model
order is the number of paths the signal takes. We have further adopted
variational inference to various tensor decompositions, where the model order
is the unknown tensor rank. This latest framework finds widespread applications
in blind CDMA receiver design, face image classification, object tracking in
surveillance video, fluorescence data analysis, email data mining, and radio
map construction.
Recently, we have
ventured into approximate message passing (AMP). AMP is another sparsity
enhancing algorithm but based on a different prior distribution from that used
in variational inference. While AMP is popular, it is actually quite
complicated if one aims to fully understand its logic and analysis, especially
most applications of AMP seem to come with some modifications. Our two related
works demonstrate how AMP should be used within the expectation-maximization
(EM) framework, thus ensuring the legitimacy of Bayesian inference.
Data detection in doubly-selective channels:
1. Jingrong Zhou, Jiayin Qin and Yik-Chung Wu, ``Variational Inference-based Joint Interference Mitigation and OFDM Equalization under High Mobility,'' IEEE Signal Processing Letters, Vol. 22, no. 11, pp. 1970 - 1974, Nov 2015.
2.
Ke Zhong, Yik-Chung Wu, and Shaoqian Li,
``Signal Detection for OFDM-Based Virtual MIMO Systems under Unknown Doubly
Selective Channels, Multiple Interferences and Phase Noises," IEEE Trans. onWireless
Communications, Vol. 12, no. 10, pp.5309-5321, Oct
2013.
3. Lanlan He, Yik-Chung Wu, Shaodan Ma, Tung-Sang Ng and H. Vincent Poor, ``Superimposed Training Based Channel Estimation and Data Detection for OFDM Amplify-and-Forward Cooperative Systems under High Mobility," IEEE Trans. on Signal Processing, Vol. 60, no. 1, pp. 274-284, Jan 2012.
4. Lanlan He, Shaodan Ma, Yik-Chung Wu, Yiqing Zhou, Tung-Sang Ng, and H. Vincent Poor, ``Pilot-Aided IQ Imbalance Compensation for OFDM Systems Operating over Doubly Selective Channels," IEEE Trans. on Signal Processing, Vol. 59, no. 5, pp. 2223-2233, May 2011.
Power system state estimation:
Channel estimation in massive MIMO systems:
6. Hao Zhang, Qingfeng Lin, Yang Li, Lei
Cheng, and Yik-Chung Wu, ``Activity Detection for Massive Connectivity
in Cell-free Networks with Unknown Large-scale Fading, Channel Statistics,
Noise Variance, and Activity Probability: A Bayesian Approach," in IEEE
Transactions on Signal Processing, vol. 72, pp. 942-957, 2024, doi:
10.1109/TSP.2024.3361090
7. Le Xu, Lei Cheng, Ngai Wong, Yik-Chung
Wu, and H. Vincent Poor, ``Overcoming Beam Squint in Dual-Wideband mmWave
MIMO Channel Estimation: A Bayesian Multi-Band Sparsity Approach," in IEEE
Transactions on Signal Processing, vol. 72, pp. 1219-1234, 2024, doi:
10.1109/TSP.2024.3368770 https://arxiv.org/pdf/2306.11149.pdf
8. Lei Cheng, Chengwen Xing, and Yik-Chung
Wu, ``Irregular Array Manifold Aided Channel Estimation in Massive
MIMO Communications," IEEE Journal of Selected Topics in Signal
Processing, Vol. 13, no. 5, pp. 974-988, Sep 2019.
9. Lei Cheng, Yik-Chung Wu,
Jianzhong (Charlie) Zhang, and Lingjia Liu, ``Subspace Identification for DOA
Estimation in Massive / Full-dimension MIMO System: Bad Data
Mitigation and Automatic Source Enumeration,'' IEEE
Trans. on Signal Processing, Vol. 63, no. 22, pp.
5897-5909, Nov 2015.
Tensor Canonical
Polyadic Decomposition:
10. Lei Cheng, Zhongtao Chen, Qingjiang
Shi, Yik-Chung Wu, and Sergios
Theodoridis, ``Towards Flexible
Sparsity-Aware Modeling: Automatic Tensor Rank Learning using The Generalized
Hyperbolic Prior," IEEE Trans. on
Signal Processing, vol.
70, pp. 1834-1849, 2022, doi: 10.1109/TSP.2022.3164200.
11. Zhongtao Chen, Lei Cheng, and Yik-Chung Wu, ``Accelerating Probabilistic
Tensor Canonical Polyadic Decomposition with Nonnegative Factors: An Inexact
BCD Approach," Signal Processing,
vol. 207, 2023, https://doi.org/10.1016/j.sigpro.2023.108966
12. Lei Cheng, Xueke Tong, Shuai Wang, Yik-Chung
Wu, and H. Vincent Poor, ``Learning Nonnegative
Factors from Tensor Data: Probabilistic Modeling and Inference Algorithm,"
in IEEE
Trans. on Signal Processing, vol. 68, pp. 1792-1806, 2020, doi:
10.1109/TSP.2020.2975353.
13. Lei Cheng, Yik-Chung Wu, and H. Vincent Poor, ``Scaling Probabilistic Tensor Canonical
Polyadic Decomposition to Massive Data," IEEE Trans. on Signal Processing, Vol. 66, no. 21, pp. 5534-5548, Nov 2018.
14.
Lei
Cheng, Yik-Chung Wu, and H. Vincent Poor,
``Probabilistic Tensor Canonical Polyadic Decomposition with Orthogonal
Factors," IEEE Trans.
on Signal Processing, Vol. 65, no. 3, pp. 663-676, Feb 2017.
Tensor Train / Tucker / Block-Term Decompositions:
15. Le Xu, Lei Cheng, Ngai Wong, and Yik-Chung
Wu, ``To Fold or not to Fold: Graph Regularized Tensor Train for Visual
Data Completion," IEEE Transactions on Pattern Analysis and Machine
Intelligence, vol. 48, no. 2, pp. 1437-1455, Feb. 2026, doi: 10.1109/TPAMI.2025.3615445
16. Le Xu, Lei Cheng, Ngai Wong, and Yik-Chung Wu, ``Tensor Train Factorization under
Noisy and Incomplete Data with Automatic Rank Estimation," Pattern Recognition, vol. 141,
2023. https://doi.org/10.1016/j.patcog.2023.109650
17. Xueke
Tong, Lei Cheng and Yik-Chung Wu, "Bayesian Tensor Tucker
Completion with a Flexible Core," in IEEE Trans. on Signal
Processing, vol. 71, pp. 4077-4091, 2023, doi:
10.1109/TSP.2023.3327845
18.
Zhongtao Chen, Lei Cheng, Yik-Chung Wu, and H. Vincent
Poor, ``Rank-Revealing Bayesian Block-Term Tensor Completion with Graph
Information," to appear in IEEE Trans. on Signal Processing.
Matrix completion with graph information:
19. Yangge Chen, Lei Cheng, and Yik-Chung Wu, ``Bayesian Low-rank Matrix Completion with Dual-graph Embedding: Prior Analysis and Tuning-free Inference," Signal Processing. volume 204, 2023, https://doi.org/10.1016/j.sigpro.2022.108826
AMP application in wireless communication users activity detection:
20. Hao Zhang, Yang Li, Qingfeng Lin, Yik-Chung Wu, and H. Vincent Poor, ``Harnessing Common Sparsity for Enhancing AMP-based Activity Detection and Channel Estimation," IEEE Trans. on Signal Processing, vol. 73, pp. 3220-3236, 2025, doi: 10.1109/TSP.2025.3585424
21. Hao Zhang, Qingfeng Lin, Yang Li, and Yik-Chung Wu, ``AMP-based Joint Activity Detection and Channel Estimation in IRS-aided Grant-free Access with Accurate Channel and Sparsity Modeling," accepted in IEEE Trans. on Communications.
The idea and algorithms of Bayesian
signal processing have been summarized in a recent book:
Some of the codes are available in
the following webpage:
https://github.com/leicheng-tensor/Reproducible-Bayesian-Tensor-Matrix-Machine-Learning-SOTA