About Me

I am a research associate at Department of Electrical Engineering (ESAT-STADIUS) of KU Leuven. I have been fortunate to work with Prof. Johan A.K. Suykens for my post-doctoral research and my visit during PhD back in 2018. I received my Ph.D. degree from Department of Automation, Tsinghua University, Beijing China, in 2020, and prior in 2014 I obtained my Bachelor degree of Control Science and Technology, from Department of Information Science and Technology, Central South University, Changsha China, with Cum Laude (GPA Rank: 1/63). I am grateful to have Prof. Shuning Wang (王书宁) as my PhD supervisor, and to be inspired from working with Prof. Li Li (李力), Prof. Xialin Huang (黄晓霖), Prof. Jun Xu (许鋆), and Prof. Panos Patrinos.

Research Interest

I am interested in exploring neural networks and general machine learning approaches towards greater efficiency and better interpretation. In the era of big data, such approaches, including the long existing classical ones, are still fundamental towards understanding and developing deep learning and generic machine learning methods. However, bridging the gap between interpretable techniques with 'black-box' deep architectures is not easy, theoretically nor empirically.
My research work starts from piecewise linear neural networks delving from shallow to deep architectures, especially on novel piecewise linear function representation models with effective optimization algorithms and interpretability in analysis and applications. [e.g., Nature Reviews Methods Primers, IEEE T-NNLS, IEEE T-ITS, Automatica, Expert Systems with Applications, IEEE CDC 2018 ] I also did some work previously on algorithms for black-box optimization and sparse penalty functions with piecewise linearity for model sparsity [e.g., Signal Processing, Journal of Global Optimization, ACC 2024] Currently, my research line has been spanned to the areas with kernel methods which exert better interpretability and analytical benefits than deep neural networks, but lack flexibility for large-scale and complicated tasks; my relevant recent work mainly leverage the techniques from Lagrangian duality and conjugate feature duality w.r.t. deep kernel machines, asymmetric kernel-based learning, and so on, with particular interest towards underdtanding and optimizing the Attention mechanism in Transformers, as the attention matrix can be intrinsically regarded as an kernel matrix. [e.g., ICML 2024a, ICML 2024b, NeurIPS 2023, Neural Networks, Information Fusion, ICASSP 2023a, arXiv:2308.16056] Both Piecewise Linearity and Kernel Methods are classical tools for flexibility, interpretability, and utility in various tasks. It is being and would longstanding be interesting directions that aim to bring new perspectives and even synergies bridging to deep neural networks. I have also been collaborating on some interesting topics about optimization towards the generalization and robustness for deep neural networks, tensor-based methods for kernel machines, etc. [e.g., IJCV, TMLR, NeurIPS Workshop-OPT 2023, NeurIPS 2022, ICLR 2023, IEEE T-PAMI, IEEE T-NNLS, Pattern Recognition, Pattern Recognition Letters, ICASSP 2023b, arXiv:2211.10882]

Updates

• [07/2024] Our paper "Revisiting Deep Ensemble for Out-of-Distribution Detection: A Loss Landscape Perspective" is now on IJCV.
• [05/2024] Our papers "Learning in Feature Spaces via Coupled Covariances: Asymmetric Kernel SVD and Nyström method" and "Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processes" are accpted to ICML (2024).
• [03/2024] Our paper "Revisiting Random Weight Perturbation for Efficiently Improving Generalization" is accepted to TMLR. A short version is on NeurIPS Workshops on Optimization for Machine Learning (2023)
• [02/2024] Our paper "Sparsity via Sparse Group k-max Regularization" is accepted to American Control Conference for Oral presentation.
• [11/2023] Our paper "Towards robust neural networks via orthogonal diversity" is accepted to Pattern Recognition.
• [11/2023] I co-organized the Workshop: Rethinking Transformers through Duality and New Directions at Leuven.AI junior researchers day.
• [11/2023] Our paper "Deep Kernel Principal Component Analysis for Multi-level Feature Learning" is accepted to Neural Networks.
• [09/2023] Our paper "Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representation" is accepted to NeurIPS.
• [09/2023] Our paper "Low-Rank Multitask Learning based on Tensorized SVMs and LSSVMs" is now on arXiv, and it was partially published at "for regression tasks with tensorized LSSVMs", in ICASSP, in Rohdes Island, Greece, June 2023.
• [06/2023] Our papers "Nonlinear SVD with Asymmetric Kernels: Feature Learning and Asymmetric Nyström Method" and "Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representation" are now on arXiv.
• [03/2023] Our paper "Query Attack by Multi-Identity Surrogates" is published at IEEE Transactions on AI.
• [02/2023] Our paper "Jigsaw-ViT: Learning jigsaw puzzles in vision transformer" is published on Pattern Recognition Letters.
• [02/2023] Our paper "Trainable Weight Averaging: Efficient Training by Optimizing Historical Solutions" is published on ICLR.
• [11/2022] Our paper "On Multi-head Ensemble of Smoothed Classifiers for Certified Robustness" is now on arXiv.
• [09/2022] Our paper "Adversarial Attack on Attackers: Post-Process to Mitigate Black-Box Score-Based Query Attacks" is accepted to NeurIPS.
• [06/2022] Our paper "Tensor-based Multi-view Spectral Clustering via Shared Latent Space" is now on arXiv.
• [06/2022] Our paper "Piecewise Linear Neural Networks and Deep Learning" is published on Nature Reviews Methods Primers
• [05/2022] Our paper "Low Dimensional Trajectory Hypothesis is True: DNNs Can Be Trained in Tiny Subspaces" is published on IEEE TPAMI.
• [04/2022] Our papers "Measuring the Transferability of l-inf Attacks by the l-2 Norm" and "Tensorized LSSVMS For Multitask Regression" have been accepted to ICASSP.
• [01/2022] Our paper "Short-term Traffic Flow Prediction based on the Efficient Hinging Hyperplanes Neural Network" is published on IEEE Transactions on ITS.
• [11/2021] Our paper "Toward Deep Adaptive Hinging Hyperplanes" have been published on IEEE TNNLS.
• [06/2021] Our paper "Center-Aware Adversarial Autoencoder for Anomaly Detection" have been published on IEEE TNNLS.
• [05/2021] Our paper "Learning with Continuous Piecewise Linear Decision Trees" is published on Expert Systems with Applications.
• [11/2020] Our paper "Towards Robust Neural Networks via Orthogonal Diversity" is now on arXiv.
• [06/2020] Our paper "Efficient Hinging Hyperplanes Neural Network and its application in Nonlinear System Identification" have been published on Automatica.

Selected Work [Full List]

(Piecewise Linear Neural Networks)
Piecewise Linear Neural Networks and Deep Learning
Qinghua Tao, Li Li, Xiaolin Huang, Xiangming Xi, Shuning Wang, and Johan A.K. Suykens
Nature Reviews Methods Primers (2022), Impact Factor - 50.1 [NRMP Version] [ArXiv Version]
Short-term Traffic Flow Prediction based on the Efficient Hinging Hyperplanes Neural Network"
Qinghua Tao, Zhen Li, Jun Xu, Shu Lin, Bart De Schutter, and Johan A.K. Suykens
IEEE Transactions on Intelligent Transportation Systems (2022) [Paper]
Toward Deep Adaptive Hinging Hyperplanes
Qinghua Tao, Jun Xu, Zhen Li, Na Xie, Shuning Wang, Xiaoli Li, and Johan A.K. Suykens
IEEE Transactions on Neural Networks and Learning Systems(2021) [Paper]
Learning with Continuous Piecewise Linear Decision Trees
Qinghua Tao Zhen Li, Jun Xu, Na Xie, Shuning Wang, and Johan AK Suykens
Expert Systems with Applications (2021) [Paper]
Efficient Hinging Hyperplanes Neural Network and its application in Nonlinear System Identification
Jun Xu, Qinghua Tao, Zhen Li, Xiangming Xi, Johan AK Suykens, and Shuning Wang
Automatica (2020) [Paper]
Fast Adaptive Hinging Hyperplanes
Qinghua Tao, Jun Xu, Johan AK Suykens, and Shuning Wang
IEEE CDC (2018)[Paper]
Adaptive block coordinate DIRECT algorithm
Qinghua Tao, Xiaolin Huang, Shuning Wang, and Li Li
Journal of Global Optimization (2017)[Paper]
Sparsity via Sparse Group k-max Regularization
Qinghua Tao*, Xiangming Xi*, Jun Xu, and Johan AK Suykens
ACC (2024)[Paper]
Multiple Gaussian Graphical Estimation with Jointly Sparse Penalty
Qinghua Tao, Xiaolin Huang, Shuning Wang, Xiangming Xi, and Li Li
Signal Processing (2016)[Paper]

(Asymmetric Kernel SVD and Self-Attention Kernel in Transformers)

Learning in Feature Spaces via Coupled Covariances: Asymmetric Kernel SVD and Nyström method
Qinghua Tao*,Francesco Tonin*, Alex Lambert, Yingyi Chen, Panagiotis Patrinos, and Johan AK Suykens
ICML (2024)
Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representation
Yingyi Chen*, Qinghua Tao*, Francesco Tonin, and Johan AK Suykens
NeurIPS (2023)[Paper]
Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processes
Yingyi Chen*, Qinghua Tao*, Francesco Tonin, and Johan A.K. Suykens
ICML (2024)[Paper]
(* equal contribution)

(Deep/Tensorized Kernel Machines and Deep Learning)

Tensor-based Multi-view Spectral Clustering via Shared Latent Space
Qinghua Tao , Francesco Tonin , Panagiotis Patrinos, and Johan AK Suykens
( corresponding authors)
Information Fusion (2024) [Paper]
Deep Kernel Principal Component Analysis for Multi-level Feature Learning
Francesco Tonin , Qinghua Tao , Panagiotis Patrinos, and Johan AK Suykens
( corresponding authors)
Neural Networks (2024) [Paper]
Tensorized LSSVMs for Multitask Regression
Jiani Liu, Qinghua Tao, Ce Zhu, Yipeng Liu, and Johan AK Suykens
ICASSP (2023) [Paper]
Kernel PCA for Out-of-Distribution Detection
Kun Fang, Qinghua Tao, Kexin Lv, Mingzhen He, Xiaolin Huang, and Jie Yang
arXiv: 2402.02949 (2024) [Paper]

(Generalization of DNNs)

Revisiting Random Weight Perturbation for Efficiently Improving Generalization
Tao Li, Qinghua Tao, Weihao Yan, Zehao Lei, Yingwen Wu, Kun Fang, Mingzhen He, Xiaolin Huang
TMLR (2024)[Paper], A short version in NeurIPS Workshops on OPT (2023)[Paper]
Trainable Weight Averaging: Efficient Training by Optimizing Historical Solutions
Tao Li, Zhehao Huang, Qinghua Tao, Yingwen Wu, and Xiaolin Huang
ICLR (2023) [Paper]
Low Dimensional Trajectory Hypothesis is True: DNNs Can Be Trained in Tiny Subspaces
Tao Li, Weihao Yan, Qinghua Tao, Zehao Lei, Yingwen Wu, Kun Fang, Mingzhen He, Xiaolin Huang
IEEE Transactions on Pattern Analysis and Machine Intelligence (2022)[Paper]

(Robustness of DNNs)

Revisiting Deep Ensemble for Out-of-Distribution Detection: A Loss Landscape Perspective
Kun Fang*, Qinghua Tao*, Xiaolin Huang, and Jie Yang
(* equal contribution)
IJCV (2024) [Paper]
Towards robust neural networks via orthogonal diversity
Kun Fang, Qinghua Tao, Yingwen Wu, Tao Li, Jia Cai, Xiaolin Huang, and Jie Yang
Pattern Recognition (2024) [Paper]
Measuring the Transferability of l-inf Attacks by the l-2 Norm
Chen, Sizhe, Qinghua Tao , Zhixing Ye, and Xiaolin Huang
ICASSP (2023) [Paper]
Center-Aware Adversarial Autoencoder for Anomaly Detection
Daoming Li*, Qinghua Tao*, Jiahao Liu, and Huangang Wang
(* equal contribution)
IEEE Transactions on Neural Networks and Learning Systems (2021) [Paper]
Query Attack by Multi-Identity Surrogates
Sizhe Chen, Zhehao Huang, Qinghua Tao, and Xiaolin Huang
IEEE Transactions on Artificial Intelligence (2023) [Paper]
Jigsaw-ViT: Learning jigsaw puzzles in Vision Transformer
Yingyi Chen, Xi Shen, Yahui Liu, Qinghua Tao, and Johan AK Suykens.
Pattern Recognition Letters (2023) [Paper]
Adversarial Attack on Attackers: Post-Process to Mitigate Black-Box Score-Based Query Attacks
Sizhe Chen, Zhehao Huang, Qinghua Tao , Yingwen Wu, Cihang Xie, and Xiaolin Huang
NeurIPS (2022) [Paper]
On Multi-head Ensemble of Smoothed Classifiers for Certified Robustness
Kun Fang, Qinghua Tao, Yingwen Wu, Tao Li, Xiaolin Huang, and Jie Yang
arXiv:2211.10882 (2022) [Paper]

Selected Honors and Awards

• Comprehensive Scholarship of Tsinghua University for graduates, 2016,2018 .
• Student Research Travel Grant, IEEE CDC, 2018.
• Meritorious Winner, Mathematical Contest in Modeling of America (MCM), 2013.
• National Scholarship for Undergraduates (top 0.2%, highest honor), Ministry of Education of P.R.China, 2011, 2013.
• The 2nd prize of National College English Contest 2013; the 3rd place in English Speaking contest of Central South University 2012; the 3rd place in English Oratorical Contest of Central South University 2012.

Academic Services and Experiences

• Conference Reviewer
ICML, NeurIPS, ICLR, ICASSP, CDC, ACC, ECC.

• Journal Reviewer
IEEE Transactions on Image Processing, IEEE Transactions on Circuits and Systems for Video Technology, IEEE Transactions on Neural Networks and Learning Systems, IEEE Signal Processing Letters, IEEE Transactions on Automation Science and Engineering, Information Fusion, Machine Learning, Neural Networks, Pattern Recognition Letters.

• Master Thesis
Assess 10+ master theses from the master programs of AI, Mathematical Engineering, and Statistics and Data Science, in KU Leuven.
Supervise the master student with thesis " Multi-view spectral clustering for unsupervised object discovery via Transformer" from Master AI program in KU Leuven.

• Teaching
TA of Convex Optimization, Tsinghua University -- A specialized course to graduate students (Course #70250403 with textbook S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge, UK, Cambridge University Press, 2004)
TA of Operational Research, Tsinghua University -- A specialized course to undergraduate students (Course #20250013 with textbook Yunquan Hu, Operations research tutorial. Beijing, China, Tsinghua University Press, 2003).