Research
EDGE GENERAL INTELLIGENCE
Multi-Agent Collaboration for Edge General Intelligence: Edge General Intelligence (EGI) aims to equip devices at the network edge with human-like cognitive and decision-making capabilities, enabling them to operate autonomously in open and dynamic real-world environments. This research direction focuses on the promising frontier of large language model (LLM)-empowered multi-agent collaboration. The central objective is to investigate how LLMs can serve as the core reasoning engine for individual agents and facilitate the construction of an efficient multi-agent collaborative framework. By exploring mechanisms for communication, knowledge sharing, and collective reasoning among agents, this research seeks to integrate the localized observations and knowledge of multiple agents. The ultimate goal is to overcome the fundamental constraint of spatiotemporal partial observability, thereby achieving a globally consistent situational awareness and enabling coordinated decision-making.
Federated learning enables distributed training across a set of clients, without requiring any of the participants to reveal their private training data to a centralized entity or each other. Due to decentralized execution, it is vulnerable to attacks from adversarial (Byzantine) clients by modifying the local updates to their desires. Therefore, it is essential to develop robust algorithms that can defend Byzantine clients without losing model convergence and performance.
Network intrusion detection is crucial for cybersecurity. However, in dynamic network environments, the threat landscape and normal operational behaviors constantly evolve, challenging static intrusion detection systems (IDSs). By incorporating continual learning, IDSs can be equipped with the ability to continuously update and adapt to new and emerging threats.
Traditional distributed learning (DL) in edge networks faces challenges from malicious attacks and privacy breaches, necessitating the integration of secure techniques, such as blockchain, for enhanced security and transparency. Existing secure DL solutions face issues related to high computational demands, communication complexities, and on-chain storage costs. Thus, developing a lightweight and secure DL system is crucial for intelligent edge networks.
Reliable and robust malicious traffic detection in open-world environments is crucial for maintaining cybersecurity in the face of evolving network threats. Open-world environments present multiple risks, including the presence of unknown threats, increased attack vectors due to diverse and dynamic systems, and the challenge of distinguishing between benign and malicious activities in a constantly changing landscape. This study explores key strategies for implementing malicious traffic detection in such contexts, emphasizing the incorporation of trustworthy and explainable AI techniques to address these challenges.
Federated learning (FL) inevitably confronts the challenge of system heterogeneity in practical scenarios. To enhance the capabilities of most model-homogeneous FL methods in handling system heterogeneity, we propose InCo Aggregation to augment the similarity in the deep layers without requiring additional communication between clients. Furthermore, the method can expand the capabilities of the existing model-homogeneous FL methods in handling the system heterogeneity.
We propose shifting the anchor modality from image to text. The text embeddings of each category serve as anchors, providing a more accurate distribution as well as protecting private data. By leveraging Vision-Language Models (VLMs), we can create a seamless connection between image and text modalities. Moreover, inspired by the clustering tendency observed in the image embeddings of VLMs, we propose generating category-specific Gaussian noise within the VLM's image latent space to serve as synthetic training data. These synthetic data act as proxies for their respective categories and are used together with the original local data during training to address data imbalance.
Non-independent and identically distributed (Non-IID) data across edge clients have long posed significant challenges to federated learning (FL) training in edge computing environments. Prior works have proposed various methods to mitigate this statistical heterogeneity. While these works can achieve good theoretical performance, in this work we provide the first investigation into a hidden over-correction phenomenon brought by the uniform model correction coefficients across clients adopted by existing methods. Such over-correction could degrade model performance and even cause failures in model convergence. To address this, we propose TACO, a novel algorithm that addresses the non-IID nature of clients' data by implementing fine-grained, client-specific gradient correction and model aggregation, steering local models towards a more accurate global optimum. Moreover, we verify that leading FL algorithms generally have better model accuracy in terms of communication rounds rather than wall-clock time, resulting from their extra computation overhead imposed on clients.
LLM & AI APPLICATIONS
Radio2Speech is a system that uses RF signals to recover speech with high quality from the loudspeaker. Radio2Speech can recover speech comparable to the quality of a microphone. In quiet scenarios, Radio2Speech can recover high quality speech like a microphone. Moreover, since RF signals are not affected by the noise, our system is still able to recover high quality speech in noisy scenarios, while microphone almost fails. The existence of soundproof materials can make the microphone deaf, but RF signals can traverse occlusions. Thus, the speech recovered by Radio2Speech remains intelligible in soundproof scenarios.
Radio2Text is the first mmWave-based system for streaming automatic speech recognition (ASR) with a vocabulary size exceeding 13,000 words. Based on the streaming Transformer, we propose a cross-modal knowledge distillation (KD) structure, to mitigate the negative effect of low quality mmWave signals on recognition performance. In the cross-modal KD, the audio streaming Transformer provides feature and response guidance that inherit fruitful and accurate speech information to supervise the training of the tailored radio streaming Transformer.
Speculative Decoding (SPD) is an innovative inference acceleration technique designed to overcome the inherent latency of autoregressive generation in large language models (LLMs). By breaking the sequential token-by-token generation bottleneck, SPD significantly improves inference speed while maintaining output quality. The core mechanism employs a two-stage process: a smaller, faster "draft" model first generates multiple candidate tokens in sequence. These candidates are then verified in parallel by the larger target LLM through a single forward pass. Tokens that match the target model's predictions are accepted, while the first mismatch triggers rejection of subsequent draft tokens, with generation resuming from that point. Unlike traditional optimization methods that compromise model capabilities through quantization or pruning, SPD preserves the target model's full capabilities while reducing latency.
Users often take notes for instructional videos to access key knowledge later without revisiting long videos. Automated note generation tools enable users to obtain informative notes efficiently. In this work, we present NoteIt, a system, which automatically converts instructional videos to interactable notes using a novel pipeline that faithfully extracts hierarchical structure and multimodal key information from videos. With NoteIt’s interface, users can interact with the system to further customize the content and presentation formats of the notes according to their preferences.
RECOMMENDATION SYSTEM
We propose a novel group recommendation method AlignGroup, which focuses on both group consensus and individual preferences of group members to infer the group decision-making. Specifically, AlignGroup explores group consensus through a well-designed hypergraph neural network that efficiently learns intra- and inter-group relationships. Moreover, AlignGroup innovatively utilizes a self-supervised alignment task to capture fine-grained group decision-making by aligning the group consensus with members' common preferences.
We propose a novel self-supervised learning framework in multimodal recommendation, named MENTOR, for multimodal recommendation. After learning the representation of all modalities by our powerful multimodal information encoder. MENTOR introduces a tailored multilevel cross-modal alignment task to align different modalities on data distribution while retaining historical interaction information. Moreover, MENTOR devises an optional general feature enhancement contrastive learning task from both the feature and graph perspectives to improve the model robustness. Our extensive experimental results on several widely used datasets show that MENTOR achieves significant accuracy improvement compared with the state-of-the-art multimodal recommendation methods.
We propose a simple yet effective paradigm, NLGCL, which leverages naturally existing contrastive views between neighbor layers within GNNs. By treating each node and its neighbors in the next layer as positive pairs and other nodes as negatives, NLGCL avoids augmentation-based noise while preserving semantic relevance. This approach reduces irrelevant noise and eliminates the additional time and space costs of traditional contrastive learning methods. We demonstrate the superiority of NLGCL over state-of-the-art baselines in terms of both effectiveness and efficiency through theoretical analysis and empirical evaluation. By eliminating the need to construct and store additional contrastive views, NLGCL breaks the limitations of traditional GCL-based recommendation paradigms and opens new avenues for research. In the future, we plan to extend this paradigm to other domains, such as multimodal, social, and group recommendations.
The evolution of Internet-of-Things (IoT) is fostering the use of intelligent controls for energy conservation. Yet, the efficacy of these strategies is largely tied to diverse load forecasting algorithms. Given the significant contribution of heating, ventilation, and air-conditioning (HVAC) systems to global energy consumption, accurate forecasting of HVAC power usage is crucial for improving overall energy efficiency. However, real-world HVAC load forecasting, bolstered by various IoT devices, is complicated by multiple factors: data variability, power load fluctuations, electronic phenomena (e.g., zero drifts), and the increased time complexity and larger model sizes required to manage accumulating historical data. To address these challenges, we propose HALO, a transformer-based framework specifically designed for forecasting HVAC load. HALO incorporates an adaptive data pre-processing stage and a local-global-scale transformer-based load forecasting stage, enabling precise forecasting of HVAC load and optimization of energy utilization. Evaluation based on real-world data traces from a prototype application demonstrates that the proposed framework significantly outperforms existing models.
Human activity recognition using wearable sensor signals presents significant challenges due to the diverse range of human activities and the similarities in sensor signals from various events.To address this issue, we leverage sensor signals from multiple modalities for more accurate predictions across a broader spectrum of activities. Harnessing the power of the contrastive learning paradigm, we move beyond the traditional one-modality-to-one-modality approach, enabling the use of multiple query modalities that provide complementary information.
Smart technologies, including IoT, machine learning, and edge computing, are investigated to develop a smart water auditing system to generate high-resolution data on household-level water consumption. Such data allow researchers to analyze domestic water consumption patterns. Findings will help optimize water supply operations and translate into timely feedback to reinforce positive water conservation behaviour. This is an interdisciplinary project with the HKU Center for Water Technology and Policy.
SMART HEALTH
Advances in AI are expanding the frontier of smart healthcare, especially in neurohealth applications. Our work synthesizes four projects across two core themes: disease diagnosis and monitoring, and brain–computer interface (BCI) applications. For diagnosis and monitoring, we integrate large language models (LLMs) with multimodal neuroimaging to analyze tinnitus, and employ brain‑inspired hyperdimensional computing to predict and detect epileptic seizures. For BCI, we develop an affect‑aware steady‑state visual evoked potential (SSVEP)‑based interface via disentangled representation learning and propose a noise‑robust learning framework for EEG‑based detection of driver fatigue. Collectively, these approaches open new possibilities for AI‑driven neurohealthcare.
In the retrosynthesis planning task, we employ reinforcement-learning agents equipped with multi-scale planning and global monitoring to avoid the pitfall of “locally optimal but globally suboptimal” decisions. Before selecting any disconnection or reaction template, the system performs forward lookahead to assess downstream feasibility in terms of route completeness, cost, and complexity. A long-horizon value network ensures that rewards reflect pathway-level quality rather than the superficial correctness of a single step. To complement this forward reasoning, the system adds a backward-verification reward: each intermediate is tested for its ability to reconstruct the target molecule through reverse traversal of the template space. Steps that pass this consistency check receive additional reward, while failures are penalized as deceptive local optima. Combined with reward relabeling and hierarchical control, this mechanism improves credit assignment across long action sequences and significantly enhances the discovery of globally optimal synthesis routes.
Medical data is inherently multimodal, which includes clinical notes, bio-signal, lab test and also demographic, capturing diverse aspects of an individual’s health status. It holds great potential for various clinical predictive tasks. However, real-world medical data is often plagued by quality issues, primarily modality heterogeneity, modality missingness, and unlabeled data, which pose significant challenges to downstream modeling. Multi-modal learning is the key tool to conduct accurate clinical under low-quality medical data.
Pervasive healthcare focuses on developing healthcare solutions that are seamlessly integrated into everyday life, making healthcare services available anytime and anywhere. In light of the increasing availability of diverse data sources, such as user-environment contexts and ambient sensor signals, we explore the new challenges and opportunities in developing mobile analytics systems to empower pervasive healthcare.
Steady State Visually Evoked Potential (SSVEP)-based Brain-Computer Interfaces (BCIs) are extensively employed in rehabilitation and disability assistance, primarily due to their high information transfer rate (ITR). The efficacy of human-machine interaction in SSVEP-BCIs is pivotal for their practical implementation. Negative emotional states can adversely affect target detection accuracy, thus diminishing the ITR and impacting user experience. To address this, real-time detection and regulation of users' emotional states are essential for optimizing ITR and enhancing interaction performance.