Publications | Hoang-Chau Luong

Conference Papers

EMNLP
Rethinking the Role of Temperature in Large Language Model Distillation

Hoang-Chau Luong and Lingwei Chen

In Conference on Empirical Methods in Natural Language Processing (EMNLP), Main Conference, Short Paper, 2026

Key idea The common verdict that reverse KL beats forward KL for LLM distillation was measured at temperature 1. We show that temperature injects signal from non-dominant tokens into forward KL but mostly just rescales reverse-KL gradients. With a well-chosen temperature, forward KL overtakes reverse KL, and simple KL objectives gain up to +6.6 ROUGE-L, competitive with recent state-of-the-art methods.

Abs arXiv Bib PDF

Reverse Kullback-Leibler (RKL) divergence is widely favored over forward KL (FKL) in large language models (LLM) distillation, yet this preference is largely based on comparisons that omit the temperature, overlooking its central role in softening teacher distributions and improving knowledge transfer. In this work, we revisit temperature in LLM distillation and show that it fundamentally changes the comparison between FKL and RKL. Our analysis reveals an asymmetric effect: temperature substantially enriches FKL with non-dominant token signals, whereas it mainly rescales RKL gradients, causing FKL to benefit much more from temperature scaling than RKL. This asymmetry overturns the standard empirical conclusion: although RKL outperforms FKL at temperature 1, FKL consistently surpasses RKL at higher temperatures across instruction-following benchmarks. Moreover, the impact of temperature is not limited to FKL; it improves a broader family of distillation objectives, enabling simple KL-based methods to achieve competitive performance against recent state-of-the-art LLM distillation approaches.
@inproceedings{luong2026temperature, title = {Rethinking the Role of Temperature in Large Language Model Distillation}, author = {Luong, Hoang-Chau and Chen, Lingwei}, booktitle = {Conference on Empirical Methods in Natural Language Processing (EMNLP), Main Conference, Short Paper}, year = {2026}, }
ACL
Why LoRA Fails to Forget: Regularized Low-Rank Adaptation Against Backdoors in Language Models

Hoang-Chau Luong and Lingwei Chen

In Findings of the Association for Computational Linguistics (ACL Findings), 2026

Key idea Fine-tuning a backdoored model with LoRA on clean data rarely removes the backdoor. We show the cause is spectral rather than just low rank: LoRA updates have singular values far below those of the pretrained weights and align with the wrong directions. RoRA raises spectral strength and corrects the alignment, lowering attack success while keeping clean accuracy.

Abs arXiv Bib PDF

Low-Rank Adaptation (LoRA) is widely used for parameter-efficient fine-tuning of large language models, but it is notably ineffective at removing backdoor behaviors from poisoned pretrained models when fine-tuning on clean dataset. Contrary to the common belief that this weakness is caused primarily by low rank, we show that LoRA’s vulnerability is fundamentally spectral. Our analysis identifies two key factors: LoRA updates (i) possess insufficient spectral strength, with singular values far below those of pretrained weights, and (ii) exhibit unfavorable spectral alignment, weakly matching clean-task directions while retaining overlap with trigger-sensitive subspaces. We further establish a critical scaling threshold beyond which LoRA can theoretically suppress trigger-induced activations, and we show empirically that standard LoRA rarely reaches this regime. We introduce Regularized Low-Rank Adaptation (RoRA), which improves forgetting by increasing spectral strength and correcting alignment through clean-strengthened regularization, trigger-insensitive constraints, and post-training spectral rescaling. Experiments across multiple NLP benchmarks and attack settings show that RoRA substantially reduces attack success rates while maintaining clean accuracy.
@inproceedings{luong2026lora, title = {Why LoRA Fails to Forget: Regularized Low-Rank Adaptation Against Backdoors in Language Models}, author = {Luong, Hoang-Chau and Chen, Lingwei}, booktitle = {Findings of the Association for Computational Linguistics (ACL Findings)}, year = {2026}, }
AISTATS
Understanding SAM’s Robustness to Noisy Labels through Gradient Down-weighting

Hoang-Chau Luong, Thuc Nguyen-Quang, Dat Ba Tran, and Minh-Triet Tran

In International Conference on Artificial Intelligence and Statistics (AISTATS), 2026

Key idea Why is SAM robust to noisy labels? Looking at each parameter, when noisy gradients dominate a direction, SAM amplifies the clean gradients more strongly, which down-weights the noisy ones. SANER makes this effect explicit, slowing the memorization of noisy labels and beating both SAM and SGD. It also plugs into other SAM variants.

Abs arXiv Bib HTML PDF

Sharpness-Aware Minimization (SAM) was introduced to improve generalization by seeking flat minima, yet it also exhibits robustness to label noise, a phenomenon that remains only partially understood. Prior work has mainly attributed this effect to SAM’s tendency to prolong the learning of clean samples. In this work, we provide a complementary explanation by analyzing SAM at the element-wise level. We show that when noisy gradients dominate a parameter direction, their influence is reduced by the stronger amplification of clean gradients. This slows the memorization of noisy labels while sustaining clean learning, offering a more complete account of SAM’s robustness. Building on this insight, we propose SANER (Sharpness-Aware Noise-Explicit Reweighting), a simple variant of SAM that explicitly magnifies this down-weighting effect. Experiments on benchmark image classification tasks with noisy labels demonstrate that SANER significantly mitigates noisy-label memorization and improves generalization over both SAM and SGD. Moreover, since SANER is designed from the mechanism of SAM, it can also be seamlessly integrated into SAM-like variants, further boosting their robustness.
@inproceedings{luong2026understandingsamsrobustnessnoisy, title = {Understanding SAM's Robustness to Noisy Labels through Gradient Down-weighting}, author = {Luong, Hoang-Chau and Nguyen-Quang, Thuc and Tran, Dat Ba and Tran, Minh-Triet}, booktitle = {International Conference on Artificial Intelligence and Statistics (AISTATS)}, year = {2026}, }
NeurIPS
Fundamental Convergence Analysis of Sharpness-Aware Minimization

Pham Duy Khanh, Hoang-Chau Luong, Boris S. Mordukhovich, and Dat Ba Tran

In Advances in Neural Information Processing Systems (NeurIPS), 2024

Key idea SAM is widely used, but its basic convergence was not fully understood. We prove that the gradients go to zero, the function values reach the optimal value, and the iterates converge to a solution. Because the proof uses an inexact gradient descent framework, it also covers VaSSO, RSAM and USAM.

Abs arXiv Bib HTML PDF

The paper investigates the fundamental convergence properties of Sharpness-Aware Minimization (SAM), a recently proposed gradient-based optimization method (Foret et al., 2021) that significantly improves the generalization of deep neural networks. The convergence properties including the stationarity of accumulation points, the convergence of the sequence of gradients to the origin, the sequence of function values to the optimal value, and the sequence of iterates to the optimal solution are established for the method. The universality of the provided convergence analysis based on inexact gradient descent frameworks (Khanh et al., 2023b) allows its extensions to the normalized versions of SAM such as VaSSO (Li and Giannakis, 2023), RSAM (Liu et al., 2022), and to the unnormalized versions of SAM such as USAM (Andriushchenko and Flammarion, 2022). Numerical experiments are conducted on classification tasks using deep learning models to confirm the practical aspects of our analysis.
@inproceedings{klmtneurips, title = {Fundamental Convergence Analysis of Sharpness-Aware Minimization}, author = {Khanh, Pham Duy and Luong, Hoang-Chau and Mordukhovich, Boris S. and Tran, Dat Ba}, booktitle = {Advances in Neural Information Processing Systems (NeurIPS)}, year = {2024}, }

Preprints & Under Review

Preprint

Unlearning by Extrapolating a Shrink-and-Heal Lookahead

Hoang-Chau Luong and Lingwei Chen

2026

Under review

Key idea Machine unlearning should make a fine-tuned LLM forget a specific subset of its training data while keeping everything else. Our method takes a short shrink-and-heal lookahead step and extrapolates along it. On TOFU (Llama models up to 8B) it lands 3.7x closer to the retrained oracle than the next-best of 11 baselines, and it stays stable under relearning attacks.
Preprint
Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation

Hoang-Chau Luong, Dat Ba Tran, and Lingwei Chen

2026

Under review

Key idea Reverse KL is the go-to objective for LLM distillation, but its gradients keep pushing the target-token logit up even after the student already matches the teacher. This makes students overconfident and less diverse, with weak supervision on tail tokens. DRKL removes the effect and strengthens non-target supervision, beating forward KL, reverse KL and recent objectives on Qwen3, Llama-3.2 and OPT with a better fidelity-diversity trade-off.

Abs arXiv Bib PDF

Reverse Kullback-Leibler (RKL) divergence has recently emerged as the preferred objective for large language model (LLM) distillation, consistently outperforming forward KL (FKL), particularly in regimes with large vocabularies and significant teacher-student capacity mismatch, where RKL focuses learning on dominant modes rather than enforcing dense alignment. However, RKL introduces a structural limitation that drives the student toward overconfident predictions. We first provide an analysis of RKL by decomposing its gradients into target and non-target components, and show that non-target gradients consistently push the target logit upward even when the student already matches the teacher, thereby reducing output diversity. In addition, RKL provides weak supervision over non-target classes, leading to poor tail alignment. To address these issues, we propose Diversity-aware RKL (DRKL), which removes this gradient effect and strengthens non-target supervision while preserving the optimization benefits of RKL. Extensive experiments across datasets and model families demonstrate that DRKL consistently outperforms FKL, RKL, and other state-of-the-art distillation objectives, achieving better performance and a superior fidelity-diversity trade-off.
@unpublished{luong2026drkl, title = {Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation}, author = {Luong, Hoang-Chau and Tran, Dat Ba and Chen, Lingwei}, note = {Under review}, year = {2026}, }
Preprint
Consistently Informative Soft-Label Temperature for Knowledge Distillation

Hoang-Chau Luong, Nghia Van Vo, Kaiqi Zhao, and Lingwei Chen

2026

Under review

Key idea A single global temperature makes some teacher soft labels too sharp and others too flat, and it forces teacher and student onto the same logit scale. CIST gives the teacher and the student their own temperature for each sample, guided by the finding that soft-label entropy is governed by the ratio of the max logit to the temperature. It gains up to +3.6% over standard KD on CIFAR-100 at negligible extra cost.

Abs arXiv Bib PDF

Knowledge distillation (KD) transfers knowledge from a high-capacity teacher to a compact student by matching their predictive distributions, with temperature scaling serving as a central mechanism for smoothing teacher predictions and exposing informative "dark knowledge" beyond the hard label. However, the standard fixed-temperature design is inherently sample-agnostic. Since samples differ in logit scale and learning difficulty, a single global temperature produces teacher soft labels with highly inconsistent entropy: some predictions remain overly sharp and provide limited inter-class information, whereas others become over-smoothed and lose class-discriminative information. Moreover, sharing the same temperature between teacher and student further imposes rigid logit-scale alignment despite their capacity mismatch. To address these limitations, we propose CIST (Consistently Informative Soft-label Temperature), which assigns separate sample-wise adaptive temperatures to the teacher and student. This design produces consistently informative teacher soft labels while relaxing rigid teacher–student logit-scale matching. It also reweights the distillation objective according to teacher confidence and student learning difficulty. Theoretically, we show that teacher-label entropy is largely governed by the ratio between the maximum teacher logit and the temperature, providing a principled basis for adaptive smoothing. Empirically, CIST mitigates the inconsistency induced by fixed temperature, and experiments on both vision and language distillation tasks show consistent improvements over standard KD and strong baselines with negligible computational overhead.
@unpublished{luong2026cist, title = {Consistently Informative Soft-Label Temperature for Knowledge Distillation}, author = {Luong, Hoang-Chau and Vo, Nghia Van and Zhao, Kaiqi and Chen, Lingwei}, note = {Under review}, year = {2026}, }