CLSEBERT: Contrastive Learning for Syntax Enhanced Code Pre-Trained Model
CLSEBERT: 구문 강화 코드 사전 훈련 모델을 위한 대조 학습
CLSEBERT: Aprendizaje contrastivo para el modelo preentrenado de código mejorado de sintaxis
CLSEBERT : modèle pré-entraîné d'apprentissage contrastif pour le code amélioré de syntaxe
CLSEBERT: сравнительное обучение для предварительно обученной модели кода с расширенным синтаксисом
Xin Wang ¹, Yasheng Wang ², Pingyi Zhou ², Meng Xiao ², Yadao Wang ², Li Li ³, Xiao Liu ⁴, Hao Wu 武浩 ⁵, Jin Liu 刘进 ¹, Xin Jiang ²
¹ School of Computer Science, Wuhan University 武汉大学 计算机学院
² Noah's Ark Lab, Huawei 华为 诺亚方舟实验室
³ Faculty of Information Technology, Monash University
⁴ School of Information Technology, Deakin University
⁵ School of Information Science and Engineering, Yunnan University 云南大学 信息学院
arXiv, 10 August 2021

Pre-trained models for programming languages have proven their significant values in various code-related tasks, such as code search, code clone detection, and code translation. Currently, most pre-trained models treat a code snippet as a sequence of tokens or only focus on the data flow between code identifiers.

However, rich code syntax and hierarchy are ignored which can provide important structure information and semantic rules of codes to help enhance code representations. In addition, although the BERT-based code pre-trained models achieve high performance on many downstream tasks, the native derived sequence representations of BERT are proven to be of low-quality, it performs poorly on code matching and similarity tasks.

To address these problems, we propose CLSEBERT, a Constrastive Learning Framework for Syntax Enhanced Code Pre-Trained Model, to deal with various code intelligence tasks. In the pre-training stage, we consider the code syntax and hierarchy contained in the Abstract Syntax Tree (AST) and leverage the constrastive learning to learn noise-invariant code representations. Besides the masked language modeling (MLM), we also introduce two novel pre-training objectives. One is to predict the edges between nodes in the abstract syntax tree, and the other is to predict the types of code tokens. Through extensive experiments on four code intelligence tasks, we successfully show the effectiveness of our proposed model.
Reviews and Discussions
An overview on the principle of inkjet printing technique and its application in micro-display for augmented/virtual realities
Nonlinear optics with structured light
Femtosecond laser-induced periodic structures: mechanisms, techniques, and applications
Giant and light modifiable third-order optical nonlinearity in a free-standing h-BN film
New approach for the digital reconstruction of complex mine faults and its application in mining
p62/SQSTM1 Participates in the Innate Immune Response of Macrophages Against Candida albicans Infection
Configurable topological beam splitting via antichiral gyromagnetic photonic crystal
Single-molecule optoelectronic devices: physical mechanism and beyond
Functional nonlinear optical nanoparticles synthesized by laser ablation
The m6A methylation regulates gonadal sex differentiation in chicken embryo
A new species in the genus Synanthedon (Lepidoptera: Sesiidae) from China
A review on the forward osmosis applications and fouling control strategies for wastewater treatment

Previous Article                                Next Article
Copyright © Hot Paper