# Distance-wise Graph Contrastive Learning

https://arxiv.org/pdf/2012.07437

Distance-wise Graph Contrastive Learning，2020，arxiv preprint

# 1. 简介

## 1.1 摘要

Contrastive learning (CL) has proven highly effective in graph-based semi-supervised learning (SSL), since it can efficiently supplement the limited task information from the annotated nodes in graph. However, existing graph CL (GCL) studies ignore the uneven distribution of task information across graph caused by the graph topology and the selection of annotated nodes. They apply CL to the whole graph evenly, which results in an incongruous combination of CL and graph learning. To address this issue, we propose to apply CL in the graph learning adaptively by taking the received task information of each node into consideration. Firstly, we introduce Group PageRank to measure the node information gain from graph and find that CL mainly works for nodes that are topologically far away from the labeled nodes. We then propose our Distance-wise Graph Contrastive Learning (DwGCL) method from two views: (1) From the global view of the task information distribution across the graph, we enhance the CL effect on nodes that are topologically far away from labeled nodes; (2) From the personal view of each node’s received information, we measure the relative distance between nodes and then we adapt the sampling strategy of GCL accordingly. Extensive experiments on five benchmark graph datasets show that DwGCL can bring a clear improvement over previous GCL methods. Our analysis on eight graph neural network with various types of architecture and three different annotation settings further demonstrates the generalizability of DwGCL.

# 2. 方法

## 2.1 动机

### 2.1.1 Group PageRank

PageRank是一种常用来计算节点重要性的算法，公式如下：

$\pi_{p r}=(1-\alpha) \boldsymbol{A}^{\prime} \pi_{p r}+\alpha \boldsymbol{I}$

$\boldsymbol{\pi}_{g p r}(c)=(1-\alpha) \boldsymbol{A}^{\prime} \boldsymbol{\pi}_{g p r}+\alpha \boldsymbol{I}_{\boldsymbol{c}}$

$\boldsymbol{I}_{c}^{i}\left\{\begin{array}{ll} \frac{1}{\left|\boldsymbol{L}_{c}\right|}, & \text { if the } i \text {-th node is a labeled node of class } c \\ 0, & \text { otherwise } \end{array}\right.$

$Z=\alpha\left(E-(1-\alpha) A^{\prime}\right)^{-1} \boldsymbol{I}^{*}$

### 2.1.2 TIG值

$\begin{array}{c} Z_{i, c}^{*}=\frac{1}{2} Z_{i, c}\left(1+\frac{X_{i} \mathcal{P}_{c}^{T}}{\left\|\boldsymbol{X}_{i}\right\|\left\|\mathcal{P}_{c}\right\|}\right), \mathcal{P}_{c}=\frac{1}{\left|L_{c}\right|} \sum_{i \in \boldsymbol{L}_{c}} \boldsymbol{X}_{i} \end{array}$

$\begin{array}{c} T_{i}=\max \left(\boldsymbol{Z}_{i}^{*}\right)-\lambda \frac{\left(\sum_{c=0}^{k-1} \boldsymbol{Z}_{i, c}^{*}\right)-\max \left(\boldsymbol{Z}_{i}^{*}\right)}{k-1} \end{array}$

## 2.2 Distance-wise图对比学习

### 2.2.1 Distance-wise增强

• TIG值：TIG值越低的节点，被选中实施数据增强的概率越大。
• 考虑节点增强的扩散，作者通过降低增强节点周围子图的概率来动态调整选择概率。

### 2.2.2 Distance-wise正/负采样

1. 全局拓扑距离。节点的PageRank值包含了全局拓扑信息和标记信息，作者计算两个节点PageRank值之间的KL散度作为全局拓扑距离：

$\boldsymbol{D}_{i, j}^{g}=\operatorname{KL}\left(\boldsymbol{p}_{i}, \boldsymbol{p}_{j}\right), \text { with } \quad p_{i}=\operatorname{NORM}\left(Z_{i}^{*}\right)$

2. 局部拓扑距离。作者将两个节点之间的mini jump hop数作为局部拓扑距离。

3. 节点嵌入距离，即两个节点嵌入间的余弦距离。

$\boldsymbol{D}_{i, j}=\mathrm{S}\left(\boldsymbol{D}_{i, j}^{g}\right)+\lambda_{1} \mathrm{~S}\left(\boldsymbol{D}_{i, j}^{l}\right)+\lambda_{2} \mathrm{~S}\left(\boldsymbol{D}_{i, j}^{e}\right)$

$\boldsymbol{P}_{i}=\boldsymbol{R}_{i}\left[0: \operatorname{post}_{e n d}\right], \boldsymbol{N}_{i}=\boldsymbol{R}_{i}\left[\operatorname{negt}_{b e g}: \text { negt }_{\text {end }}\right]$

## 2.3 DwGCL目标函数

### 2.3.1 有监督损失

$\mathcal F(·)$表示GNN编码器，对于有标签节点其交叉熵损失定义为：

\begin{aligned} t &=\mathcal{F}(\boldsymbol{X}, \boldsymbol{A}, \theta) \\ \mathcal{L}_{C E} &=-\frac{1}{|\boldsymbol{L}|} \sum_{i \in \boldsymbol{L}} \sum_{c=0}^{k-1} \boldsymbol{y}_{i} \log p\left(\boldsymbol{t}_{i}^{c}, \tau\right) \end{aligned}

### 2.3.2 无监督损失

self-consistency损失定义如下：

$\mathcal{L}_{s}^{i}=\mathrm{KL}\left(\mathcal{F}\left(\boldsymbol{X}_{\boldsymbol{p}}, \boldsymbol{A}_{\boldsymbol{p}}, \boldsymbol{\theta}\right)_{i}, \mathcal{F}(\boldsymbol{X}, \boldsymbol{A}, \widetilde{\boldsymbol{\theta}})_{i}\right)$

\begin{aligned} \mathcal{L}_{p}^{i} &=\frac{1}{\left|\boldsymbol{P}_{i}\right|} \sum_{j \in \boldsymbol{P}_{i}} \mathrm{KL}\left(\mathcal{F}\left(\boldsymbol{X}_{\boldsymbol{p}}, \boldsymbol{A}_{\boldsymbol{p}}, \boldsymbol{\theta}\right)_{i}, \mathcal{F}(\boldsymbol{X} \boldsymbol{A}, \widetilde{\boldsymbol{\theta}})_{j}\right) \\ \mathcal{L}_{n}^{i} &=\frac{1}{\left|\boldsymbol{N}_{i}\right|} \sum_{j \in \boldsymbol{N}_{i}} \mathrm{KL}\left(\mathcal{F}\left(\boldsymbol{X}_{\boldsymbol{p}}, \boldsymbol{A}_{\boldsymbol{p}}, \boldsymbol{\theta}\right)_{i}, \mathcal{F}(\boldsymbol{X} \boldsymbol{A}, \widetilde{\boldsymbol{\theta}})_{j}\right) \end{aligned}

$\mathcal{L}_{U}^{i}=\mathcal{L}_{s}^{i}+\mu_{1} \mathcal{L}_{p}^{i}-\mu_{2} \mathcal{L}_{n}^{i}$

$\boldsymbol{w}_{i}=w_{\min }+\frac{1}{2}\left(w_{\max }-w_{\min }\right)\left(1+\cos \left(\frac{\operatorname{Rank}\left(\boldsymbol{T}_{i}\right)}{n} \pi\right)\right)$

$\mathcal{L}=\mathcal{L}_{C E}+\frac{1}{n} \sum_{i=0}^{n-1} \boldsymbol{w}_{i} \mathcal{L}_{U}^{i}$

# 3. 实验

## 3.2 案例分析

### 3.2.3 样本大小和样本难度之间的权衡

Contrastive Pair的采样在对比学习中十分重要。从上图5可以看到，负样本和锚点间距离太远或者太近都不好，这是符合常理的。因为太远了，两者太容易被区分，太近了两者太相似起不到对比的作用，都会影响对比学习的效果。另外，起初负样本数量越大，模型性能越好，但是大到一定数量反而会降低模型性能。

• 版权声明： 本博客所有文章除特别声明外，著作权归作者所有。转载请注明出处！