Haoyu Lu

Haoyu Lu

If it works, just don't touch it.

LCA-on-the-Line Explained

2 minute read

Published:

Using in-distribution (ID) accuracy to analyze out-of-distribution (OOD) performance of models can encounter severe disintegrity, especially for models trained with diverse supervision and distributions, e.g., vision models (VM) and vision-language models (VLM). This paper, accepted at ICML 20241, proposed a novel, yet simple approach to measure OOD performance without introduction of any OOD data.

LCA Distance

LCA, or lowest common ancestor, is a concept in graph theory (particularly for tree - a special type of DAG). When I first look at the title and the abstract, what I am most curious about is how the graph/tree is constructed - you need a graph/tree first to find the LCA for an arbitrary pair of nodes. In this paper, the authors first employ existing large-scale lexical database like WordNet as the class taxonomy, and then propose to construct such hierarchies using K-means based on the similarities of the semantic information of class labels.

With that in mind, we can simply regard the LCA Distance as how close to the ground truth the prediction can be in a way that is somewhat knowledge-graph-based. And Figure 1 of this paper depicts how LCA distance, as a simple, robust measurement, excel in measuring OOD performance and outweigh conventional approaches where ID accuracy is adopted.

Overall, the LCA distance is described as

\[D_{\text{LCA}}(y^\prime, y)=f(y)-f(N_{\text{LCA}}(y^\prime, y)),\]

where $N_{\text{LCA}}(y^\prime, y)$ denotes the LCA node (a class label) for predicted class $y^\prime$ and ground truth $y$. $f(\cdot)$ is a function on nodes, such as the tree depth or entropy.

Using LCA Distance to Enhance Generalization

With aforementioned definition of LCA distance, it’s safe now to design respective loss functions to guide the optimization so that the model can be trained in a more generalized way. For a dataset with $n$ classes, we first establish an $n \times n$ LCA distance matrix $\boldsymbol{M}$, where $M_{ik}$ denotes the pairwise LCA distance of class nodes $i$ and $k$. Then the matrix is scaled by a temperature term T, and min-max scaling is applied to normalize the values between 0 and 1.

\[\boldsymbol{M}_{\text{LCA}}=MinMax(\boldsymbol{M}^T).\]

Now cross entropy (CE) or binary cross entropy (BCE) losses can be employed, and in the original paper, the authors use a weighted combination of ID CE and LCA CE to make it robust in both scenarios, that is,

\[\mathcal{L}=\lambda\mathcal{L}_{\text{CE}}+\mathcal{L}_{\text{CE}_{\text{LCA}}}.\]

  1. For the original paper/project page, see https://elvishelvis.github.io/papers/lca/ ↩

You May Also Enjoy

DeepONet and Operator Learning

4 minute read

Published:

Conventional trials of physics-informed deep learning (say, PINNs) integrates the dynamics as constraints to guide the approximation of a certain function within. An obvious problem with this idea is that such models cannot be directly applied to different initial and boundary conditions and observations. DeepONet, representing a new trial in physics-informed learning, employed some tricks to get around with this. Read more

DDPO: Aligning Diffusion Models with Human Preferences via Reinforcement Learning

8 minute read

Published:

Mainstream AIGC models, like ChatGPT and Stable Diffusion, have been drawing much attention over the recent years. Despite the massive amount of high-quality dataset, advanced network structure and loads of GPUs, one innegligible factor that contributes to their success is the use of RLHF, or reinforcement learning from human feedback. This blog is mostly based on a paper in May 2023 by the Levine Lab at UCB, and probes into how RLHF can be elegantly integrated into diffusion framework. Read more

Radiative Transfer Equation

3 minute read

Published:

Long time no see. After a tiring semester, I finally have the chance to update posts. Though there is much research progress to be talked about, it demands some more time to collect my thoughts about it. So this post will be a rather simple introduction to a famous equation in both physics and computer graphics, the radiative transfer equation (RTE). Read more

Integro-Differential Equations and Neural Network

5 minute read

Published:

Integro-Differential Equations, IDEs for short, are equations involving both partial differential and integral operators. They are commonly used in a variety of disciplines of science and engineering topics, while nonlinear IDEs are particularly difficult to solve analytically. Therefore, the idea to incorporate neural networks arises naturally. This post is dedicated to recording my understanding of IDEs and NNs. Read more