Haoyu Lu

Haoyu Lu

If it works, just don't touch it.

Reflections on Researches

2 minute read

Published:

A general overview of papers read recently, and a reflection on what we have done and what to do next.

端到端的医学影像重建 (Compton/PET)

GAN

  • Shen, G., Dwivedi, K., Majima, K., Horikawa, T., & Kamitani, Y. (2019). End-to-end deep image reconstruction from human brain activity. Frontiers in computational neuroscience, 13, 21

Flow

  • RealNVP: Hajij, M., Zamzmi, G., Paul, R., & Thukar, L. (2022, March). Normalizing Flow for Synthetic Medical Images Generation. In 2022 IEEE Healthcare Innovations and Point of Care Technologies (HI-POCT) (pp. 46-49). IEEE.

UNet

  • Zhao, R., Zhang, Y., Yaman, B., Lungren, M. P., & Hansen, M. S. (2021). End-to-end AI-based MRI reconstruction and lesion detection pipeline for evaluation of deep learning image reconstruction. arXiv preprint arXiv:2109.11524.

问题

  • 多数end-to-end是从sinogram (projection domain) 映射变换到重建图像 (image domain)，而Compton没有sinogram，且sinogram的引入会带来误差
  • 受限于ground truth成像质量，ground truth成像质量即其理论上界；产生高质量的ground truth、标注数据也会带来较大的时间代价
  • 如果辐射源不是点，而是面，那么这种条件下的“定位”，似乎跟图像重建没有区别
  • 如果从list mode数据出发，模型重建图像的可解释性？

Object Localization

WSOL: 弱监督目标定位

什么是WSOL

从image-level annotation出发，对目标进行定位

图像级标注 image-level annotation

仅标注图像中相关物体所属的类别，是最简单的标注

基于图像级标注，WSOL旨在给出带预测图像中包含主体的类别以及位置（bounding box）

通常思路: CAM

CAM，即Class Activation Map。通常模型首先生成CAM，然后分割出激活值最高的区域作为一个粗糙的定位结果。

CI-CAM

[[CI-CAM]]是基于因果推断的CAM模型。主要解决在定位问题中，受到背景信息$C$纠缠从而使输入$X$和目标$Y$之间存在虚假相关性的问题 image.png

Shao, F., Luo, Y., Zhang, L., Ye, L., Tang, S., Yang, Y., & Xiao, J. (2021, October). Improving Weakly Supervised Object Localization via Causal Intervention. In Proceedings of the 29th ACM International Conference on Multimedia (pp. 3321-3329).

Causal Inference: 因果推断

新坑，待填

GCN 图卷积神经网络

对每个事件，absorber和scatterer上的两个响应位置，再加上source location可以构成子图。事件累积，也即子图累积，最终形成一个图。理想情况下，该图具有以下特点

  • source location是一个关键点，连接所有子图
  • 该图不同于一般的抽象图，具有固定的几何构型（边长+角度），不可变形
    • 所以理论上图同构没有意义
  • 单向图：只有scatter指向absorber的边
  • 重边：相同的一组scatter和absorber可能重复形成event

相关应用

Conformation Generation

预测分子的可能坐标，即Conformation Generation，常用于蛋白质结构预测。

Mansimov, E., Mahmood, O., Kang, S. et al. Molecular Geometry Prediction using a Deep Generative Graph Neural Network. Sci Rep 9, 20381 (2019). https://doi.org/10.1038/s41598-019-56773-5

GraphNet

一个用于中微子望远镜图像重建的开源框架

https://github.com/graphnet-team/graphnet 没看懂，至少说明核物理也存在这种思路的应用

Point-GNN

对3D点云进行目标检测

Shi, W., & Rajkumar, R. (2020). Point-gnn: Graph neural network for 3d object detection in a point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 1711-1719). https://github.com/WeijingShi/Point-GNN

Track Reconstruction

高能物理实验中的粒子轨迹追踪/重建

Ju, X., Farrell, S., Calafiura, P., Murnane, D., Gray, L., Klijnsma, T., … & Usher, T. (2020). Graph neural networks for particle reconstruction in high energy physics detectors. arXiv preprint arXiv:2003.11603. https://jduarte.physics.ucsd.edu/phys139_239/finalprojects/Group3_Report.pdf

Interaction Network

  • The first general-purpose, learnable physics engine, and a powerful general framework for reasoning about object and relations in a wide variety of complex real-world domains
  • takes graphs as input, performs object- and relation-centric reasoning in a way that is analogous to a simulation, and is implemented using deep neural networks 就是一个针对物理问题的传统GCN模型，其核心也是Message Passing；但是原文以及多处应用均提到了它对于解决不同领域物理问题的泛用性

Battaglia, P., Pascanu, R., Lai, M., & Jimenez Rezende, D. (2016). Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems, 29.

数据集组织

如果不考虑event的先后顺序

  • 建立邻接矩阵$E_{adj}[21][21][3]$：每个邻接矩阵放21个events的feature
    • axis=0：absorber对应的scatterer
    • axis=1：$E_{1}$
    • axis=2：$E_{2}$
  • 保留响应点的几何特征：$(x_{1}, y_{1}, z_{1}), (x_{2}, y_{2}, z_{2})$

可以通过预测需要的邻接矩阵个数来判断数据量上的开销 是否同时需要$E_{1}$和$E_{2}$有待测试；可以只保留保留$E_{1}$，或者另一个换成$511-E_{1}$

To Start with: GraphSAGE

GraphSAGE针对之前的网络表示学习的transductive的问题，提出了一个inductive的GraphSAGE算法；同时利用节点特征信息和结构信息得到Graph Embedding的映射。相比之前的方法，之前都是保存了映射后的结果，而GraphSAGE保存了生成embedding的映射，可扩展性更强，对于节点分类和链接预测问题的表现也比较突出

You May Also Enjoy

DeepONet and Operator Learning

4 minute read

Published:

Conventional trials of physics-informed deep learning (say, PINNs) integrates the dynamics as constraints to guide the approximation of a certain function within. An obvious problem with this idea is that such models cannot be directly applied to different initial and boundary conditions and observations. DeepONet, representing a new trial in physics-informed learning, employed some tricks to get around with this. Read more

LCA-on-the-Line Explained

2 minute read

Published:

Using in-distribution (ID) accuracy to analyze out-of-distribution (OOD) performance of models can encounter severe disintegrity, especially for models trained with diverse supervision and distributions, e.g., vision models (VM) and vision-language models (VLM). This paper, accepted at ICML 20241, proposed a novel, yet simple approach to measure OOD performance without introduction of any OOD data.

  1. For the original paper/project page, see https://elvishelvis.github.io/papers/lca/ ↩
Read more

DDPO: Aligning Diffusion Models with Human Preferences via Reinforcement Learning

8 minute read

Published:

Mainstream AIGC models, like ChatGPT and Stable Diffusion, have been drawing much attention over the recent years. Despite the massive amount of high-quality dataset, advanced network structure and loads of GPUs, one innegligible factor that contributes to their success is the use of RLHF, or reinforcement learning from human feedback. This blog is mostly based on a paper in May 2023 by the Levine Lab at UCB, and probes into how RLHF can be elegantly integrated into diffusion framework. Read more

Radiative Transfer Equation

4 minute read

Published:

Long time no see. After a tiring semester, I finally have the chance to update posts. Though there is much research progress to be talked about, it demands some more time to collect my thoughts about it. So this post will be a rather simple introduction to a famous equation in both physics and computer graphics, the radiative transfer equation (RTE). Read more