Haiyang Wang

Haiyang Wang

Address: Max-Planck-Institut für Informatik
Saarland Informatics Campus
Campus E1 4
66123 Saarbrücken
Location: E1 4 - 622
Phone: +49 681 9325 2122
Fax: +49 681 9325 2099
E-mail: Get email via email

Personal Information

Publications

2024

Conference paper

D2

H. Wang, H. Tang, L. Jiang, S. Shi, M. F. Naeem, H. Li, B. Schiele, and L. Wang

“GiT: Towards Generalist Vision Transformer through Universal Language Interface,” in Computer Vision -- ECCV 2024, Milano, Italy.

more

BibTeX

@inproceedings{WangECCV24,
TITLE = {{GiT}: {T}owards Generalist Vision Transformer through Universal Language Interface},
AUTHOR = {Wang, Haiyang and Tang, Hao and Jiang, Li and Shi, Shaoshuai and Naeem, Muhammad Ferjad and Li, Hongsheng and Schiele, Bernt and Wang, Liwei},
LANGUAGE = {eng},
PUBLISHER = {Springer},
YEAR = {2024},
PUBLREMARK = {Accepted},
BOOKTITLE = {Computer Vision -- ECCV 2024},
SERIES = {Lecture Notes in Computer Science},
ADDRESS = {Milano, Italy},
}

Endnote

%0 Conference Proceedings
%A Wang, Haiyang
%A Tang, Hao
%A Jiang, Li
%A Shi, Shaoshuai
%A Naeem, Muhammad Ferjad
%A Li, Hongsheng
%A Schiele, Bernt
%A Wang, Liwei
%+ Computer Vision and Machine Learning, MPI for Informatics, Max Planck Society
External Organizations
Computer Vision and Machine Learning, MPI for Informatics, Max Planck Society
Computer Vision and Machine Learning, MPI for Informatics, Max Planck Society
External Organizations
External Organizations
Computer Vision and Machine Learning, MPI for Informatics, Max Planck Society
External Organizations
%T GiT: Towards Generalist Vision Transformer through Universal Language Interface : 
%G eng
%U http://hdl.handle.net/21.11116/0000-000F-D87B-4
%D 2024
%B 18th European Conference on Computer Vision 
%Z date of event: 2024-09-29 - 2024-10-04
%C Milano, Italy
%B Computer Vision -- ECCV 2024
%I Springer
%B Lecture Notes in Computer Science

Paper

D2

H. Wang, Y. Fan, M. F. Naeem, Y. Xian, J. E. Lenssen, L. Wang, F. Tombari, and B. Schiele

“TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters,” 2024. [Online]. Available: https://arxiv.org/abs/2410.23168.

more

Abstract

Transformers have become the predominant architecture in foundation models
due to their excellent performance across various domains. However, the
substantial cost of scaling these models remains a significant concern. This
problem arises primarily from their dependence on a fixed number of parameters
within linear projections. When architectural modifications (e.g., channel
dimensions) are introduced, the entire model typically requires retraining from
scratch. As model sizes continue growing, this strategy results in increasingly
high computational costs and becomes unsustainable. To overcome this problem,
we introduce TokenFormer, a natively scalable architecture that leverages the
attention mechanism not only for computations among input tokens but also for
interactions between tokens and model parameters, thereby enhancing
architectural flexibility. By treating model parameters as tokens, we replace
all the linear projections in Transformers with our token-parameter attention
layer, where input tokens act as queries and model parameters as keys and
values. This reformulation allows for progressive and efficient scaling without
necessitating retraining from scratch. Our model scales from 124M to 1.4B
parameters by incrementally adding new key-value parameter pairs, achieving
performance comparable to Transformers trained from scratch while greatly
reducing training costs. Code and models are available at
\url{https://github.com/Haiyang-W/TokenFormer}.

BibTeX

@online{Wang_2410.23168,
TITLE = {{TokenFormer}: Rethinking Transformer Scaling with Tokenized Model Parameters},
AUTHOR = {Wang, Haiyang and Fan, Yue and Naeem, Muhammad Ferjad and Xian, Yongqin and Lenssen, Jan Eric and Wang, Liwei and Tombari, Federico and Schiele, Bernt},
LANGUAGE = {eng},
URL = {https://arxiv.org/abs/2410.23168},
EPRINT = {2410.23168},
EPRINTTYPE = {arXiv},
YEAR = {2024},
ABSTRACT = {Transformers have become the predominant architecture in foundation models<br>due to their excellent performance across various domains. However, the<br>substantial cost of scaling these models remains a significant concern. This<br>problem arises primarily from their dependence on a fixed number of parameters<br>within linear projections. When architectural modifications (e.g., channel<br>dimensions) are introduced, the entire model typically requires retraining from<br>scratch. As model sizes continue growing, this strategy results in increasingly<br>high computational costs and becomes unsustainable. To overcome this problem,<br>we introduce TokenFormer, a natively scalable architecture that leverages the<br>attention mechanism not only for computations among input tokens but also for<br>interactions between tokens and model parameters, thereby enhancing<br>architectural flexibility. By treating model parameters as tokens, we replace<br>all the linear projections in Transformers with our token-parameter attention<br>layer, where input tokens act as queries and model parameters as keys and<br>values. This reformulation allows for progressive and efficient scaling without<br>necessitating retraining from scratch. Our model scales from 124M to 1.4B<br>parameters by incrementally adding new key-value parameter pairs, achieving<br>performance comparable to Transformers trained from scratch while greatly<br>reducing training costs. Code and models are available at<br>\url{https://github.com/Haiyang-W/TokenFormer}.<br>},
}

Endnote

%0 Report
%A Wang, Haiyang
%A Fan, Yue
%A Naeem, Muhammad Ferjad
%A Xian, Yongqin
%A Lenssen, Jan Eric
%A Wang, Liwei
%A Tombari, Federico
%A Schiele, Bernt
%+ Computer Vision and Machine Learning, MPI for Informatics, Max Planck Society
Computer Vision and Machine Learning, MPI for Informatics, Max Planck Society
External Organizations
External Organizations
Computer Vision and Machine Learning, MPI for Informatics, Max Planck Society
External Organizations
External Organizations
Computer Vision and Machine Learning, MPI for Informatics, Max Planck Society
%T TokenFormer: Rethinking Transformer Scaling with Tokenized Model
  Parameters : 
%G eng
%U http://hdl.handle.net/21.11116/0000-0010-0FC1-4
%U https://arxiv.org/abs/2410.23168
%D 2024
%X   Transformers have become the predominant architecture in foundation models<br>due to their excellent performance across various domains. However, the<br>substantial cost of scaling these models remains a significant concern. This<br>problem arises primarily from their dependence on a fixed number of parameters<br>within linear projections. When architectural modifications (e.g., channel<br>dimensions) are introduced, the entire model typically requires retraining from<br>scratch. As model sizes continue growing, this strategy results in increasingly<br>high computational costs and becomes unsustainable. To overcome this problem,<br>we introduce TokenFormer, a natively scalable architecture that leverages the<br>attention mechanism not only for computations among input tokens but also for<br>interactions between tokens and model parameters, thereby enhancing<br>architectural flexibility. By treating model parameters as tokens, we replace<br>all the linear projections in Transformers with our token-parameter attention<br>layer, where input tokens act as queries and model parameters as keys and<br>values. This reformulation allows for progressive and efficient scaling without<br>necessitating retraining from scratch. Our model scales from 124M to 1.4B<br>parameters by incrementally adding new key-value parameter pairs, achieving<br>performance comparable to Transformers trained from scratch while greatly<br>reducing training costs. Code and models are available at<br>\url{https://github.com/Haiyang-W/TokenFormer}.<br>
%K Computer Science, Learning, cs.LG