Optimization for Chain of Thought via Sparse Attention Mechanism | Muslims in ML at NeurIPS'24
Title: Reducing Reasoning Costs - The Path of Optimization for Chain of Thought via Sparse Attention Mechanism
Authors: Libo Wang
Abstract: In order to address the chain of thought in the large language model inference cost surge, this research proposes to use a sparse attention mechanism that only focuses on a few relevant tokens. The researcher constructed a new attention mechanism and used GiantRabbit trained with custom GPTs as an experimental tool. The experiment tested and compared the reasoning time, correctness score and chain of thought length of this model and o1 Preview in solving the linear algebra test questions of MIT OpenCourseWare. The results show that GiantRabbit's reasoning time and chain of thought length are significantly lower than o1 Preview. It verifies the feasibility of sparse attention mechanism for optimizing chain of thought reasoning. Detailed architectural details and experimental process have been uploaded to Github, the link is:https://github.com/brucewang123456789/GeniusTrail.git.
Read Full Paper: https://openreview.net/pdf?id=3OMJxXKvmS
For more information: https://neurips.cc/virtual/2024/affinity-event/105018
Muslims in ML Workshop Website: https://www.musiml.org/
Видео Optimization for Chain of Thought via Sparse Attention Mechanism | Muslims in ML at NeurIPS'24 канала Muslims In ML
Authors: Libo Wang
Abstract: In order to address the chain of thought in the large language model inference cost surge, this research proposes to use a sparse attention mechanism that only focuses on a few relevant tokens. The researcher constructed a new attention mechanism and used GiantRabbit trained with custom GPTs as an experimental tool. The experiment tested and compared the reasoning time, correctness score and chain of thought length of this model and o1 Preview in solving the linear algebra test questions of MIT OpenCourseWare. The results show that GiantRabbit's reasoning time and chain of thought length are significantly lower than o1 Preview. It verifies the feasibility of sparse attention mechanism for optimizing chain of thought reasoning. Detailed architectural details and experimental process have been uploaded to Github, the link is:https://github.com/brucewang123456789/GeniusTrail.git.
Read Full Paper: https://openreview.net/pdf?id=3OMJxXKvmS
For more information: https://neurips.cc/virtual/2024/affinity-event/105018
Muslims in ML Workshop Website: https://www.musiml.org/
Видео Optimization for Chain of Thought via Sparse Attention Mechanism | Muslims in ML at NeurIPS'24 канала Muslims In ML
Комментарии отсутствуют
Информация о видео
11 декабря 2024 г. 7:22:49
00:07:51
Другие видео канала
Oscar: The Generative AI student assistant | Muslims in ML at NeurIPS'24
3D localization and autofocus of the particle field | Muslims in ML at NeurIPS'24
NMTs as Efficient Forward and Backward Arabic Transliterators | Muslims in ML at NeurIPS'24
Multi-Modal Pipeline Defect Localization | Muslims in ML at NeurIPS'24
Stabilizing the Kuramoto–Sivashinsky Equation Using Deep Reinforcement Learning | MusIML ICML'25
ColFlor: Towards BERT-Size Vision-Language Document Retrieval Models | Muslims in ML at NeurIPS'24
G-RAG: Knowledge Expansion in Material Science | Muslims in ML at NeurIPS'24
MIMIC: Multimodal Islamophobic Meme Identification and Classification | Muslims in ML at NeurIPS'24
Federated Learning with Differential Privacy and Secure Aggregation | Muslims in ML at NeurIPS'24
LLM-Based Approach for Automated Seerah-Hadith Mapping | Muslims in ML at NeurIPS'24
A Contextualized BERT model for Knowledge Graph Completion | Muslims in ML at NeurIPS'24
GazaVHR: AI Driven Legally Grounded Conflict Harm Documentation | MusIML ICML'25
Virtual Presentations | Muslims in ML at NeurIPS 2024
HALT CoT Model Agnostic Early Stopping for CoT Reasoning via Answer Entropy | MusIML ICML'25
Compassionately: Increasing Plurality Awareness through AI | Muslims in ML at NeurIPS'24
RL for Controlling Synchronization of Weakly Coupled Kuramoto System | Muslims in ML at NeurIPS'24
Intelligent Departure Metering Advisory Tool for Airports| Muslims in ML at NeurIPS'24
Adaptation of Kruskal’s Uniqueness Conditions to multiview CP | Muslims in ML at NeurIPS'24
CoMRIAD: Neuroimage Analysis for Alzheimer’s Disease Detection | Muslims in ML at NeurIPS'24
A Dual Path Hybrid Convolutional Neural Network and Bidirectional LSTM | MusIML ICML'25
