Improving Neural Network Efficiency: Quantization - Live Podcast

​Guest:
Jan Akhremchik, AI/ML Researcher, AIFoundry.org

​Host: 
Greg Chase, AIFoundry.org Community Organizer

​Any user of Llama.cpp or its derivatives, such as Llamafile or Ollama, generally understands that quantized LLMs are versions reduced in size to run effectively on smaller systems such as laptops. The reduced model size allows for lower memory consumption and faster execution in exchange for reduced precision of answers.

​However, what are the different choices a model developer or user can make to increase the efficiency of their LLMs, and what are their tradeoffs?

​In this podcast, Jan Achremchik reviews the paper “INTEGER QUANTIZATION FOR DEEP LEARNING INFERENCE: PRINCIPLES AND EMPIRICAL EVALUATION” by Hao Wu et al., and gives a basic introduction to quantization reviewing different approaches to improve the efficiency of neural network inferencing, such as:

• ​Scale quantization

• ​Affine quantization

• ​Post-training quantization 

• ​Quantization-aware training