#GoogleQuant is a game changer for #LocalAI. Once we see models incorporating it, one can run larger than 30b on 32 GB of #VRAM on processor platforms like #Intel.
Maybe even a 30b qant4 on 24GB of ram on say an #Intel B60 GB.
Google has this advantage to use it in the #GEMMA4 open source series.
1
0
0
0