llama.cpp has vision language model support now! ❤️🔥
get started with sota VLMs (gemma 3, Qwen2.5VL, InternVL3 & more) and serve them wherever you want 🤩
learn more github.com/ggml-org/lla... 📖
If you want to ✨ speed-up & harden ✨ your RAG pipelines, use visual document retrieval models ⬇️
We have shipped a how-to guide for VDR models in Hugging Face transformers 🤗📖 huggingface.co/docs/transfo...
here's a good blog on successful DSE model MCDSE, compression and more huggingface.co/blog/marco/a...
Why do people sleep on DSE multimodal retrieval models? 👀
They're just like ColPali, but highly scalable, fast and you can even make them more efficient with binarization or matryoshka with little degradation 🪆⚡️
I collected some here huggingface.co/collections/...
I'm so hooked on @hf.co Inference Providers (specifically Qwen2.5-VL-72B) for multimodal agentic workflows with smolagents 🥹
get started ⤵️
> filter models provided by different providers
> test them through widget or Python/JS/cURL
my weekly summary on what's released in open AI is up on @hf.co huggingface.co/posts/merve/...
collection is here huggingface.co/collections/...
fan-favorite open-source PDF rendering model OlmOCR goes faster and more efficient ⚡️
RolmOCR-7B follows same recipe with OlmOCR, builds on Qwen2.5VL with training set modifications and improves accuracy & performance 🤝
huggingface.co/reducto/Rolm...
Hello friends 👋🏼
If visit Turkey this summer, know that millions of Turkish people are doing a boycott, once a week not buying anything and rest of the week only buying necessities
if you have plans, here's a post that summarizes where you should buy stuff from www.instagram.com/share/BADrkS...
SmolVLM paper is out and it's packed with great findings on training a good smol vision LM!
Andi summarized them below, give it a read if you want to see more insights 🤠
the model also has impressive OCR capabilities ⬇️
we'll give this model a test on agentic capabilities but here's an example from paper:
This model consists of a dynamic res handling MoonViT encoder, a projection layer and a 16B MoE decoder (with 2.8B active params)
the paper introduces an interesting pre-training pipeline to handle long context and the model saw 4.4T tokens arxiv.org/pdf/2504.07491
DO NOT SLEEP ON THIS MODEL
Kimi-VL-A3B-Thinking is the first ever capable open-source reasoning VLM with MIT license ❤️
> it has only 2.8B activated params 👏
> it's agentic 🔥 works on GUIs
> surpasses gpt-4o
I've put it to test (see below ⤵️) huggingface.co/spaces/moons...
InternVL3 is out 💥
> 7 ckpts with various sizes (1B to 78B)
> Built on InternViT encoder and Qwen2.5VL decoder, improves on Qwen2.5VL
> Can do reasoning, document tasks, extending to tool use and agentic capabilities 🤖
> easily use with Hugging Face transformers 🤗 huggingface.co/collections/...
Xet infra now backs 1000s of repos on @hf.co , which means we get to put on our researcher hats and peer into the bytes 👀 🤓
Xet clients chunk files (~64KB) and skip uploads of duplicate content, but what if those chunks are already in _another_ repo? We skip those too.
SmolVLM paper is out and it's packed with great findings on training a good smol vision LM!
Andi summarized them below, give it a read if you want to see more insights 🤠
X'in politikaları sebebiyle işimle alakalı post'ları burada da paylaşıyor olacağım, takip edebilirsiniz 😊
icymi I shipped a tutorial on fine-tuning vision language models on videos ⏯️
learn how to fine-tune SmolVLM2 on Video Feedback dataset 📖 github.com/merveenoyan/...
All the multimodal document retrieval models (ColPali, DSE et al) are now under visual document retrieval at @hf.co 📝🤗
take your favorite VDR model out for multimodal RAG 🤝
Smol but mighty:
• 256M delivers 80% of the performance of our 2.2B model.
• 500M hits 90%.
Both beat our SOTA 80B model from 17 months ago! 🎉
Efficiency 🤝 Performance
Explore the collection here: huggingface.co/collections/...
Blog: huggingface.co/blog/smolervlm
Introducing the smollest VLMs yet! 🤏
SmolVLM (256M & 500M) runs on <1GB GPU memory.
Fine-tune it on your laptop and run it on your toaster. 🚀
Even the 256M model outperforms our Idefics 80B (Aug '23).
How small can we go? 👀
Everything that was released passed week in open AI 🤠
> Link to all models, datasets, demos huggingface.co/collections/...
> Text-readable version is here huggingface.co/posts/merve/...
there's a new multimodal retrieval model in town 🤠
@llamaindex.bsky.social released vdr-2b-multi-v1
> uses 70% less image tokens, yet outperforming other dse-qwen2 based models
> 3x faster inference with less VRAM 💨
> shrinkable with matryoshka 🪆
huggingface.co/collections/...
What a week to open the year in open ML, all the things released at @hf.co 🤠
Here's everything released, find text-readable version here huggingface.co/posts/merve/...
All models are here huggingface.co/collections/...
ViTPose -- best open-source pose estimation model just landed to @hf.co transformers 🕺🏻💃🏻
🔖 Model collection: huggingface.co/collections/...
🔖 Notebook on how to use: colab.research.google.com/drive/1e8fcb...
🔖 Try it here: huggingface.co/spaces/hysts...
The model is very interesting, it has different encoders for different modalities each (visual prompt, text prompt, image and video) then it concatenates these to feed into LLM 💬
the output segmentation tokens are passed to SAM2, to sort of match text (captions or semantic classes) to masks ⤵️
ByteDance just dropped SA2VA: a new family of vision LMs combining Qwen2VL/InternVL and SAM2 with MIT license 💗
The models are capable of tasks involving vision-language understanding and visual referrals (referring segmentation) both for images and videos ⏯️
see the blog and our docs for more insights around native agentic skills of LLMs and getting started with smolagents, courtesy of the amazing
@m--ric.bsky.social
> Blog: hf.co/blog/smolage...
> Quickstart: huggingface.co/docs/smolage...
