The user can select the Metal device by setting the environment variable GGML_METAL_DEVICE=X where X is an integer representing the device number as displayed when llama is run. The patch for selecting the GPU works well however the speed of token generation was abysmal.
llama.cpp doesnt really go in for providing backward compatibility so i developed a patch to the objective-c module that can be enabled at build time. If you find you need it (see next few posts first though) you can find it here gist.github.com/nbx3/b367a2e...
First issue with llama.cpp's implementation is that it supports only the first detected GPU which is always the internal one, in my case an Intel UHD Graphics 630, which was is useable but very slow.
Went down a rabbit hole with llama.cpp and Metal compatibility. If you want to use llama.cpp with an x86 mac, don't use Metal is the short answer. Although Metal is supposed to be cross architecture the implementation in llama.cpp doesn't seem to be. I ran into several issues when i tried.
Well put. It's very convenient to use inquiries to delay and allow those who might be implicated, time to retire. How do other countries do this kind of thing?
Read through the llama.cpp docs, it has support for lots of GPU backends with acceleration and compute shader API libraries including Metal, BLAS, BLIS, SYCTL, MUSA, CUDA, HIP, Vulkan and CANN. Metal seems the most obvious choice as Metal 3 is supported in MacOS for both internal and eGPU.
Snowy Surrey
Discovered Ollama, great tool to simplify deployment of LLM models, well supported and with a good community. I wanted to deploy on my x86 Mac mini running Sequoia which has an eGPU attached with a RX 5700 XT installed. Not very surprisingly isn't supported. GOTO llama.cpp was the advice.
Surveillance Self-Defense is EFF's online guide to defending yourself and your friends from surveillance by using seecure technology and developing careful practices. Learn more today:
Decided to dive into the Generative AI maelstrom. I want to understand the practicalities around both using and running these models, how to adapt and use them effectively
Let's make Bluesky the home of independent news.
Unlike X, where algorithms crush links & push content you don't want, news flourishes here. Particularly news from independent & non-profit outlets.
If you prefer news away from MSM, here's a starter pack to share with valuable & verified accounts.
Overhead shot of 1000s of protesters in Rustaveli Avenue by the Georgia Parliament
And overhead shot of 1000s of protestors in Slavija Square, Belgrade
A remarkable synergy in these two photos.
Both taken tonight, but 1500 miles apart.
The first is Tbilisi, Georgia 🇬🇪
The second is Belgrade, Serbia 🇷🇸
Around 100,000 protesting in each city. People separated by geography but united in a fight to protect their democracy.
The world needs to watch.
If you haven't heard Max Richter's music, then you should... emotional, inspirational, beautifully composed. I could listen on repeat all day.
youtu.be/oNLDJp83YAQ?...
What a great cover of All the small things
www.youtube.com/watch?v=vO8a...
Tbilisi, Day 18, 7pm
Here they come again. The people of Georgia refusing to bow to regime repression.
Early today, the election monitors who oversaw the October election marched through the capital demanding fresh elections.
#GeorgiaProtests 🇬🇪🇪🇺
(🎥Credit: Publika)
We just released 1.10.0 of #OWASP #WrongSecrets, now running on Java 23, with 51 challenges, and it is now using #renovate. See github.com/OWASP/wrongs... for more details or play at www.wrongsecrets.com. Like what you see? Please give the repo a 🌟.
Averting the Digital Dark Age: How Archivists, Librarians, and Technologists Built the Web a Memory is released _today_.
You can buy a copy, or, you can read it FOR FREE via Project MUSE. Please spread the news to your colleagues, your friends, your syllabus makers.
muse.jhu.edu/book/123276
Back in 2021, I wrote about how the success of the mRNA vaccine opened to the door to way, way more than just rapidly developed vaccines - and now there's another great example of the promise of mRNA therapies 🧪🧵 1/
www.newscientist.com/article/mg25...
New publisher Li'l Factory is on the hunt for the best unpublished science fiction short stories and novellas (10-20k words) for beautifully designed ltd editions. More details, including terms can be found at lilfac.com
Spread the word! #sciencefiction #scifi #amwriting #amwritingscifi #booksky
forgejo.org is another good option for self hosting if it's git you need. But you can get free app/static site deployments and a usable URL from both digital ocean and cloud flare pages.
Great to see so many positive life changing drugs reach stage 3 and nice science journalism
New post: My substack is back, finally!
I meant to keep this to 1,000 words but I failed.
In any case, here's a round up of 5 major medical breakthroughs in 2024.
A tortoise in a pond, feeding on large mouthfuls of slimy green algae. The light is low, it is close to sunset.
Eat your greens this #TortoiseTuesday - just like this Aldabra giant tortoise, bobbing around in a pond on Eastern Grande Terre, munching on thick blobs of green algae. 🐢🌿 #FieldworkArchive
Since I've been here for a while… here are some useful features on Bluesky that you might not have discovered yet! 🧵
Just discovered One: A new #React framework simplifying cross-platform development. Target web & native with a single #Vite plugin & fully shared code.
A promising framework trying to bridge the gap between building for both the web & mobile simultaneously.
Excited to try it out!
onestack.dev
First post through PDS... 😍
