Lol

if this is your first time hearing about bedrocklinux.org then you're in for an adventure with this one

Round Earth v2: with some tweaking the effect becomes rather nice, I love how distant mountains & clouds disappear into the horizon. However, zooming around this map makes me really dizzy somehow 😅

#indiedev #gamedev #indiegames #screenshotsaturday

I will try it, but I think as long as its not generating less than like 1t/s the main bottleneck would be prompt processing for big system prompts and cold starts for existing conversations.

I'm currently working on patch that saves KV Cache to disk and I think that would help.

I mean, it works well with small context, right?

Have you tried with TurboQuant enabled?

This what happens when you implement correct tests and then ask LLM to fix the bug they were testing and it decides to simplify tests

Look at this little Qwen box :3

also as for memory management, i have set vram to 4gb of global memory, so llama.cpp falls back to offloading kv cache and weights to ram while still using gpu for compute. As far as i tested there is no performance implications as ram and vram is physically same memory

This set up hot swappes small models instantly and takes around 35s for bigger ones.

I still need to finalize it and especially what models i run. But Im very happy so far.

So my set up for Strix Halo so far:

1. Llama.cpp with --no-mmap flag (otherwise big models take forever to load)

2. llama-swap with following config:

It can either run qwen122b q4 (coding) or hermes4 70b (for RP and general conversations)

OR

It runs one or many of: qwen9b, lfm24b.

I'm gonna try UD-Q5_K_XL :3

I think hermes4-70b is very nice for conversations and reasoning though

Hbu @astrra.space ? ^^

Alr alr, fair.

Any thoughts on nemotron 3 super?

Also, regarding inference engine, I'm very curios about mistral.rs. Just.. No rocm support :(

So llama.cpp that is for now

I'm trying Qwen3 coder next, but the one you suggested sounds good. Going to pull this rn

@astrra.space hey. So what is the best model on fw desktop you ended up with?

Archipelago
#ProcGen

🙀

I mean this is generally my first reflex when dealing with llm. But then I remember I should work

I just ctrl+d

Soft clouds

I spent 2h figuring out what caused NixOS to default ~ to /. I gave up and hardcoded $HOME to config as env var. :/

My day started with my laptop suddenly turning off and turning unusable after startup

I would say it is rather a right size :)

there is no way im not buying it now. i got my fw12 and so far its being absolutely perfect. :)