Allow multiple model slots even when detected VRAM is 0

[krissetto]

Right now if the detected VRAM is 0mb, we only allow 1 model to run at a time. In workflows where we need to query multiple models this can be quite the limitation because DMR will constantly swap the models in and out of memory even when they'd both fit, thrashing the storage and hugely slowing down the UX.

In my specific example, I'm using AMD Strix Halo with 128gb of ram with Vulkan on the llama.cpp backend.

I think it'd be nice if either:

• DMR didn't limit the number of model slots;
• There was an option to configure this behavior system-wide

Any thoughts around this?

[doringeman]

Hey @krissetto, thanks! This is something we wanted to do in the past but didn't get the time to actually do it.
WDYT about #454? It's not super complete, as currently you can't query whether you have disabled the loader memory check, but it might be useful for you.
CC @ericcurtin @ilopezluna

[krissetto]

@doringeman I gave #454 a look. It could certainly work, but it would place the burden of knowing if we're in a "unknown vram" scenario on the user/client and i'm not convinced that's the best approach to maintain a simple UX for users.

For context:

Right now I'm writing and testing new RAG strategies in cagent, making sure everything works well also with DMR.
This includes defining:

• models for the agentic workflow of answering a user;
• a model to generate semantic meaning of snippets of code or documents;
• a model to generate the embeddings for the semantic meaning;
• a re-ranking model to re-order results before the rag returns them to the main LLM;

To use the feature in #454 to get around the issue with my strix-halo machine I'd have to configure each and every model I use in that way explicitly, the current sequence of unload-reload on each step of the workflow above becomes quite painful and slow. It's definitely something I can work with to test things out, but IMHO it'd still be best to have a more robust strategy around this.

I'm thinking maybe something i can configure system-wide, for example, so it's a "set and forget" type of setting needed only on my system. If i have to set the setting on the cagent configuration side, it also becomes a bit more problematic to share it with others (as maybe they can't run with that flag because they actually go over the VRAM limit). I can make sure cagent filters out these sort of settings when pushing to hub but it doesn't feel very right

[ericcurtin]

I would say remove all memory estimation long-term, almost impossible to get that right multi-platform.

When I open too many Chrome tabs on my machine and use all my memory, the solution isn't to prevent opening tabs if we estimate memory is low, the solution is... Close some tabs... Or let the OS kill the most expensive tabs

[krissetto]

@ericcurtin I'd tend to agree, after all its what already happens now if you misconfigure the llama.cpp backend even with a single model

[doringeman]

To use the feature in #454 to get around the issue with my strix-halo machine I'd have to configure each and every model I use in that way explicitly

You can use docker model configure --disable-loader-memory-check only and it'll disable it globally.

[doringeman]

I'll get back to this. I created that PR ASAP to see whether you can make use of it. I don't mind converting it to draft and revisit it.

[krissetto]

For now i manually removed the slot limitation logic when detected VRAM is unknown and I'm running a local build, just to unblock myself with other work and allow time to get the proper solution into the project instead of rushing things