The ORACLE-BASE Blog

A few months ago I wrote a post about running Large Language Models (LLMs) locally using Ollama and Open WebUI. That works well, and feels very much like how you might want to run an LLM on a server in your organisation, but it’s not exactly friendly if you are not used to getting your hands dirty with virtual machines (VMs) and Docker/Podman.

I was watching a YouTube video where in passing they mentioned LM Studio, and I thought I would take a look. It is exactly what the average Windows/Mac/Linux user would find comfortable to run LLMs locally. Just download the app and off you go. No messing about.

Performance

The performance is going to vary depending on your computer, GPU and the model you choose, but it is much more lightweight than my Ollama example.

I’m using the same hardware I used for my Ollama tests. For that I was running a VM and a Podman container, and the extra overhead made it pretty sluggish on my crappy old laptop with a bad GPU. LM Studio was much faster in comparison, no doubt because of the reduced complexity.

I suspect most people will find it very useable on their existing kit, provided it is less than about 5 years old. Or in my case an 8 year old laptop, which was quite good in its day. 🙂

Getting started is easy

It’s really simple. Download it from here.

Once installed it will ask you if you want to download your first model. It defaults to “Gemma 3 4b”, which is based on Google Gemini, so it’s a pretty good place to start. Once that’s downloaded you can start a new chat by clicking a button, and can organise previous chats into folders. The interface is pretty simple and self explanatory.

If you want to try out some different models, click the “Search Model” button on the left toolbar, and pick something you fancy. I’ve found the “granite-4-h-tiny” from IBM to be really quick. It gives quite abbreviated answers, so you might prefer something more verbose, but you have hundreds of models to choose from, so play around and find something you prefer. In my Ollama post I gave examples of using these models.

# General models.
ollama pull llama3
ollama pull deepseek-r1
ollama pull gpt-oss:20b

# Code-focussed models.
ollama pull codellama
ollama pull deepseek-coder
ollama pull codestral

Silly example

To give you an idea of how it looks, here is a silly example. I asked the following question with 2 engines.

“How do I write a python script that accepts command line arguments?”

Here is the result using “granite-4-h-tiny”. Notice the model selection (1), the question I asked (2) and the output (3).

Here is the result of the same question using “gemma-3-4b”. It gave a different answer. I asked if that was the recommended way, and it said no, and showed me a different answer that looked more like that produced by “granite-4-h-tiny”. 🙂

You can refine your answer by asking additional questions in the same chat, just like you would on a web-based LLM.

Turn on the local API server

• Click on the Developer icon in the left-hand toolbar.
• Click on the “Local Server” tab.
• Enable the local server using the toggle on the top-left.
• Load the default model of choice using the “Load Model” button on the top-left.

The API should now be available on port 1234 by default.

$ curl http://localhost:1234/api/v1/models
{
  "models": [
    {
      "type": "llm",
      "publisher": "google",
      "key": "google/gemma-3-4b",
      "display_name": "Gemma 3 4B",
      "architecture": "gemma3",
      "quantization": {
        "name": "Q4_K_M",
        "bits_per_weight": 4
      },
      "size_bytes": 3341081486,
      "params_string": "4B",
      "loaded_instances": [
        {
          "id": "google/gemma-3-4b",
          "config": {
            "context_length": 4096,
            "eval_batch_size": 512,
            "flash_attention": true,
            "offload_kv_cache_to_gpu": true
          }
        }
      ],
      "max_context_length": 131072,
      "format": "gguf",
      "capabilities": {
        "vision": true,
        "trained_for_tool_use": false
      },
      "description": null,
      "variants": [
        "google/gemma-3-4b@q4_k_m"
      ],
      "selected_variant": "google/gemma-3-4b@q4_k_m"
    },
    {
      "type": "llm",
      "publisher": "ibm",
      "key": "ibm/granite-4-h-tiny",
      "display_name": "Granite 4 H Tiny",
      "architecture": "granitehybrid",
      "quantization": {
        "name": "Q4_K_M",
        "bits_per_weight": 4
      },
      "size_bytes": 4231029544,
      "params_string": "7B",
      "loaded_instances": [],
      "max_context_length": 1048576,
      "format": "gguf",
      "capabilities": {
        "vision": false,
        "trained_for_tool_use": true
      },
      "description": null,
      "variants": [
        "ibm/granite-4-h-tiny@q4_k_m"
      ],
      "selected_variant": "ibm/granite-4-h-tiny@q4_k_m"
    },
    {
      "type": "embedding",
      "publisher": "nomic-ai",
      "key": "text-embedding-nomic-embed-text-v1.5",
      "display_name": "Nomic Embed Text v1.5",
      "quantization": {
        "name": "Q4_K_M",
        "bits_per_weight": 4
      },
      "size_bytes": 84106624,
      "params_string": null,
      "loaded_instances": [],
      "max_context_length": 2048,
      "format": "gguf"
    }
  ]
}
$

Why bother running LLMs locally?

To repeat what I said in my previous post…

There are potential issues with using a cloud-based LLM.

• Security: Passing sensitive data or your company intellectual property to the cloud could be a problem. Using a local LLM means you can do pretty much what you want with it. It also means your prompts are not being stored and available to the outside world when the provider of your cloud-based LLM inevitably gets hacked, or sells your details.
• Consistency: The cloud vendors update their models from time to time, and you may find a “new and improved” version of a model works less well for what you are doing. Just look what the “upgrade” to ChatGPT 5 broke. Using a local LLM means you are in control of if/when you update models.
• Cost: You have the cost of the local kit, but after that it’s all “free”. Running a model requires a lot less kit than training a model. Depending on your needs, maybe running something locally will make more sense than paying for a cloud subscription. It can get really expensive if you require a large call volume on a cloud subscription.
• Choice: There are lots of open models to choose from (see here). Maybe one of the more targeted models will work better for you than an all purpose model.

Overall

Your experience will vary depending your kit and the model you pick. There are some really big models, but they will need a lot more memory and a good GPU to function. Even these small models are really useful though, and importantly will run on crappy kit.

I have no intention of paying for a cloud subscription when I can run these for free and not be spied on. Have fun!

Cheers

Tim…