Using Local Models with NORA

NORA’s built-in AI nodes (AI Agent, AI Router, AI Autonomous Agent) connect to cloud providers — Anthropic, OpenAI, and Google Gemini. But you can run any local model — including Gemma 4, Llama, Mistral, Phi, and others — using the Custom Script Agent (CSA) node.

This guide walks through exactly how to do that.

How Local Models Work in NORA

The CSA node runs a Python or Node.js script you write. Your script manages all API calls — NORA handles workflow orchestration, routing, cost display, and tool execution. Because your script makes the LLM call directly, you can point it at any inference server running locally on your machine.

Local model tools like Ollama and LM Studio expose an OpenAI-compatible API. This means you can use the same openai Python SDK you already know — just change the base_url to point at your local server instead of OpenAI’s API.

No cloud API key required. Local models run entirely on your machine. Inference is free, offline, and private.

Before You Begin

You’ll need Python 3.8 or later installed on your machine. Open a terminal and check:

python --version

If Python isn’t installed, download it from python.org. Make sure to check “Add Python to PATH” during installation.

You’ll also need the openai package:

pip install openai

Option 1: Ollama (Recommended)

Ollama is the easiest way to run local models on Windows. It downloads and manages models, and exposes a local API that works out of the box.

Step 1: Install Ollama

Download and install Ollama from ollama.com. After installation, Ollama runs as a background service at http://localhost:11434.

Step 2: Download a Model

Open a terminal (PowerShell or CMD) and pull your chosen model:

# Gemma 4 — Google's open-weight model (pick one based on your GPU)
ollama pull gemma4:e4b      # Recommended — ~9.6 GB, best quality for 8 GB VRAM
ollama pull gemma4:e2b      # Lighter option — ~7.2 GB, fits fully in 8 GB VRAM
ollama pull gemma4:26b      # Large — ~18 GB, needs 24 GB+ VRAM

# Other popular models
ollama pull llama3.3        # Meta Llama 3.3
ollama pull mistral         # Mistral 7B
ollama pull phi4            # Microsoft Phi-4 (14B)
ollama pull qwen3:14b       # Qwen 3 14B

Not sure which to pick? See Which Gemma 4 Variant to Use? below.

First run will download the model weights — this can take several minutes depending on model size and your connection. Once downloaded, the model is cached and subsequent runs start instantly.

To verify it’s working:

ollama run gemma4:e4b "Say hello"

Step 3: Create a CSA Script

Create a new Python script (e.g., gemma4_agent.py) in a folder of your choice:

#!/usr/bin/env python3
"""
Local Model Agent — Gemma 4 via Ollama
Uses the openai SDK's base_url override to call Ollama's OpenAI-compatible API.
No API key required.
"""

import sys
import json
import time

# ── Helpers ──────────────────────────────────────────────────────────────────

def emit(obj):
    print(json.dumps(obj), flush=True)

def emit_log(message, level="info"):
    emit({"event": "log", "message": message, "level": level})

def emit_thinking(iteration, message):
    emit({"event": "thinking", "iteration": iteration, "message": message})

def emit_message(content):
    emit({"event": "message", "role": "assistant", "content": content})

def emit_complete(route="success", output=""):
    emit({"event": "complete", "route": route, "output": output})

def emit_error(message):
    emit({"event": "error", "message": message})

def read_message():
    line = sys.stdin.readline()
    if not line:
        return None
    return json.loads(line.strip())

# ── Main ─────────────────────────────────────────────────────────────────────

def main():
    # Wait for the 'start' message from NORA
    msg = read_message()
    if not msg or msg.get("action") != "start":
        emit_error("Expected 'start' message")
        return

    config = msg["config"]
    user_request = config.get("userRequest", "")
    history = config.get("conversationHistory", [])

    emit_log("Connecting to local Ollama model...")
    emit_thinking(1, "Calling Gemma 4 locally...")

    # Build the OpenAI client pointed at Ollama
    try:
        from openai import OpenAI
    except ImportError:
        emit_error("openai package not installed. Run: pip install openai")
        return

    client = OpenAI(
        base_url="http://localhost:11434/v1",
        api_key="ollama"          # required by the SDK but ignored by Ollama
    )

    # Build message list (supports multi-turn conversation)
    messages = []
    for h in history:
        messages.append({"role": h["role"], "content": h["content"]})
    messages.append({"role": "user", "content": user_request})

    try:
        response = client.chat.completions.create(
            model="gemma4:e4b",   # change to gemma4:e2b or any model you have pulled
            messages=messages,
        )
    except Exception as e:
        emit_error(f"Ollama call failed: {str(e)}")
        return

    result = response.choices[0].message.content

    # Local models are free — emit $0 cost so the UI shows something
    emit({"event": "cost", "inputTokens": response.usage.prompt_tokens,
          "outputTokens": response.usage.completion_tokens,
          "model": "gemma4:e4b", "usd": 0.0})

    emit_message(result)
    emit_complete("success", result)


if __name__ == "__main__":
    try:
        main()
    except KeyboardInterrupt:
        sys.exit(0)
    except Exception as e:
        emit_error(f"Agent crashed: {str(e)}")
        sys.exit(1)

Step 4: Configure the CSA Node

1. Add a Custom Script Agent node to your canvas
2. Set Script Path to the full path of your gemma4_agent.py
3. Set Script Type to Python
4. Set Working Directory to the folder containing the script
5. Type your task into User Request
6. Click Run

That’s it. The script calls your local Ollama instance and streams the response back to NORA.

Option 2: LM Studio

LM Studio provides a GUI for downloading and running models, and exposes an OpenAI-compatible API at http://localhost:1234/v1 when the local server is enabled.

Setup

1. Download LM Studio from lmstudio.ai
2. Open LM Studio → Discover tab → search for “gemma 4” and download
3. Go to Local Server tab and click Start Server

Script (LM Studio)

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:1234/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="gemma4",           # LM Studio uses the loaded model name
    messages=[{"role": "user", "content": user_request}]
)

result = response.choices[0].message.content
emit_message(result)
emit_complete("success", result)

Which Gemma 4 Variant to Use?

Recommendation for most users: start with gemma4:e4b (the default). If it feels slow, switch to gemma4:e2b. Run ollama list to see what models you’ve already downloaded.

No GPU? Ollama can run models on CPU, but it will be significantly slower — expect 1–5 tokens per second for 12B models on a modern CPU.

Storing models on an external drive?

By default, Ollama saves model weights to C:\Users\<username>\.ollama\models. You can redirect downloads to any drive using the OLLAMA_MODELS environment variable. Order matters — follow these steps exactly or Ollama will silently ignore the setting.

Step 1 — Quit Ollama completely
Right-click the Ollama tray icon and choose Quit. Then confirm it’s stopped:
powershell
Get-Process ollama -ErrorAction SilentlyContinue
If the command returns nothing, Ollama is stopped. If it returns a process, it’s still running — quit again and recheck.

Step 2 — Set the environment variable (PowerShell)
“`powershell

Verify it was saved:

``
You should seeF:\ollama-modelsprinted back. Alternatively, set it via Windows Settings: search for **"environment variables"** → **Edit environment variables for your account** → createOLLAMA_MODELS` with your desired path.

Step 3 — Relaunch Ollama from the Start menu
Open Ollama from the Start menu (or run it as Administrator if on a restricted account). Do not launch it from the same terminal where you set the variable — open it from the Start menu so it inherits the persistent user environment.

Step 4 — Pull your model
Run ollama pull gemma4:e4b. Check that F:\ollama-models is created and growing:
powershell
Test-Path "F:\ollama-models" # should return True

If you already started a pull before doing this (and it went to C:), delete the partial download first:
powershell
Remove-Item "$env:USERPROFILE\.ollama\models" -Recurse -Force
Then pull again — it will restart cleanly from F:.

A USB 3.0+ external SSD works well. Model loading is a one-time read at startup, so drive speed only affects the initial cold-load, not inference speed once the model is in memory.

Handling Multiple Route Labels

If you configure Route Labels on your CSA node (e.g., approved, rejected, review), have your local model decide the route:

# Ask the model to end its response with one of the route labels
system_prompt = f"""You are a decision-making assistant.
At the end of your response, output exactly one of these routing decisions on its own line:
ROUTE: approved
ROUTE: rejected  
ROUTE: review"""

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": user_request}
]

response = client.chat.completions.create(model="gemma4:e4b", messages=messages)
text = response.choices[0].message.content

# Extract the route
route = "success"
for line in text.split("\n"):
    if line.startswith("ROUTE:"):
        route = line.replace("ROUTE:", "").strip().lower()
        break

emit_message(text)
emit_complete(route, text)

Cost Tracking with Local Models

Local models are free, so usd should always be 0.0. You can still emit the cost event to show token counts in the UI:

emit({
    "event": "cost",
    "inputTokens": response.usage.prompt_tokens,
    "outputTokens": response.usage.completion_tokens,
    "model": "gemma4:e4b",
    "usd": 0.0
})

The node header will show the model name and token counts, with $0.00 cost.

Installing the Required Package

If you haven’t already (see Before You Begin), install the openai package:

pip install openai

No other packages are required for the Ollama approach. If you prefer to use raw HTTP requests instead:

import requests

response = requests.post("http://localhost:11434/api/chat", json={
    "model": "gemma4:e4b",
    "messages": [{"role": "user", "content": user_request}],
    "stream": False
})
result = response.json()["message"]["content"]

Troubleshooting

“Connection refused” error
Ollama isn’t running. Open PowerShell and run ollama serve, or check that the Ollama desktop app is running in your system tray.

“Model not found” error
The model hasn’t been pulled yet. Run ollama pull gemma4:e4b in a terminal. Note that gemma4:12b is not a valid tag — the correct tags are e2b, e4b, 26b, and 31b.

Responses are very slow
You may be running on CPU or with significant RAM offloading. Run ollama ps to see which layers are on GPU vs RAM. Consider switching to gemma4:e2b (7.2 GB) if gemma4:e4b is exceeding your VRAM.

I have lots of browser tabs and apps open – does that matter?
Yes. Local models share the same machine resources as everything else. Heavy browser/app usage can reduce available RAM/VRAM and increase CPU contention, which directly impacts response speed and reliability.

What typically happens under load:
– Slower first response: model warm-up takes longer
– Lower tokens/sec: generation slows when CPU/GPU are busy
– More RAM offload: if VRAM is tight, Ollama shifts more layers to system RAM
– Intermittent errors: low free memory can trigger model load/generation failures

For CSA workflows in NORA specifically:
– Each CSA node call still needs enough free memory at runtime
– Workflows with multiple AI-heavy steps can feel much slower if other apps are consuming resources
– If you run parallel AI branches, resource pressure increases further

Best practice before a heavy run:
1. Close high-memory browser tabs and apps
2. Keep only NORA + Ollama + essential tools open
3. If needed, switch from gemma4:e4b to gemma4:e2b for better stability
4. Run one workflow first, then scale up concurrency

Script crashes on import
The openai package isn’t installed. Run pip install openai in a terminal.

Want to test outside of NORA first?
Use the Ollama CLI: ollama run gemma4:e4b "Your prompt here". If that works, the script will too.

Other Local Model Options

All of these use the same OpenAI-compatible API pattern — just change base_url in your script to match the local server’s address.

Related

• AI Features — Custom Script Agent — full CSA protocol reference, stdin/stdout events, tool requests
• Node Types — Custom Script Agent — node configuration fields