llama.cpp

llama.cpp is an open-source C/C++ library and toolset for running large language model inference locally or in the cloud with minimal setup and high performance across many hardware platforms.

Tip

  1. Check /common/data/models/ first. Before you download a model from Hugging Face, see whether it is already there. The shared copy works exactly like your own and doesn’t use any of your storage quota.

  2. If the model you want isn’t there and you plan to use it regularly, submit a request via 4help and we can add it. To try a model out, download it yourself instead (see Running a model that is not in /common).

  3. Either way, you must accept the model’s terms and conditions with your Hugging Face account.

Tip

When serving a model that is in /common, use its path rather than its Hugging Face name. E.g. use llama-server --model /common/data/models/unsloth--Qwen3.8-27B-GGUF--UD-Q4_K_M/Qwen3.8-27B-UD-Q4_K_M.gguf rather than llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M. The model loads immediately instead of being downloaded again for every job.

How llama-server runs on ARC

module load llama.cpp provides a llama-server command that runs inside an Apptainer container. This has two consequences:

  • Only some paths are visible to the server: /common/data/models, /projects, and $TMPDIR. Your home directory is not available, so --model ~/models/my-model.gguf will fail with a “file not found” error. Keep your own GGUF files in /projects or copy them to $TMPDIR.

  • Environment variables must be prefixed with APPTAINERENV_ to reach the server. For example, set APPTAINERENV_HF_TOKEN rather than HF_TOKEN. Variables without the prefix are ignored.

Running your own LLM using llama.cpp

The following example Slurm script launches a llama.cpp instance running the model unsloth/Qwen3.8-27B-GGUF (UD-Q4_K_M) on one NVIDIA L40s GPU on the Falcon cluster. Specifications include a job duration limited to 1 hour, the server listens on a free port chosen by llama.cpp (--port 0), and a random API key is generated for each job and printed to the log. You should adjust the settings to select the model and number of GPUs you need. By default, llama.cpp uses the largest context length that fits in GPU memory; set --ctx-size to choose a different one. If you need the server for longer, increase --time, and when you are done, stop the job with scancel <jobid> so the GPU is freed for others.

#!/bin/bash
#SBATCH --account=<account_name>
#SBATCH --partition=l40s_normal_q
#SBATCH --time=1:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=16
#SBATCH --gres gpu:l40s:1
#SBATCH --output=qwen3.8-27b.log
#SBATCH --mail-type=BEGIN

module load llama.cpp

# Generate a random API key for this job and print it to the log
export APPTAINERENV_LLAMA_API_KEY=$(< /proc/sys/kernel/random/uuid)
echo "API key: $APPTAINERENV_LLAMA_API_KEY"
echo "Node: $(hostname -s)"

llama-server --model /common/data/models/unsloth--Qwen3.8-27B-GGUF--UD-Q4_K_M/Qwen3.8-27B-UD-Q4_K_M.gguf \
--port 0

Run the Slurm script using sbatch myscript.sh and monitor the status of the job using squeue. Slurm emails you when the job starts; allow a few more minutes after that for the model to load. Once the endpoint is ready, the log file will indicate:

API key: 9f1c2a7e-4b3d-4e8a-a6f2-0d5c7b1e3a94
Node: fal036
...
srv  server_run_i: model loaded
srv  server_on_li: listening on http://0.0.0.0:34773

The Node line is the compute node your server runs on, and the number at the end of the listening on line is its port. The node, port, and API key can change each time the job runs. To show all three, run grep -E "API key|Node|listening on" qwen3.8-27b.log.

The script uses --port 0 and a generated API key for two reasons:

  • --port 0 avoids port conflicts. Other users’ servers may run on the same compute node. If you choose a fixed port such as 8000 and it is already taken, llama.cpp fails only after loading the model, which can take several minutes. With --port 0, llama.cpp always picks a free port.

  • The API key keeps others out. Without one, anyone on the cluster can send requests to your server, and llama.cpp logs security: no API key is set. Pass the key in the APPTAINERENV_LLAMA_API_KEY environment variable, not with --api-key: every user on the node can see command-line arguments with ps, but not environment variables.

Tip

To keep the same API key across jobs, create a key file once with (umask 077; cat /proc/sys/kernel/random/uuid > ~/.llama-api-key), then load it in your script instead of generating a new key: export APPTAINERENV_LLAMA_API_KEY=$(< ~/.llama-api-key).

At this point, you can use the OpenAI API to submit queries. Use the node, port, and API key from your log. In this example, the node is fal036, the port is 34773, and the API key is 9f1c2a7e-4b3d-4e8a-a6f2-0d5c7b1e3a94. Therefore, the following query will work from anywhere within the Falcon cluster.

curl http://fal036:34773/v1/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer 9f1c2a7e-4b3d-4e8a-a6f2-0d5c7b1e3a94" \
  -d '{
    "prompt": "Why is the sky blue?",
    "max_tokens": 2048,
    "temperature": 1,
    "top_p": 0.9,
    "seed": 10
  }'

Qwen3.8 is a reasoning model: it thinks before it answers, and the thinking counts toward max_tokens. If max_tokens is too low, the response can stop before the answer begins.

If you wish to connect software running on your computer to the LLM running on the compute node of the cluster, you must run SSH port forwarding to redirect the network traffic from your computer to the compute node via the login node. For example:

ssh -N -L 8000:fal036:34773 user@falcon2.arc.vt.edu

This forwards port 8000 on your computer to port 34773 on fal036. Replace fal036 and 34773 with your node and the port from your log.

Now you can use the OpenAI API to submit queries via localhost on your computer.

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer 9f1c2a7e-4b3d-4e8a-a6f2-0d5c7b1e3a94" \
  -d '{
    "prompt": "Why is the sky blue?",
    "max_tokens": 2048,
    "temperature": 1,
    "top_p": 0.9,
    "seed": 10
  }'

OpenAI chat completion example. Run this from your computer:

import argparse

from openai import OpenAI

# Modify OpenAI's API key and API base to use the server.
openai_api_key = "9f1c2a7e-4b3d-4e8a-a6f2-0d5c7b1e3a94"
openai_api_base = "http://localhost:8000/v1"

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is Virginia Tech known for?"},
]


def parse_args():
    parser = argparse.ArgumentParser(description="Client for API server")
    parser.add_argument(
        "--stream", action="store_true", help="Enable streaming response"
    )
    return parser.parse_args()


def main(args):
    client = OpenAI(
        api_key=openai_api_key,
        base_url=openai_api_base,
    )

    models = client.models.list()
    model = models.data[0].id

    # Chat Completion API
    chat_completion = client.chat.completions.create(
        messages=messages,
        model=model,
        stream=args.stream,
    )

    print("-" * 50)
    print("Chat completion results:")
    if args.stream:
        for c in chat_completion:
            print(c)
    else:
        print(chat_completion)
    print("-" * 50)


if __name__ == "__main__":
    args = parse_args()
    main(args)

See the llama.cpp server documentation for the full list of endpoints and options.

Running a model that is not in /common

To try a model that isn’t in /common/data/models, you can have llama.cpp download it directly from Hugging Face with -hf. Download it to the job’s local scratch space ($TMPDIR) so that it does not count against your storage quota:

interact -A <account_name> --partition l40s_normal_q --gres gpu:l40s:1 --time 1:00:00

module load llama.cpp
export APPTAINERENV_HF_HOME=$TMPDIR
export APPTAINERENV_LLAMA_API_KEY=$(< /proc/sys/kernel/random/uuid)
echo "API key: $APPTAINERENV_LLAMA_API_KEY"
echo "Node: $(hostname -s)"
llama-server -hf bartowski/OpenGVLab_InternVL3_5-8B-GGUF --port 0

The output shows your API key, where the model was downloaded, and which port the server is listening on:

API key: 9f1c2a7e-4b3d-4e8a-a6f2-0d5c7b1e3a94
Node: fal036
...
srv    load_model: loaded multimodal model, '/localscratch/728082/hub/models--bartowski--OpenGVLab_InternVL3_5-8B-GGUF/snapshots/.../mmproj-OpenGVLab_InternVL3_5-8B-bf16.gguf'
srv  server_run_i: model loaded
srv  server_on_li: listening on http://0.0.0.0:34773

Some notes:

  • -hf also downloads the multimodal projector (mmproj) file for vision models, so image input works without extra flags.

  • $TMPDIR is deleted when the job ends, so the model is downloaded again in every job. This is fine for trying out a model. If you plan to use it regularly, request that it be added to /common, or download it once to /projects and use --model.

  • For gated models (ones that require you to accept terms on Hugging Face), also set export APPTAINERENV_HF_TOKEN=<your_hf_token>. See Hugging Face for how to create a token.