Llama.cpp on PS5: How to Run Local AI at 50+ Tokens/Sec

Running Llama.cpp on a PlayStation 5 (PS5) allows you to turn your gaming console into a local AI workstation capable of running large language models like Llama 5 8B at speeds exceeding 50 tokens per second. By utilizing the console's 16GB of high-bandwidth GDDR6 memory and a specialized Linux environment, you can execute private AI queries without needing an expensive PC graphics card. This setup typically takes about 30 to 60 minutes to configure once you have the necessary custom firmware access.

Why is Llama.cpp important for local AI?

Llama.cpp is a software library written in C++ that allows you to run AI models on standard hardware instead of requiring massive data centers. It uses a technique called quantization (compressing large AI models into smaller files) to make them fit into the memory of everyday devices. This means you can run powerful assistants entirely offline, ensuring your data never leaves your local network.

In our experience, Llama.cpp is the most reliable way for beginners to start with local AI because it supports almost every type of processor. It specifically focuses on "inference," which is the process of an AI model generating a response to your prompt. Because the code is lightweight and highly efficient, it can extract high performance from hardware that wasn't originally designed for AI.

The software supports GGUF (GPT-Generated Unified Format), which is a specific file type that stores all the information an AI needs to function. You can find thousands of these GGUF files on platforms like Hugging Face, ranging from small conversational models to massive reasoning engines. Using Llama.cpp means you aren't locked into a single provider like OpenAI or Google.

How does the PS5 hardware handle AI tasks?

The PS5 contains a powerful APU (Accelerated Processing Unit - a single chip that combines a central processor and a graphics processor). This chip is built on AMD's RDNA 2 architecture, which is very similar to the hardware used in high-end gaming PCs. The most important feature for AI is the 16GB of GDDR6 RAM (Random Access Memory), which provides the high speeds needed to move data quickly.

Standard computers often struggle with AI because the data has to travel between the CPU and a separate graphics card. The PS5 uses a unified memory architecture, meaning the entire 16GB can be accessed by both the processor and the graphics units simultaneously. This reduces "bottlenecks" (points where data gets stuck) and allows the AI to "think" much faster than a typical laptop.

To use this hardware for AI, you must run a version of the Linux operating system on the console. Since the PS5's native software is locked down for gaming, Linux provides the "drivers" (software that tells the hardware how to work) needed to talk to the GPU. Once Linux is running, Llama.cpp treats the PS5 like a standard AMD-based computer.

What do you need to get started?

Before trying to run AI on a console, you need to ensure you have the right environment. This process is more advanced than installing a standard app, so don't worry if it feels complex at first.

What You'll Need:

A PS5 console with a firmware version that supports Linux installation (typically older versions).
A USB drive (32GB or larger) to act as your Linux boot drive.
A USB keyboard and mouse for entering commands.
A basic understanding of the Terminal (a text-based interface for giving the computer instructions).

How do you install Llama.cpp on the PS5?

Once you have Linux running on your PS5, you need to compile the software. Compiling is the process of turning human-readable code into instructions the PS5 processor can understand.

Step 1: Install the necessary tools Open your terminal and type the following command to install the building blocks for the software:

# Update the package list and install the C++ compiler and Git
sudo apt update && sudo apt install build-essential git cmake -y

What you should see: A long list of text as your system downloads and installs the tools.

Step 2: Download the Llama.cpp code You will "clone" (copy) the latest version of the code from the official repository.

# Clone the repository from GitHub
git clone https://github.com/ggerganov/llama.cpp
# Enter the folder you just created
cd llama.cpp

What you should see: A message saying "Cloning into 'llama.cpp'..." followed by a progress percentage.

Step 3: Build the software for the PS5 GPU Since the PS5 uses AMD hardware, you want to use the HIP (Heterogeneous-compute Interface for Portability) backend to fine-tune performance.

# Create a build directory and compile using HIP for AMD hardware
mkdir build
cd build
cmake .. -DGGML_HIPBLAS=ON
cmake --build . --config Release

What you should see: A series of percentage updates (1% to 100%) as the code compiles.

How do you run your first AI model?

Now that the software is ready, you need a model file. For the PS5, we recommend using Llama 5 8B or Claude Sonnet 4-equivalent open-source models in GGUF format.

Step 1: Download a model You can use a tool called wget to download a pre-quantized model directly to your PS5.

# Download a Llama 5 8B model file (roughly 5GB)
wget https://huggingface.co/models/llama-5-8b-q4_k_m.gguf

What you should see: A progress bar showing the download speed and estimated time remaining.

Step 2: Start the AI Run the model using the command line. The -m flag tells the program which model to use, and -ngl 33 tells it to move the AI layers to the GPU.

# Run the model and start a conversation
./bin/llama-cli -m llama-5-8b-q4_k_m.gguf -ngl 33 -p "Hello! How can you help me today?"

What you should see: The AI will begin typing a response to your greeting almost instantly.

Which common mistakes should you avoid?

It is normal to run into errors when working with experimental hardware like a game console. One common issue is "Out of Memory" (OOM) errors, which happen if you try to load a model that is too large for the PS5's 16GB of RAM. Always check the file size; if the model is larger than 12GB, the system will likely crash because Linux also needs memory to run.

Another typical mistake is forgetting to enable the GPU layers. If you don't include the -ngl (number of GPU layers) flag in your command, the PS5 will use its CPU instead of its graphics chip. This will make the AI respond very slowly, often taking several seconds per word rather than the instant speeds the GPU can provide.

Finally, ensure your PS5 is well-ventilated. Running AI models is computationally intensive and generates as much heat as playing a high-end game like God of War. If the console gets too hot, it may slow down to protect itself, which will decrease your tokens per second (the speed at which the AI generates text).

Next Steps

Once you have Llama.cpp running, you can explore more advanced features like setting up a local web server. This allows you to chat with your PS5 AI from your phone or laptop while the console does all the heavy lifting in the background. You might also want to experiment with different models, such as Claude Opus 4.5 or Sonnet 4 variants, which are often converted to GGUF format by the community for local use.

If you want to dive deeper into the technical details of how these models are built and how to improve performance further, you should check out the documentation.

official llama.cpp documentation