node-llama-cpp

Run AI models locally on your machine with node.js bindings for llama.cpp. Enforce a JSON schema on the model output on the generation level

What is node-llama-cpp?

Run AI models locally on your machine with node.js bindings for llama.cpp. Enforce a JSON schema on the model output on the generation level

Overview of node-llama-cpp

What it is

node-llama-cpp provides Node.js bindings for llama.cpp, enabling local execution of AI models on your machine. It includes pre-built binaries for macOS, Linux, and Windows, with a fallback to building from source using cmake if binaries are unavailable. The package supports Metal, CUDA, and Vulkan, and automatically adapts to your hardware without manual configuration. It also allows enforcement of a JSON schema on model output at the generation level.

Key features

  • Run LLMs locally on your machine - Metal, CUDA and Vulkan support - Pre-built binaries provided, with fallback to building from source without node-gyp or Python - Adapts to your hardware automatically, no need to configure anything - Enforce a model to generate output in a parseable format, like JSON, or force it to follow a specific JSON schema - Provide a model with functions it can call on demand to retrieve information or perform actions - Embedding and reranking support - Structured decisions support with any model - Safe against special token injection attacks - Full TypeScript support with complete documentation

Use cases

  • Running large language models locally for privacy or offline scenarios - Enforcing structured output (e.g., JSON) from a model for integration with applications - Building chatbots or interactive AI tools that run entirely on the user's machine - Using function calling to let a model retrieve external data or trigger actions - Embedding and reranking for search or retrieval-augmented generation pipelines

Getting started

To try it without installing, chat with a model in your terminal using a single command: bash npx -y node-llama-cpp chat To install: bash npm install node-llama-cpp For usage examples, see the repository README.

License and pricing

This project is licensed under the MIT license. It is open source and free to use.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/withcatainode-llama-cpp.svg)](https://yunodb.com/tools/withcatainode-llama-cpp?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/withcatainode-llama-cpp?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/withcatainode-llama-cpp.svg" alt="Featured on YunoDB" height="20"></a>