Gemini not free?

OpenRouter sucks now?

No free alternatives?

Well, shut your mouth because I'll show you how to run an LLM on your own!

About this guide:

You just need a computer for this. It will take 10 GB of storage, but I think your GPU doesn't have to be that good.

What you will be doing is:

1: Get llama.cpp

2: Get cmake

3: Get your LLM in gguf format

4: Compile llama.cpp with cmake

5: Run llama-server with the model

6: Set up your Janitor Proxy

7: Congrats! You are very smarts!

You know it is bad when even I post something about this. I used to use gemini, but the severe lack of free LLM APIs is horrible.

Get llama.cpp

Go to https://github.com/ggml-org/llama.cpp, click the "Code" button, and click "Download ZIP". Extract it anywhere you want.

Get cmake

Go to https://cmake.org/download/, get the binary version for your OS (the .zip or .tar.gz for your architecture under "Binary Distributions"), and extract it inside your llama.cpp folder and call it cmakebin.

Get your LLM in gguf format

Get any LLM you want in GGUF format.

I recommend https://huggingface.co/DavidAU/Llama-3.2-8X3B-MOE-Dark-Champion-Instruct-uncensored-abliterated-18.4B-GGUF as it is relatively lightweight, fast, uncensored, and specialized in NSFW. In the model card, check the right panel as it has a hardware compatibility table. When you decide which version you want to get, click it, and hit download.

I recommend you get Q3_K_M or Q4_K_M. I don't know what quantization, K or M mean, so just ask an AI for which version is best for your hardware.

Put your model in a folder called "mymodels" on the llama.cpp folder from earlier.

Compile llama.cpp with cmake

Open a terminal on your llama.cpp folder (right click the folder and hit the option for opening with terminal), and run the following commands:

Windows:

.\cmakebin\bin\cmake.exe .

.\cmakebin\bin\cmake.exe --build . -j 12

MacOS or Linux:

./cmakebin/bin/cmake .

./cmakebin/bin/cmake --build . -j 12

Run llama-server with the model

Every time you want to start the server, run a terminal on the bin folder of llama.cpp (the bin folder, not the root llama.cpp folder), and run the following command:

Windows:

.\llama-server.exe --models-dir ..\mymodels --api-key foo

MacOS or Linux:

./llama-server --models-dir ../mymodels --api-key foo

Copy the server URL from the terminal by selecting it, right clicking it, and hitting "Copy". The server URL is on the line that says: "router server is listening on xxx"

Set up your Janitor Proxy

On a chat with any character with proxy, hit the proxy button on the top right (the one that says "using xxx"). Click "Add Configuration" or edit an existing one. Paste the server URL from earlier on "Proxy URL" and add "/chat/completions". On "Model Name" put the file name of the model you downloaded without the extension (if it is called "model.gguf", put model). On "Api Key" put "foo". Save the configuration and your settings, then reload the page.

Congrats! You are very smarts!

If you got here, you should proud of your intelliglence. Thanks for reading this guide and happy Ai-ing!

[[Ask this character questions if you get stuck]]