Local LLMs and Gemma 4: Complete Guide
A detailed guide on how local neural networks work, what HuggingFace and Ollama are, and how to run the powerful Gemma 4 model even on a regular computer.
Watch original videoWhat are local models and why are they popular?
Local models are catching up very quickly with large closed counterparts. For example, Gemma 4 performs approximately at the level of GPT-4O Mini.
- Privacy: Your data never leaves your computer.
- Offline access: The model works even without the Internet.
- Flexibility: Possibility of fine-tuning for specific tasks (writing texts, code, medicine).
1 Why should you pay attention to Gemma 4?
Google released a whole family of Gemma 4 models that have a number of significant advantages:
- Two small models (2.4B parameters): Ideal for phones, small devices (e.g., Raspberry Pi).
- Two large models (26B and 31B parameters): Designed for powerful computers. For their parameters, they show higher benchmarks than some 100-billion models (like DeepSeek V3).
- Agent capabilities (Tool Calling): The model can call functions and create full-fledged agent systems.
- Multimodality: Gemma 4 can recognize audio and images, as well as generate them.
- Open License (Apache 2): Unlike previous versions, you can use it for commercial purposes without restrictions.
2 HuggingFace: Models Marketplace
HuggingFace is the largest marketplace for local AI models (also called the accelerator or GitHub for machine learning).
When you search for a model there, file names can look intimidating. Let's break them down using Unsloth/Gemma4-24B-A4BIT-IT-GGUF as an example:
- Unsloth: The author or organization that optimized and uploaded the model.
- Gemma 4: The name of the model itself.
- 24B: Number of parameters (24 billion).
- A4BIT: Out of 24 billion parameters, only 4 billion are actively used.
- IT (Instruction Tuned): The most important parameter. It means the model is trained as a chat, i.e., understands instructions and maintains a dialogue (rather than just continuing text).
- GGUF: File format optimized for local running via Ollama or LM Studio.
3 Fine-tuning and Quantization
To run large models on home computers, two approaches are used:
This is the process of training a base model for specific tasks (medicine, programming, finance). Instead of knowing a little about everything, the model becomes an expert in one narrow field.
This is "compressing" the model by reducing the precision of numbers (for example, from 16-bit to 4-bit). Due to this, a model that initially weighs 15 GB can be compressed to 5 GB, which allows it to run on computers with less RAM.
On HuggingFace, you can check the compatibility of the selected quantization with your computer by adding your hardware parameters (e.g., Mac M4 Max) in the corresponding section on the model page.
4 Installing and Using Ollama
Ollama is one of the easiest applications for managing local models.
1. Go to the Ollama website and download the desktop version (for macOS, Windows, or Linux).
2. After installation, create an account on their website.
3. To download a model from HuggingFace, copy the launch command (e.g., ollama run...) from the model page.
4. Open the terminal and paste this command. Wait for the download to finish.
5. After that, you can chat with the model directly in the Ollama app or via terminal (answers are generated quickly — about 6-8 seconds for simple questions).
5 Cloud Integration and Claude Code
If your computer cannot handle heavy models, Ollama has an excellent cloud alternative. You can connect the cloud version of the heavy Gemma 4 (31B parameters) and work with it locally without loading your system.
- To use a local or cloud model as an AI developer assistant, you can integrate them into Claude Code.
- By running Claude Code, select the required model (e.g., cloud Gemma 4) from the Ollama list.
- Make a request, and the model will write code or create a file (e.g., an HTML presentation or web page) right in your project folder.