Researchers are using Large Language Models (LLMs) for a wide variety of tasks, including generating text, summarizing papers, and brainstorming ideas. LLMs also assist with programming: debugging code, writing scripts or snippets, and accessing specialized models for advanced applications like computer vision and multi-modal tasks.
Did you know that your laptop may be powerful enough to run an LLM, without depending on cloud services or third-party providers?
In this short introduction, you’ll learn how to take advantage of open-source models and tools to run and even customize local LLMs.
Benefits to Local LLMs
- Data Privacy: Keep sensitive research data secure on your device.
- Transparency: Open source with a permissive license, allowing visibility and customization. Full control over model updates, versions, embeddings and dependencies for reproducible research.
- Collaboration: Share models and configurations easily within teams for collaborative research.
- Cost Efficiency: Save on expensive cloud-based services by running models on your hardware.
What is Ollama?
Ollama is an an open-source tool that allows users to run cutting-edge generative AI locally from the command line. LLMs operate offline, entirely on your device, keeping data private and avoiding external, commercial platforms. Visit Ollama’s site to learn more.
Not familiar with the command line? Open WebUI lets you use available models, or to customize or create your own. By default, it uses the models available in Ollama, and the entire process remains fully offline—no internet connection or public server needed.
Limitations and Risks
- Hardware Requirements: Models are resource-intensive and work best with GPU acceleration.
- Institutional Support: Neither Ollama nor Open WebUI are tools that are centrally supported by UZH IT. Do check with your local IT Administrator in case you need support.
- Security: Always download from trusted sources and make sure you use the most recent, patched version to reduce risks.
- Note from the Department of Data Protection at the UZH: In general, if you use an AI or LLM application installed on your personal computer you need to ensure your data (in particular person or otherwise sensitive data) is not uploaded or otherwise communicated to any other machine and that your personal computer itself is secured in a way appropriate for the type of data you are processing. In particular, you need to be careful not to open and allow others access to the Ollama REST API on your personal computer which could compromise the security of your data.
Quick Start
- Installation: Visit the Ollama repository to learn more.
- Run Your First Model: Try running a model with "ollama run deepseek-r1:8b" or “ollama run llama3.2” in the terminal.
- Explore Models: Browse and download available models to expand your options.
- Advanced Topics: Integrate Ollama into your code using PyPI, Docker, or its REST API. To monitor hardware or GPU usage, use tools like nvtop or Activity Monitor (macOS).
