
At first glance the file structure used by Ollama can be a bit overwhelming. It's pretty common for folks to want Ollama to just use an unmodified model weights file without anything extra like so many other tools out there. But after you spend a little time with it, you can't help but wonder why every tool doesn't do it in this much more elegant and obvious way. There are so many benefits to be had by this simple tweak and I'll show you more in this episode of the Ollama Course. 

Welcome back to the Ollama Course. If this is actually your first introduction to the course, I am going through all the aspects of using and working with Ollama. Ollama is the easiest way to work with AI models on whatever machine you use every day, assuming it has a relatively recent...and decent GPU. You can get it up and running at ollama.com. Click the big button in the middle of the page and you can get it going in just a few minutes. 

Then come back to the videos in the Ollama Course playlist to learn everything you need to know to start working with it. I'm Matt Williams and I was a founding member of the Ollama team. I am no longer working there and have decided to focus full time on my YouTube channel. So if you like content like this, be sure to like and subscribe to stay up to date on everything I put out. 

OK, so let's dive into what is special about the file structure used by Ollama. First, where are the files? Well it depends on the OS and how you choose to run the service. The documentation for Ollama can be found at this page. Docs update all the time, but as of the recording of this video, there is a faq, or F A Q, or frequently asked questions doc in that directory. Scroll down and you will find a section that right now is titled "Where are models stored?". Ignore the models part for now, and you can see that on MacOS, everything is under .ollama in your user directory. On Linux, assuming you installed with the official install script, it's at /usr/share/ollama/.ollama. And on Windows it's just like the Mac, .ollama in your user directory. So from now on I will just refer to the .ollama directory and you remember where that is for your OS. 

So in .ollama you will find the history file, which shows all the questions you have asked and the private and public key used to pull and push models. These keys may look like SSH keys, but they aren't, so don't use sshkeygen to replace them or you will run into issues. Then on Mac there is a logs directory which has logs for the service. Other platforms put them elsewhere so you can find the details in the troubleshooting doc. then there is a models directory. This is where all the files for the models live. So in that directory there is a blobs folder and a manifests folder. If you look at blobs right now, you may be very confused. We will come back to that, but never delete anything from this directory or you will break stuff. 

The manifests folder is the magic one to start with. I'm going to assume you have at least one model installed on your system. It's probably llama 3.1 or 3.2. And it's probably the weights that are quantized to 4 bit. I talked a bit about quantization earlier in this course right here. You installed that model by running ollama pull llama3.2 for the 3.2 model. This model comes from the official library on ollama.com. The actual location of that is registry.ollama.ai. Ollama.ai was the first domain we had before buying ollama.com. And it's called registry because it's based on the registry for docker. In fact it was created by the same guy who I think created and definitely ran Docker Hub for a long time. In that directory you will find library. And in there you will find a folder for llama3.2 and every other model you have pulled from the library. 

Now if you only pulled llama3.2 without specifying any tags, you will have a file called latest in here. This is a terrible name because it has nothing to do with being the latest, its just the word chosen for the default size. I have also pulled llama3.2:1b and llama3.2:3b and so I have those files. Now I'll list the contents of latest using the cat command. This is a bit hard to read, so I'll pipe that to jq which is a way to pretty print json. 

Up at the top of this file is a block that describes the file. it's not all that important to understand, but no, this is not docker. 

The important stuff is in layers. For each layer there is a mediatype, a digest, and a size. The media type is just a standard way to describe each layer. The first one says model and this is the source gguf file that is the model weights. Digest lists the sha256 hash of the file. SHA, S H A, stands for Secure Hash Algorithm and was designed by the US NSA to be a simple way to understand if a file has been modified in any way since the hash was generated. It's usually used with digital signatures and password hashing, but here it is used to help ollama see if the files it has already downloaded includes the file needed for this model. If the file only went by it's original filename, there would be no way to tell. 

This particular model has layers for the model weights, the template, two license files, and the parameters for the model. So let's consider the template. For this model the template has this sha256 hash. Now if we go back to the .ollama/models/blobs directory, we can find that exact file. I'll use cat on the file to show the contents and we see this. And that is the template for the model. We can confirm this by going to ollama.com, finding the llama3.2 model and then click on the template. And that looks like the same contents. 

Layers are a useful concept for models because in order to be of any use, the model weights need to be combined with a template and in many cases a system prompt to be useful, and most other tools guess what the prompt should be or require you to figure it out. Also, to abide by the rules of the licenses many models use, the model should be distributed with the license. Many tools skip this part, potentially putting themselves into legal jeopardy. 

Now lets say I create a new modelfile and set the model to use as a base to llama3.2. Then I'll set num_ctx to 10000. No reason for that specific number, its just a number that is easy to type. Then I will create the model. My username on ollama is m, so I will create m/llama32-10000. And then push it up to ollama. If you want to see it, you can find it at this url. 

But let's look at the manifest for that model... and the manifest for the original llama3.2 model. See the sha256 for the model weights for both are the same. So if you already have the llama3.2 model and then you pull m/llama32-10000 you will be done in seconds because the weights file doesn't have to download. You already have the file. 

Now if I want to delete llama3.2, then I need to run `ollama rm llama3.2`. But look at the blob files and we will see that the weights for that model are still on my system, because llama32-10000 is using it. It's not until all models that used any blob are gone that that blob will finally get deleted. 

But if you had gone into the blobs directory and deleted that file, then you potentially mess up 1, 2, or maybe many other models that use the same weights file. the next time you restart ollama, all of those orphaned models will be cleaned up, but if you need to use any of them, you will have to redownload the larger file again. Depending on your actions, you may get a cryptic error before that happens.

If you did want to use one of the model weights files in another tool, its totally possible but not really a use case ollama was designed for. You could figure out the blob file to use, then copy the file to a new directory, calling it whatever you want. It is still the same gguf format. There are a number of other tools out there that will do this for you. One of my favorites is called gollama and it's a simple cli tool to sync models to any location you want with a friendlier name. I have a video about that tool that I made a few months back. Check it out to learn more, but I am sure the author has added a bunch of new features since then. 

And that is pretty much everything you need to know about how the files are stored that ollama uses. As you can see, its not all that hard to understand, and once you do, it becomes obvious why its like that and how every project just gets this wrong. It really is one of the key reasons Ollama really is magical. 

I hope you found this episode of the Ollama course interesting. If you have any specific questions, feel free to leave a comment below. I also have a newsletter that I send out every now and then. You can sign up at technovangelist.com/newsletter.

Thanks so much for watching, goodbye. 
