The last video I put out was 2 days ago, and that was about version 0.1.18. And now we have 0.1.19. There are a bunch of fixes in this one, rather than any new features, but most of them are very important, especially if you are affected by them. The maintainers of Ollama have worked hard at supporting new platforms, like AMD cards as well as a native windows app and improvements to Mac, and gradually as more users are starting to work with it we saw more memory problems. There are a number of variations on "memory" issues, many of which are addressed here. 

So there are a few issues cleared up around context size, as well as out of memory errors. There were also some problems with older nvidia gpus. Some improvements were made to lower the requirements for mixtral. Intel Macs got some performance improvements. One of the big ones for a lot of users of some of the community integrations is support for browser extension urls in OLLAMA_ORIGINS. And ollama will offload to GPUs more where possible. There is also a new cool model to take a look at.

One of biggest and weirdest issues, and maybe most ornery to figure out was around context size. There were new models supported that offered up to 32K or even bigger context sizes. Calculating the memory required for the models themselves is relatively easy. 7b, at least 8gb-ish. 13b, at least 16gb ish. But figuring out the memory for context has been harder. And so if you didn't have the memory needed, and Ollama didn't expect it, you could hit a rather severe crash, either just ollama, or sometimes,  the whole system comes down. I saw my MacBook reboot a few times unexpectedly.

Most folks wanted to try the models but didn't really take advantage of the bigger context, which were enabled in the modelfiles using a larger value of the num_ctx parameter. So a few weeks ago, all the models were updated to use the default context size of 2k or 4k. If you wanted to use something bigger, you could force a larger context using the num_ctx option. Only problem was that that option didn't always work right. Well, now it does. Let's take a look at how to set the context size. First in the cli or repl. so ollama run mistral. This is one that accepts a 32k context size. `ollama run mistral`. According to the ollama.ai library, this model is version 0.2. So going to the source repo on Hugging Face I saw that the initial version of mistral had an 8k context, but v2 had a 32k context. So let's go ahead and use that now. Type /? and we see  the ollama help screen. /? and set and we see all the things we can set here in the repl. I want to set a parameter. so /set parameter num_ctx  32768. and press enter. I don't have an example large context to use but this now actually works. Though you definitely are going to need some memory to handle this.

And now lets use the api to set the size. I am in vscode and I find the easiest way to test rest endpoints is to use the rest client from huachao mao. its at 4 million downloads so I am not alone. 

so POST to the api generate endpoint. the next line is for headers, so skip a line and add your object. I'll set model to mistral, prompt to why is the sky blue. stream to false, just to make this easier. and then add an options object with num_ctx set to 32768. Send that request and we see that it processed. 

The last example I'll do is if we want to create a new model that always has that set. So in a new directory, I'll create a Modelfile. FROM mistral. PARAMETER num_ctx 32768. And since that is all we are changing, that’s all we need. Now back at the command line i can run `ollama create bigmistral`. Normally i need to say `-f modelfile` but if we are in the same directory and its called Modelfile, we can skip that. and now `ollama run bigmistral` and we have our mistral with a big context. 

It's hard to demonstrate the rest of the improvements but I definitely want to create a video that shows the benefit of a larger context as well as it's impact on memory usage. If that is something you are interested in, or you would like to see anything else, let me know in the comments below. I'd also love to hear about how some of these fixes affected you.

thanks so much for watching, goodbye.