technovangelist.videoprojects/scripts/starcoder.txt

I woke up with a little tickle in my throat, so this one sounds maybe a little gravelly at times and maybe my voice will break like I am 12 like it did a few times in preparation for this. So... There is a new version of Ollama out and its headliner is support for...no...but it was a mind blowing ride i 1987... no...but man when that came out when I was 14 in 84, wow and it seems to have held up well, ... there we go,  starcoder 2. This is a new software development model by the BigCode Project. I have seen a lot of requests out there on the Ollama Discord asking for this model. It required updates from llamacpp in order for it to work and those changes were merged a few days ago. 

Why are folks excited? I'm not exactly sure....I think the main thing going for it is that it is new and shiny. Everyone loves new and shiny. You look at The Bloke's discord and the requests for new models seems to be flooded by folks who just want to be first on the request. They tend to not actually care about what is new and interesting about the model. They just want to be first to point to the new and shiny. Maybe it's the same here. 

So Starcoder v2 is out. Is it interesting? I guess so. I was probably the 3rd or 4th youtuber you have seen covering this model. And if you look at all the others reviewing the model, it turns out they don't actually say anything beyond reading snippets from the blog post and paper. And so you may come to the end of those videos thinking that the model seems to be incredible. 

But I think the more interesting story is the team behind it and seeing them deliver a model that moves incrementally forward and may make room for some amazing things in the future. BigCode is led by HuggingFace and ServiceNow. Sure, HuggingFace makes a lot of sense there, but ServiceNow? That seems weird. I remember them being just another PagerDuty clone but I guess they have been successful and done some acquisitions. But anyway, let's move on...

When starcoder 1 came out it was a pretty big deal. It may have been the first of the models to really focus in on coding. Unfortunately it was in a format that wasn't supported by llamacpp and thus not supported by Ollama. But when it came out,  llamacpp was still a young project if it even existed, and the Ollama team was focused on making an amazing tool for ssh key pairs at keypair.com. It was a few months into the Ollama project that support for starcoder came along and by that time there were plenty of other coding models on the platform. So what was amazing before, was kinda ho hum by the time it got onto Ollama. 

Starcoder2 looks like a nice update to the original, and days after its release, it's on Ollama, though as of the recording of this video its still a prerelease of the software. Hopefully by the time you see this, it will be a full release. 

So what's different?  Well a lot has changed in the world of LLM's in the year since v1. The training set has grown a lot. It's trained on a dataset called 'The Stack V2' which is 4 trillion tokens of coding projects. That's up from the 1 trillion tokens in v1. And whereas v1 of The Stack pulled from 80 or so languages, v2 uses over 600 programming languages. I got a few questions on a recent video asking if a few super niche esoteric languages were supported by some other model. Of course they weren't, but there is a good chance that that minecraft specific language you love is supported by this model.  If you wanted to download the full dataset, prepare to delete everything else on your system, because it comes in at 67.5 terrabytes.  The context window has also increased from 8k tokens to 16k tokens which is great. There is also the introduction of sliding window attention. This doesn't really decrease the 'memory' size but allows for the massive reduction in number of dot products it needs to generate for each token. so there is a minimal reduction in capability but with a massive reduction in compute needed to calculate the next token, ideally making it feel faster. I believe this sliding window is something that was introduced in the Mistral model. 

Now there are a bunch of models that support these larger context sizes, but when you load up the default model in Ollama, it will allow for a context size of 2k. Originally the models were set to the full context size, but folks were trying out 7b models on laptops with 8 gigabytes of ram and irate when they got out of memory errors. The context size blows the requirements up a lot. If you want the model to have the full context size, create a new Modelfile with a FROM instruction pointing at the model name  and a PARAMETER instruction of num_ctx set to 16,384. Then run `ollama create fullstarcoder` in that directory and you are set. 

StarCoder 2 is focused on the "fill in the middle" use case. The idea there is that you provide code before and after what you want the model to generate. This is quite a bit different from providing a question that you want the model to answer, which is the case for instruct or chat models. So if you open starcoder in most gui's and ask a question, you will most likely get garbage. 

So to use StarCoder 2 for a fill in the middle, you need to use a few special tokens to indicate the beginning and end, and where in the middle you want new stuff to go. We can go to the Hugging Face repo for one of the StarCoder2 models and take a look in the tokenizer config file to see the actual values. They are fim prefix, fim suffix, and fim middle. So you would have your code block and at the beginning add fim underscore prefix with angle brackets at the beginning of all that. And then fim underscore suffix with angle brackets at the end. Then whatever you want to be filled in would be indicated by the fim_middle with angle brackets. Here I have taken some of my bad code from my mentors example, and deleted some stuff from the middle and replaced it with the fim_middle token. 

Normally you aren't going to put these tokens in yourself, but rely on some other tool to fill them in for you. Twinny is a great option for this in vscode and it supports fill in the middle and it looks like it already has support for the starcoder format. 

So is starcoder v2 any good? Well, I'm not entirely sure. And no matter what I say, you should ignore me and judge it for yourself. Now I am sure someone is going to get to this point and scream at their monitor that the benchmarks prove that this is better than x or y or z. Sure, if you believe benchmarks, then this seems good. We can look at a cool chart and it seems convincing, but if it does, you should check out this book called how to lie with statistics by darrell huff. I have had this for 15 years but it was published in the 1950's and is just as great today. Often with the benchmarks you find models that perform really well according to the benchmark but then are terrible in actual practice. And every model shows the other models they perform better than and mysteriously leave out all the others. And there are all sorts of other reasons to not take the benchmarks as the Gospel. That said, they are a great indicator to show that perhaps things are getting better, but they should never be used to say that one model is categorically better than any other all the time. And then there is the whole need for benchmarks to be open source but let's not go there. 

If you made it this far, you are waiting for one of two things. Either you are waiting for my 20 seconds of silence at the end of all my videos, or you are wondering when I am going to cover the rest of the new things in version 0.1.28 of Ollama. Apparently there is better support for llava 1.6. Those are the vision models that let you process an image into a text description. In some initial testing it seems to do a better job of working with multiple images which could be really interesting. 

There is one user who demonstrated if they changed models super quickly over and over again 20 times in a row, Ollama would hang. So for that one person, its fixed. 

Apparently there is another person who tried to install ollama on a Windows box with less than 4gb of disk. That now works, though it doesn't take many models plus the OS to rip through that number. 

A more common issue was that on Apple Silicon, folks would run out of memory, even if the model should fit. Ollama is doing a better job there. 

In my last video about docker and Ollama someone asked for the commands for Podman. I used to cohost a podcast with a guy named Bret Fisher called Devops and Docker Talk. We had the core devs for podman on the show and they tried to convince us that Podman was better than Docker in some way, but I'm still using Docker. Anyway, Podman was having problems detecting Nvidia GPUs and now that should be fixed. Another side note, if you miss my appearance on Devops and Docker Talk, I will be on another episode very very soon. 

And now that embeddings in Ollama is actually useful, one specific scenario with an empty prompt won't hang the embedding endpoint. And that's what is new in Ollama 0.1.28 as of the recording of this video, and whats special about StarCoder 2. Whenever there is a new pre release, things get added and taken away as the release gets close, so don't be surprised if the final feature list is a touch different. 

What do you think? Do you like StarCoder? Do you prefer a different Coding Model?

Let me know in the comments below and if you have any ideas for future videos let me know about those too. Thanks so much for being here. goodbye.