Skip to content

Setting up InstructLab

Now that we have a working VSCode instance working with the granite model, lets talk about more things you can do with an InstructLab system. This is where the InstructLab project comes into play.

Now that we've used the foundation model from upstream, lets add some knowledge to our model that we want it to have.

Note

As of this writing (and we haven't updated it so it's still true) this Does NOT work with Granite-Code, but instead the core Granite model. When this note is removed and we have a qna.yaml like way to train Granite-Code we will update this tutorial.

Requirements

  • 🍎 Apple M1/M2/M3 Mac or 🐧 Linux system (tested on Fedora).
  • C++ compiler
  • Python 3.10 or Python 3.11
  • Approximately 60GB disk space (entire process)

Note

When installing the ilab CLI on macOS, you may have to run the xcode-select --install command, installing the required packages previously listed.

Warning

If you are running on Windows, you should use WSL2 for this, but it is unsupported but we will do our best to get you success if you run into any problems.

Installing ilab

Create a new directory called instructlab to store the files the ilab CLI needs when running and cd into the directory by running the following command:

mkdir instructlab
cd instructlab

See the GPU acceleration documentation for how to to enable hardware acceleration for interaction and training on AMD ROCm, Apple Metal Performance Shaders (MPS), and Nvidia CUDA.

Install using PyTorch without CUDA bindings and no GPU acceleration

python3 -m venv --upgrade-deps venv
source venv/bin/activate
pip cache remove llama_cpp_python
pip install instructlab[cpu] \
   --extra-index-url=https://download.pytorch.org/whl/cpu \
   -C cmake.args="-DLLAMA_NATIVE=off"

Note

Additional Build Argument for Intel Macs

If you have an Mac with an Intel CPU, you must add a prefix of CMAKE_ARGS="-DLLAMA_METAL=off" to the pip install command to ensure that the build is done without Apple M-series GPU support.

(venv) $ CMAKE_ARGS="-DLLAMA_METAL=off" pip install ...

Install with Apple Metal on M1/M2/M3 Macs

Note

Make sure your system Python build is Mach-O 64-bit executable arm64 by using file -b $(command -v python), or if your system is setup with pyenv by using the file -b $(pyenv which python) command.

python3 -m venv --upgrade-deps venv
source venv/bin/activate
pip cache remove llama_cpp_python
pip install instructlab[mps]

Install with Nvidia CUDA

python3 -m venv --upgrade-deps venv
source venv/bin/activate
pip cache remove llama_cpp_python
pip install instructlab[cuda] \
   -C cmake.args="-DLLAMA_CUDA=on" \
   -C cmake.args="-DLLAMA_NATIVE=off"

Verify ilab is installed

From your venv environment, verify ilab is installed correctly, by running the ilab command.

ilab

Example output of the ilab command

(venv) $ ilab
Usage: ilab [OPTIONS] COMMAND [ARGS]...

CLI for interacting with InstructLab.

If this is your first time running InstructLab, it's best to start with `ilab config init` to create the environment.

Options:
--config PATH  Path to a configuration file.  [default: config.yaml]
--version      Show the version and exit.
--help         Show this message and exit.

Command:
   config      Command group for Interacting with the Config of InstructLab
   data        Command group for Interacting with the Data of generated by...
   model       Command group for Interacting with the Models in InstructLab
   sysinfo     Print system information
   taxonomy    Command group for Interacting with the Taxonomy in InstructLab

Aliases:
   chat: model chat
   convert: model convert
   diff: taxonomy diff
   download: model download
   generate: data generate
   init: config init
   serve: model serve
   test: model test
   train: model train

🏗️ Initialize ilab

  1. Initialize ilab by running the following command:
ilab config init

Example output

Welcome to InstructLab CLI. This guide will help you set up your environment.
Please provide the following values to initiate the environment [press Enter for defaults]:
Path to taxonomy repo [taxonomy]: <ENTER>

  1. When prompted by the interface, press Enter to add a new default config.yaml file.

  2. When prompted, clone the https://github.com/instructlab/taxonomy.git repository into the current directory by typing y.

Optional: If you want to point to an existing local clone of the taxonomy repository, you can pass the path interactively or alternatively with the --taxonomy-path flag.

Example output after initializing ilab

(venv) $ ilab config init
Welcome to InstructLab CLI. This guide will help you set up your environment.
Please provide the following values to initiate the environment [press Enter for defaults]:
Path to taxonomy repo [taxonomy]: <ENTER>
`taxonomy` seems to not exists or is empty. Should I clone https://github.com/instructlab/taxonomy.git for you? [y/N]: y
Cloning https://github.com/instructlab/taxonomy.git...
Generating `config.yaml` in the current directory...
Initialization completed successfully, you're ready to start using `ilab`. Enjoy!

ilab will use the default configuration file unless otherwise specified. You can override this behavior with the --config parameter for any ilab command.

📥 Download the model

  • Run the ilab model download command. 
    ilab download --repository instructlab/granite-7b-lab-GGUF --filename=granite-7b-lab-Q4_K_M.gguf

Serve the granite model

  • Let serve the granite model to verify it's working.
ilab serve --model-path models/granite-7b-lab-Q4_K_M.gguf
  • In another terminal run the ilab chat
cd instructlab
source venv/bin/activate
ilab model chat

Chat with the model!

Ask it some questions, things like:

  • Who is batman?
  • What is the capital of Sweden?

Now that you have a working instructlab setup, lets get to actually tuning the model. Lets move over to Lab 4!