Skip to content

Getting Started with InstructLab

Now that we have a working VSCode instance working with the granite model, lets talk about more things you can do with an Open Source AI system. This is where the InstructLab project comes into play.

Now that we've used the foundation model from upstream, lets add some knowledge to our model that we want it to have.

Warning

If you are running on Windows, you should use WSL2 for this, but it is unsupported but we will do our best to get you success if you run into any problems.

tl;dr

Note

These steps will pull down a premade qna.yaml so you can do a local build. Skip the wget, mv, and ilab taxonomy diff if you don't want to do this.

python3.11 -m venv venv-instructlab-0.18-3.11
source venv-instructlab-0.18-3.11/bin/activate
pip install 'instructlab[mps]'
which ilab
ilab config init
cd ~/Library/Application\ Support/instructlab/
mkdir -p taxonomy/knowledge/astronomy/constellations/Phoenix/
wget https://raw.githubusercontent.com/instructlab/taxonomy/26b3fe21ccbb95adc06fe8ce76c7c18559e8dd05/knowledge/science/astronomy/constellations/phoenix/qna.yaml
mv qna.yaml taxonomy/knowledge/astronomy/constellations/Phoenix/
ilab taxonomy diff
ilab data generate
ilab model train
ilab model convert --model-dir checkpoints/instructlab-granite-7b-lab-mlx-q
ilab model serve --model-path instructlab-granite-7b-lab-trained/instructlab-granite-7b-lab-Q4_K_M.gguf

Steps

Install ilab

1) Create a new directory called instructlab to store the files the ilab CLI needs when running and cd into the directory by running the following command:

mkdir instructlab
cd instructlab

Note

The following steps in this document use Python venv for virtual environments. However, if you use another tool such as pyenv or Conda Miniforge for managing Python environments on your machine continue to use that tool instead. Otherwise, you may have issues with packages that are installed but not found in venv.

2) There are a few ways you can locally install the ilab CLI. Select your preferred installation method from the following instructions. You can then install ilab and activate your venv environment.

Note

⏳ pip install may take some time, depending on your internet connection. In case installation fails with error unsupported instruction `vpdpbusd', append -C cmake.args="-DLLAMA_NATIVE=off" to pip install command.

3) Install with Apple Metal on M1/M2/M3 Macs

Note

Make sure your system Python build is Mach-O 64-bit executable arm64 by using file -b $(command -v python), or if your system is setup with pyenv by using the file -b $(pyenv which python) command.

python3 -m venv --upgrade-deps venv
source venv/bin/activate
pip cache remove llama_cpp_python
pip install 'instructlab[mps]'

4) From your venv environment, verify ilab is installed correctly, by running the ilab command.

ilab

Example output of the ilab command

(venv) $ ilab
Usage: ilab [OPTIONS] COMMAND [ARGS]...

CLI for interacting with InstructLab.

If this is your first time running InstructLab, it's best to start with `ilab config init` to create the environment.

Options:
--config PATH  Path to a configuration file.  [default:
               /home/user/.config/instructlab/config.yaml]
-v, --verbose  Enable debug logging (repeat for even more verbosity)
--version      Show the version and exit.
--help         Show this message and exit.

Commands:
config    Command Group for Interacting with the Config of InstructLab.
data      Command Group for Interacting with the Data generated by...
model     Command Group for Interacting with the Models in InstructLab.
system    Command group for all system-related command calls
taxonomy  Command Group for Interacting with the Taxonomy of InstructLab.

Aliases:
chat      model chat
convert   model convert
diff      taxonomy diff
download  model download
evaluate  model evaluate
generate  data generate
init      config init
list      model model_list
serve     model serve
sysinfo   system info
test      model test
train     model train

Important

Every ilab command needs to be run from within your Python virtual environment. You can enter the Python environment by running the source venv/bin/activate command.

5) Optional: You can enable tab completion for the ilab command.

Bash (version 4.4 or newer)

Enable tab completion in bash with the following command:

eval "$(_ILAB_COMPLETE=bash_source ilab)"

To have this enabled automatically every time you open a new shell, you can save the completion script and source it from ~/.bashrc:

_ILAB_COMPLETE=bash_source ilab > ~/.ilab-complete.bash
echo ". ~/.ilab-complete.bash" >> ~/.bashrc

Zsh

Enable tab completion in zsh with the following command:

eval "$(_ILAB_COMPLETE=zsh_source ilab)"

To have this enabled automatically every time you open a new shell, you can save the completion script and source it from ~/.zshrc:

_ILAB_COMPLETE=zsh_source ilab > ~/.ilab-complete.zsh
echo ". ~/.ilab-complete.zsh" >> ~/.zshrc

Fish

Enable tab completion in fish with the following command:

_ILAB_COMPLETE=fish_source ilab | source

To have this enabled automatically every time you open a new shell, you can save the completion script and source it from ~/.bashrc:

_ILAB_COMPLETE=fish_source ilab > ~/.config/fish/completions/ilab.fish

🏗️ Initialize ilab

1) Initialize ilab by running the following command:

ilab config init

Example output

Welcome to InstructLab CLI. This guide will help you set up your environment.
Please provide the following values to initiate the environment [press Enter for defaults]:
Path to taxonomy repo [taxonomy]: <ENTER>

2) When prompted by the interface, press Enter to add a new default config.yaml file.

3) When prompted, clone the https://github.com/instructlab/taxonomy.git repository into the current directory by typing y.

Optional: If you want to point to an existing local clone of the taxonomy repository, you can pass the path interactively or alternatively with the --taxonomy-path flag.

Example output after initializing ilab

(venv) $ ilab config init
Welcome to InstructLab CLI. This guide will help you set up your environment.
Please provide the following values to initiate the environment [press Enter for defaults]:
Path to taxonomy repo [taxonomy]: <ENTER>
`taxonomy` seems to not exists or is empty. Should I clone https://github.com/instructlab/taxonomy.git for you? [y/N]: y
Cloning https://github.com/instructlab/taxonomy.git...

ilab will use the default configuration file unless otherwise specified. You can override this behavior with the --config parameter for any ilab command.

4) When prompted, provide the path to your default model. Otherwise, the default of a quantized Merlinite model will be used - you can download this model with ilab model download (see below).

(venv) $ ilab config init
Welcome to InstructLab CLI. This guide will help you set up your environment.
Please provide the following values to initiate the environment [press Enter for defaults]:
Path to taxonomy repo [taxonomy]: <ENTER>
`taxonomy` seems to not exists or is empty. Should I clone https://github.com/instructlab/taxonomy.git for you? [y/N]: y
Cloning https://github.com/instructlab/taxonomy.git...
Path to your model [/Users/USERNAME/Library/Caches/instructlab/models/merlinite-7b-lab-Q4_K_M.gguf]: <ENTER>

5) When prompted, please choose a train profile. Train profiles are GPU specific profiles that enable accelerated training behavior. YOU ARE ON MacOS, please choose No Profile (CPU-Only) by hitting Enter. There are various flags you can utilize with individual ilab commands that will allow you to utilize your GPU if applicable.

Welcome to InstructLab CLI. This guide will help you to setup your environment.
Please provide the following values to initiate the environment [press Enter for defaults]:
Path to taxonomy repo [~/Library/Application\ Support/instructlab/taxonomy]:
Path to your model [/Users/USERNAME/Library/Caches//instructlab/models/merlinite-7b-lab-Q4_K_M.gguf]:
Generating `~/Library/Application\ Support/instructlab/config.yaml`...
Please choose a train profile to use:
[0] No profile (CPU-only)
[1] A100_H100_x2.yaml
[2] A100_H100_x4.yaml
[3] A100_H100_x8.yaml
[4] L40_x4.yaml
[5] L40_x8.yaml
[6] L4_x8.yaml
Enter the number of your choice [hit enter for the default CPU-only profile] [0]:
Using default CPU-only train profile.
Initialization completed successfully, you're ready to start using `ilab`. Enjoy!

The GPU profiles are listed by GPU type and number. If you happen to have a GPU configuration with a similar amount of VRAM as any of the above profiles, feel free to try them out!

ilab directory layout after initializing your system

After running ilab config init your directories will look like the following on a Linux system:

├─ ~/Library/Application\ Support/instructlab/models/ (1)
├─ ~/Library/Application\ Support/instructlab/datasets (2)
├─ ~/Library/Application\ Support/instructlab/taxonomy (3)
├─ ~/Library/Application\ Support/instructlab/checkpoints (4)

1) /Users/USERNAME/Library/Caches/instructlab/models/: Contains all downloaded large language models, including the saved output of ones you generate with ilab. 2) ~/Library/Application\ Support/instructlab/datasets/: Contains data output from the SDG phase, built on modifications to the taxonomy repository. 3) ~/Library/Application\ Support/instructlab/taxonomy/: Contains the skill and knowledge data. 4) ~/Users/USERNAME/Library/Caches/instructlab/checkpoints/: Contains the output of the training process

📥 Download the model

  • Run the ilab model download command.
ilab model download

ilab model download downloads a compact pre-trained version of the model (~4.4G) from HuggingFace:

(venv) $ ilab model download
Downloading model from Hugging Face: instructlab/merlinite-7b-lab-GGUF@main to /Users/USERNAME/Library/Caches/instructlab/models...
...
INFO 2024-08-01 15:05:48,464 huggingface_hub.file_download:1893: Download complete. Moving file to /Users/USERNAME/Library/Caches/instructlab/models/merlinite-7b-lab-Q4_K_M.gguf

Note

⏳ This command can take few minutes or immediately depending on your internet connection or model is cached. If you have issues connecting to Hugging Face, refer to the Hugging Face discussion forum for more details.

Downloading an entire Hugging Face repository (Safetensors Model)

  • Specify repository, and a Hugging Face token if necessary. For example:
HF_TOKEN=<YOUR HUGGINGFACE TOKEN GOES HERE> ilab model download --repository=instructlab/granite-7b-lab

These types of models are useful for GPU-enabled systems or anyone looking to serve a model using vLLM. InstructLab provides Safetensor versions of our Granite models on HuggingFace.

Listing downloaded models

All downloaded models can be seen with ilab model list.

ilab model list

Example output of ilab model list after ilab model download

(venv) $ ilab model list
+------------------------------+---------------------+--------+
| Model Name                   | Last Modified       | Size   |
+------------------------------+---------------------+--------+
| merlinite-7b-lab-Q4_K_M.gguf | 2024-08-01 15:05:48 | 4.1 GB |
+------------------------------+---------------------+--------+

🍴 Serving the model

  • Serve the model by running the following command:
ilab model serve

erve a non-default model (e.g. Mixtral-8x7B-Instruct-v0.1):

ilab model serve --model-path models/mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf

nce the model is served and ready, you'll see the following output:

(venv) $ ilab model serve
INFO 2024-03-02 02:21:11,352 lab.py:201 Using model 'models/ggml-merlinite-7b-lab-Q4_K_M.gguf' with -1 gpu-layers and 4096 max context size.
Starting server process
After application startup complete see http://127.0.0.1:8000/docs for API.
Press CTRL+C to shut down the server.

Note

If multiple ilab clients try to connect to the same InstructLab server at the same time, the 1st will connect to the server while the others will start their own temporary server. This will require additional resources on the host machine.

  • Serve a non-default Safetensors model (e.g. granite-7b-lab). NOTE: this requires a GPU.

Ensure vllm is installed:

pip show vllm

If it is not, please run:

pip install vllm@git+https://github.com/opendatahub-io/vllm@2024.08.01

ilab model serve --model-path ~/.cache/instructlab/models/instructlab/granite-7b-lab

📣 Chat with the model (Optional)

Because you're serving the model in one terminal window, you will have to create a new window and re-activate your Python virtual environment to run ilab model chat command:

source venv/bin/activate
ilab model chat

Chat with a non-default model (e.g. Mixtral-8x7B-Instruct-v0.1):

source venv/bin/activate
ilab model chat --model models/mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf

Please note that usage of --model necessitates that the existing server has that model. If not, you must exit the server. --model in ilab model chat has the ability to start a server on your behalf with the specified model if one is not already running on the port.

Before you start adding new skills and knowledge to your model, you can check its baseline performance by asking it a question such as what is the capital of Canada?.

Note

The model needs to be trained with the generated synthetic data to use the new skills or knowledge

(venv) $ ilab model chat
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────── system ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Welcome to InstructLab Chat w/ GGML-MERLINITE-7B-lab-Q4_K_M (type /h for help)                                                                                                                                                                    │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
>>> what is the capital of Canada                                                                                                                                                                                                 [S][default]
╭────────────────────────────────────────────────────────────────────────────────────────────────────── ggml-merlinite-7b-lab-Q4_K_M ───────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ The capital city of Canada is Ottawa. It is located in the province of Ontario, on the southern banks of the Ottawa River in the eastern portion of southern Ontario. The city serves as the political center for Canada, as it is home to │
│ Parliament Hill, which houses the House of Commons, Senate, Supreme Court, and Cabinet of Canada. Ottawa has a rich history and cultural significance, making it an essential part of Canada's identity.                                   │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 12.008 seconds ─╯