🪴 Quartz 4.0

Search

SearchSearch

llm_augmentoolkit_generating-data-with-augmentoolkit

Aug 05, 20246 min read

Generating Data with autogentoolkit

🧭

Created a directory at:

d=~/Sync/Studies/LLM/data-generation
repo='https://github.com/e-p-armstrong/augmentoolkit'
git clone \
    $repo $d/
 
cd $d/augmentoolkit

Created a venv at:

vdir=~/.local/share/virtualenvs/augmentoolkit/bin/activate.fish
python3 -m venv $vdir
source $vdir/bin/activate
pip install -r requirements.txt

Investigating the config I was presented with the following warning:

  • config

      BASE_URL: "https://api.together.xyz"

  • Comment

    add the base url for a provider, or local server, here. Some possible values:

I decided to go with GPT3 first, I’ll track the cost closely and then try Mistral.

According to openai.com/pricing, the cheapest model is gpt-3.5-turbo-0125, so I set that.

Ultimately I settled with this config:

config.yaml

 
PATH:
  INPUT: "./raw_txt_input"
  OUTPUT: "./output"
  DEFAULT_PROMPTS: "./prompts" # the baseline prompt folder that Augmentoolkit falls back to if it can't find a step in the PROMPTS path
  PROMPTS: "./prompts" # Where Augmentoolkit first looks for prompts
API:
  API_KEY: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  BASE_URL: "https://api.openai.com/v1/" # add the base url for a provider, or local server, here. Some possible values:  http://127.0.0.1:5000/v1/ # <- local models. # https://api.together.xyz # <- together.ai, which is real cheap, real flexible, and real high-quality, if a tad unreliable. # https://api.openai.com/v1/ # <- OpenAI. Will bankrupt you very fast. # anything else that accepts OAI-style requests, so basically any API out there (openrouter, fireworks, etc etc etc...)
  LOGICAL_MODEL: "gpt-3.5-turbo-0125" # model used for everything except conversation generation at the very end
  LARGE_LOGICAL_MODEL: "gpt-3.5-turbo-0125" # model used for conversation generation at the very end. A pretty tough task, if ASSISTANT_MODE isn't on.
  QUANTIZATION_SMALL: "gptq" # Only use if Aphrodite mode is on.
  QUANTIZATION_LARGE: "gptq" # Only use if Aphrodite mode is on.
SYSTEM:
  USE_FILENAMES: False # give the AI context from the filenames provided to it. Useful if the filenames are meaningful, otherwise turn them off.
  ASSISTANT_MODE: True # If False, the conversations generated are between a user and an AI assistant. If True, the generated convs are between fictional characters in historical or fictional settings, with randomized personalities (some are nsfw by default, because a lot of model creators make models for that purpose. Change this (or amplify it) in ./augmentoolkit/generation_functions/special_instructions.py, it only requires changes to some strings.)
  DOUBLE_CHECK_COUNTER: 3 # How many times to check a question and answer pair during each validation step. Majority vote decides if it passes that step. There are three steps. So most questions are by default checked around 9 times (fewer if the first two checks for a step pass, obviously).
  USE_SUBSET: True # Whether to take only the first 13 chunks from a text during the run. Useful for experimenting and iterating and seeing all the steps without costing too much money or time.
  REARRANGEMENTS_TO_TAKE: 3 # How many times to rearrange the questions and answers for generating different conversations from the same group of questions and answers.
  CONCURRENCY_LIMIT: 50 # Hard limit of how many calls can be run at the same time, useful for API mode (aphrodite automatically manages this and queues things, as far as I know)
  COMPLETION_MODE: False # Change to false if you want to use chat (instruct) mode; this requires .json files in your chosen prompts directory, in the OpenAI API format. Not all APIs support completion mode.
  MODE: "api" # can be one of "api"|"aphrodite"
  GRAPH: True # Whether to show a pretty graph after filtering out stuff not worthy for questions, useful for seeing whether or not your text is suitable for making data from using Augmentoolkit by default. Will pause the pipeline's execution until you close the window, which is why this is false by default.
  STOP: False # True = Use stop tokens, False = do not use stop tokens. OpenAI's API restricts you to four stop tokens and all steps have way more than four stop tokens, so you'll need to turn this to False if you're using OAI's API. Also NOTE that if you turn this OFF while using COMPLETION MODE, EVERYTHING WILL BREAK and it will cost you money in the process. Don't do that.

As for the data, I went to neovim.io/doc/user/lua-guide and followed that to the source at github.com/neovim/neovim/runtime/doc/lua-guide.txt, I merged all of these into a single document under ./raw_txt_input.

# https://github.com/neovim/neovim/blob/master/runtime/doc/lua-guide.txt
# https://raw.githubusercontent.com/neovim/neovim/master/runtime/doc/lua-guide.txt

out=./raw_txt_input/lua-guide.txt

curl_it() {
    curl $1 >> $out
}

curl_it 'https://raw.githubusercontent.com/neovim/neovim/master/runtime/doc/lua-guide.txt'
curl_it 'https://raw.githubusercontent.com/neovim/neovim/master/runtime/doc/lsp.txt'
curl_it 'https://raw.githubusercontent.com/neovim/neovim/master/runtime/doc/luaref.txt'
curl_it 'https://github.com/neovim/neovim/blob/master/runtime/doc/lua.txt'

Then I ran it with:

source $vdir/bin/activate
python processing.py

I’ll have to keep an eye on the output to make sure it’s not too costly

Next I’ll try it on the broot docs.

This failed because it needs a larger context window, this could get costly with that, I’ll change it but keep USE_SUBSET: True so it doesn’t get costly.

I couldn’t find it on the OpenAI website, but my script:

~/Sync-Archive/Studies/openAI/chat-cli-rs/src/main.rs

Mentioned gpt-3.5-turbo-16k so I’ll try that.

After running it with that I got a lot of output. It started working and I wrote a python script to parse the output:

parse.py

 
#!/usr/bin/env python3
# Path: ~/Sync/Studies/LLM/data-generation/augmentoolkit/json_to_markdown.py
# -*- coding: utf-8 -*-
 
import json
 
with open("output/processed_master_list.json", "r") as f:
    data = json.load(f)
 
 
def main():
    for i, d in enumerate(data):
        print(f"{i+1}. Q/A")
 
        sys = d[0][0]
        user = d[0][1]
        response = d[0][2]
 
        print_md(sys, user, response)
 
 
def print_md(sys, user, response):
    print(
        """
    <details closed><summary>
 
    **Q/A Pair**
    </summary>
    """
    )
 
    print(f"    - {sys}")
    print(f"    - {user}")
    print(f"    - {response}")
    print(
        """
    </details>
    """
    )
 
 
if __name__ == "__main__":
    main()

I’ve saved the output in: - Output

And according to platform.openai.com/usage, that cost me about 5 AUD. This only generated 30 questions, research has shown that 1000 carefully crafted questions can be enough to train a model, iff the data is very good, see Lima. This means that it would cost about 170 AUD for each dataset. Also, the dataset was good, but it was not amazing.

Gap — Running local models and generating cheap data sets is really valuable, I should look into that next.

Next I’ll try re-running that with Oobabooga using the Inferencing Openai Api.

To use Oobabooga I set the the following:

API:
  API_KEY: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  BASE_URL: "http://vale:5000/v1/"
  COMPLETION_MODE: False
  LOGICAL_MODEL: "TheBloke_dolphin-2.5-mixtral-8x7b-GPTQ"
  LARGE_LOGICAL_MODEL: "TheBloke_dolphin-2.5-mixtral-8x7b-GPTQ"

I ensured the model was loaded in Oobabooga with:

  • 2 expert tokens
  • 8000 context
    • too much will exceed memory
  • Exllama_v2_hf

Then I ran nvidia-smi to ensure it was running on the GPU:

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.154.05             Driver Version: 535.154.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4090        Off | 00000000:01:00.0 Off |                  Off |
| 65%   55C    P2             256W / 450W |  23828MiB / 24564MiB |     95%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      2677      G   /usr/libexec/Xorg                            35MiB |
|    0   N/A  N/A     28491      C   python3                                   23780MiB |
+---------------------------------------------------------------------------------------+

Graph View

Backlinks