---
inference: false
datasets:
- bigcode/commitpackft
model-index:
  - name: patched-coder-34b
    results:
      - task:
          type: text-generation
        dataset:
          type: openai_humaneval
          name: HumanEval
        metrics:
          - name: pass@1
            type: pass@1
            value: 53.567
            verified: false
      - task:
          type: text-generation
        dataset:
          type: bigcode/humanevalpack
          name: HumanEvalFix Python
        metrics:
          - name: pass@1
            type: pass@1
            value: 41.341
            verified: false
      - task:
          type: text-generation
        dataset:
          type: patched-codes/static-analysis-eval
          name: Static Analysis Eval
        metrics:
          - name: pass@1
            type: pass@1
            value: 51.316
            verified: false
---
# Model Card for patched-coder-34b


This is an instruction fine-tuned model focussed on the task of patching code. Patching may include fixing bugs, remediating security vulnerabilities, 
doing API migrations and other kinds of code matainence.

## Model Details

### Model Description

- **Developed by:** [codelion](https://huggingface.co/codelion)
- **Model type:** Code Llama
- **Finetuned from model:** [CodeLlama-34b-Python](https://huggingface.co/codellama/CodeLlama-34b-Python-hf)


## How to Get Started with the Model

Make sure to install Transformers from the main git branch:

```bash
pip install git+https://github.com/huggingface/transformers.git
```

## How to Prompt the Model

This model accepts the alpaca instruction format.

For example: 

```
### Instruction:
{instruction}

### Input:
{input}

### Response:
...
```

## Bias, Risks, and Limitations

This model has undergone very limited testing. Additional safety testing should be performed before any real-world deployments.

## Training Details

- **GPU:** A100 80 GB
- **Time:** ~8 hrs

### Training Data

The model was fine-tuned on [commitpackft](https://huggingface.co/datasets/bigcode/commitpackft), an open dataset consisting of commits.
We started with the commits for the `python` langauge from the dataset and then filtered all the commits that were related to fixing bugs. 

### Training Procedure 

Instruction fine-tuning to follow instructions in natural langauge related to code. We load the quantized base model in 4 bits 
and then use QLoRA for Parameter-Efficient Fine-Tuning (PEFT) with Flash Attention. The model was trained for 2 epochs.

#### Training Hyperparameters

**Training regime:** 

The following `bitsandbytes` quantization config was used during training:
  - quant_method: bitsandbytes
  - load_in_8bit: False
  - load_in_4bit: True
  - llm_int8_threshold: 6.0
  - llm_int8_skip_modules: None
  - llm_int8_enable_fp32_cpu_offload: False
  - llm_int8_has_fp16_weight: False
  - bnb_4bit_quant_type: nf4
  - bnb_4bit_use_double_quant: True
  - bnb_4bit_compute_dtype: bfloat16

## Evaluation

We evaluate the model on `HumanEval` and `HumanEvalPack` benchmarks using 
[Code Generation LM Evaluation Harness](https://github.com/bigcode-project/bigcode-evaluation-harness).

We also evaluate the model for vulnerability remediation using the `Static Analysis Eval` benchmark available [here](https://huggingface.co/datasets/patched-codes/static-analysis-eval).

### Results