สร้างimage_classification_peft_lora.ipynb

image_classification_peft_lora.ipynb

@@ -0,0 +1,712 @@
+Introduction
+In this notebook, we will learn how to use LoRA from 🤗 PEFT to fine-tune an image classification model by ONLY using 0.77% of the original trainable parameters of the model.
+
+LoRA adds low-rank "update matrices" to certain blocks in the underlying model (in this case the attention blocks) and ONLY trains those matrices during fine-tuning. During inference, these update matrices are merged with the original model parameters. For more details, check out the original LoRA paper.
+
+Let's get started by installing the dependencies.
+
+Note that this notebook builds on top the official image classification example notebook.
+
+Install dependencies
+Here we're installing peft from source to ensure we have access to all the bleeding edge features of peft.
+
+!pip install transformers accelerate evaluate datasets git+https://github.com/huggingface/peft -q
+  Installing build dependencies ... done
+  Getting requirements to build wheel ... done
+  Preparing metadata (pyproject.toml) ... done
+     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/6.3 MB 53.1 MB/s eta 0:00:00
+     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 199.7/199.7 KB 24.5 MB/s eta 0:00:00
+     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 81.4/81.4 KB 11.3 MB/s eta 0:00:00
+     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 462.8/462.8 KB 46.9 MB/s eta 0:00:00
+     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 190.3/190.3 KB 23.1 MB/s eta 0:00:00
+     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.6/7.6 MB 102.9 MB/s eta 0:00:00
+     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 213.0/213.0 KB 25.4 MB/s eta 0:00:00
+     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 132.0/132.0 KB 15.2 MB/s eta 0:00:00
+     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 76.3/76.3 MB 23.0 MB/s eta 0:00:00
+     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.6/140.6 KB 20.0 MB/s eta 0:00:00
+  Building wheel for peft (pyproject.toml) ... done
+Authentication
+We will share our fine-tuned model at the end of training. So, to do that we just authenticate using our 🤗 token. This token is available from here. If you don't have a 🤗 account already, we highly encourage you to do so; it's free!
+
+from huggingface_hub import notebook_login
+
+notebook_login()
+Token is valid.
+Your token has been saved in your configured git credential helpers (store).
+Your token has been saved to /root/.cache/huggingface/token
+Login successful
+Check the library versions
+import transformers
+import accelerate
+import peft
+===================================BUG REPORT===================================
+Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
+================================================================================
+print(f"Transformers version: {transformers.__version__}")
+print(f"Accelerate version: {accelerate.__version__}")
+print(f"PEFT version: {peft.__version__}")
+Transformers version: 4.26.0
+Accelerate version: 0.16.0
+PEFT version: 0.1.0.dev0
+Select a model checkpoint to fine-tune
+model_checkpoint = "google/vit-base-patch16-224-in21k"  # pre-trained model from which to fine-tune
+Load a dataset
+We're only loading the first 5000 instances from the training set of the Food-101 dataset to keep this example runtime short.
+
+from datasets import load_dataset
+
+dataset = load_dataset("food101", split="train[:5000]")
+Downloading builder script:   0%|          | 0.00/6.21k [00:00<?, ?B/s]
+Downloading metadata:   0%|          | 0.00/5.56k [00:00<?, ?B/s]
+Downloading readme:   0%|          | 0.00/10.3k [00:00<?, ?B/s]
+Downloading and preparing dataset food101/default to /root/.cache/huggingface/datasets/food101/default/0.0.0/7cebe41a80fb2da3f08fcbef769c8874073a86346f7fb96dc0847d4dfc318295...
+Downloading data:   0%|          | 0.00/5.00G [00:00<?, ?B/s]
+WARNING:datasets.download.download_manager:Computing checksums of downloaded files. They can be used for integrity verification. You can disable this by passing ignore_verifications=True to load_dataset
+Computing checksums: 100%|##########| 1/1 [00:14<00:00, 14.25s/it]
+Downloading data files:   0%|          | 0/2 [00:00<?, ?it/s]
+Downloading data:   0%|          | 0.00/1.47M [00:00<?, ?B/s]
+Downloading data:   0%|          | 0.00/489k [00:00<?, ?B/s]
+Generating train split:   0%|          | 0/75750 [00:00<?, ? examples/s]
+Generating validation split:   0%|          | 0/25250 [00:00<?, ? examples/s]
+Dataset food101 downloaded and prepared to /root/.cache/huggingface/datasets/food101/default/0.0.0/7cebe41a80fb2da3f08fcbef769c8874073a86346f7fb96dc0847d4dfc318295. Subsequent calls will reuse this data.
+Prepare datasets for training and evaluation
+Prepare label2id and id2label dictionaries. This will come in handy when performing inference and for metadata information.
+labels = dataset.features["label"].names
+label2id, id2label = dict(), dict()
+for i, label in enumerate(labels):
+    label2id[label] = i
+    id2label[i] = label
+
+id2label[2]
+'baklava'
+We load the image processor of the model we're fine-tuning.
+from transformers import AutoImageProcessor
+
+image_processor = AutoImageProcessor.from_pretrained(model_checkpoint)
+image_processor
+Downloading (…)rocessor_config.json:   0%|          | 0.00/160 [00:00<?, ?B/s]
+Downloading (…)lve/main/config.json:   0%|          | 0.00/502 [00:00<?, ?B/s]
+ViTImageProcessor {
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "ViTImageProcessor",
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "resample": 2,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "height": 224,
+    "width": 224
+  }
+}
+As one might notice, the image_processor has useful information on which size the training and evaluation images should be resized, stats that should be used to normalize the pixel values, etc.
+
+Using the image processor we prepare transformation functions for the datasets. These functions will include augmentation and pixel scaling.
+from torchvision.transforms import (
+    CenterCrop,
+    Compose,
+    Normalize,
+    RandomHorizontalFlip,
+    RandomResizedCrop,
+    Resize,
+    ToTensor,
+)
+
+normalize = Normalize(mean=image_processor.image_mean, std=image_processor.image_std)
+train_transforms = Compose(
+    [
+        RandomResizedCrop(image_processor.size["height"]),
+        RandomHorizontalFlip(),
+        ToTensor(),
+        normalize,
+    ]
+)
+
+val_transforms = Compose(
+    [
+        Resize(image_processor.size["height"]),
+        CenterCrop(image_processor.size["height"]),
+        ToTensor(),
+        normalize,
+    ]
+)
+
+
+def preprocess_train(example_batch):
+    """Apply train_transforms across a batch."""
+    example_batch["pixel_values"] = [train_transforms(image.convert("RGB")) for image in example_batch["image"]]
+    return example_batch
+
+
+def preprocess_val(example_batch):
+    """Apply val_transforms across a batch."""
+    example_batch["pixel_values"] = [val_transforms(image.convert("RGB")) for image in example_batch["image"]]
+    return example_batch
+We split our mini dataset into training and validation.
+# split up training into training + validation
+splits = dataset.train_test_split(test_size=0.1)
+train_ds = splits["train"]
+val_ds = splits["test"]
+We set the transformation functions to the datasets accordingly.
+train_ds.set_transform(preprocess_train)
+val_ds.set_transform(preprocess_val)
+Load and prepare a model
+In this section, we first load the model we want to fine-tune.
+
+def print_trainable_parameters(model):
+    """
+    Prints the number of trainable parameters in the model.
+    """
+    trainable_params = 0
+    all_param = 0
+    for _, param in model.named_parameters():
+        all_param += param.numel()
+        if param.requires_grad:
+            trainable_params += param.numel()
+    print(
+        f"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param:.2f}"
+    )
+The get_peft_model() method that we will use in a moment wraps the original model to be fine-tuned as a PeftModel. So, it's important for us to initialize the original model correctly. As such, we initialize it by specifying the label2id and id2label so that AutoModelForImageClassification can initialize a append classification head to the underlying model, adapted for our dataset. We can confirm this from the warning below:
+
+Some weights of ViTForImageClassification were not initialized from the model checkpoint at google/vit-base-patch16-224-in21k and are newly initialized: ['classifier.weight', 'classifier.bias']
+from transformers import AutoModelForImageClassification, TrainingArguments, Trainer
+
+model = AutoModelForImageClassification.from_pretrained(
+    model_checkpoint,
+    label2id=label2id,
+    id2label=id2label,
+    ignore_mismatched_sizes=True,  # provide this in case you're planning to fine-tune an already fine-tuned checkpoint
+)
+print_trainable_parameters(model)
+Downloading (…)"pytorch_model.bin";:   0%|          | 0.00/346M [00:00<?, ?B/s]
+Some weights of the model checkpoint at google/vit-base-patch16-224-in21k were not used when initializing ViTForImageClassification: ['pooler.dense.weight', 'pooler.dense.bias']
+- This IS expected if you are initializing ViTForImageClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
+- This IS NOT expected if you are initializing ViTForImageClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
+Some weights of ViTForImageClassification were not initialized from the model checkpoint at google/vit-base-patch16-224-in21k and are newly initialized: ['classifier.bias', 'classifier.weight']
+You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
+trainable params: 85876325 || all params: 85876325 || trainable%: 100.00
+Also, take note of the number of total trainable parameters of model: it's 100%! We'll compare this number to that of the LoRA model.
+
+We now use the PeftModel to wrap model so that the "update" matrices are added to the respective places.
+
+from peft import LoraConfig, get_peft_model
+
+config = LoraConfig(
+    r=16,
+    lora_alpha=16,
+    target_modules=["query", "value"],
+    lora_dropout=0.1,
+    bias="none",
+    modules_to_save=["classifier"],
+)
+lora_model = get_peft_model(model, config)
+print_trainable_parameters(lora_model)
+trainable params: 667493 || all params: 86466149 || trainable%: 0.77
+Let's unpack what's going on here.
+
+In order for LoRA to take effect, we need to specify the target modules to LoraConfig so that get_peft_model() knows which modules inside our model needs to be amended with LoRA matrices. In this case, we're only interested in targetting the query and value matrices of the attention blocks of the base model. Since the parameters corresponding to these matrices are "named" with query and value respectively, we specify them accordingly in the target_modules argument of LoraConfig.
+
+We also specify modules_to_save. After we wrap our base model model with get_peft_model() along with the config, we get a new model where only the LoRA parameters are trainable (so-called "update matrices") while the pre-trained parameters are kept frozen. These include the parameters of the randomly initialized classifier parameters too. This is NOT we want when fine-tuning the base model on our custom dataset. To ensure that the classifier parameters are also trained, we specify modules_to_save. This also ensures that these modules are serialized alongside the LoRA trainable parameters when using utilities like save_pretrained() and push_to_hub().
+
+Regarding the other parameters:
+
+r: The dimension used by the LoRA update matrices.
+alpha: Scaling factor.
+bias: Specifying if the bias parameters should be trained. None denotes none of the bias parameters will be trained.
+r and alpha together control the total number of final trainable parameters when using LoRA giving us the flexbility to balance a trade-off between end performance and compute efficiency.
+
+We can also how many parameters we're actually training. Since we're interested in performing parameter-efficient fine-tuning, we should expect to notice a less number of trainable parameters from the lora_model in comparison to the original model which is indeed the case here.
+
+Training arguments
+We will leverage 🤗 Trainer for fine-tuning. It accepts several arguments which we wrap using TrainingArguments.
+
+from transformers import TrainingArguments, Trainer
+
+
+model_name = model_checkpoint.split("/")[-1]
+batch_size = 128
+
+args = TrainingArguments(
+    f"{model_name}-finetuned-lora-food101",
+    remove_unused_columns=False,
+    evaluation_strategy="epoch",
+    save_strategy="epoch",
+    learning_rate=5e-3,
+    per_device_train_batch_size=batch_size,
+    gradient_accumulation_steps=4,
+    per_device_eval_batch_size=batch_size,
+    fp16=True,
+    num_train_epochs=5,
+    logging_steps=10,
+    load_best_model_at_end=True,
+    metric_for_best_model="accuracy",
+    push_to_hub=True,
+    label_names=["labels"],
+)
+Some things to note here:
+
+We're using a larger batch size since there is only a handful of parameters to train.
+Larger learning rate than the normal (1e-5 for example).
+All of these things are a byproduct of the fact that we're training only a small number of parameters. This can potentially also reduce the need to conduct expensive hyperparameter tuning experiments.
+
+Prepare evaluation metric
+import numpy as np
+import evaluate
+
+metric = evaluate.load("accuracy")
+
+
+# the compute_metrics function takes a Named Tuple as input:
+# predictions, which are the logits of the model as Numpy arrays,
+# and label_ids, which are the ground-truth labels as Numpy arrays.
+def compute_metrics(eval_pred):
+    """Computes accuracy on a batch of predictions"""
+    predictions = np.argmax(eval_pred.predictions, axis=1)
+    return metric.compute(predictions=predictions, references=eval_pred.label_ids)
+Downloading builder script:   0%|          | 0.00/4.20k [00:00<?, ?B/s]
+Collation function
+This is used by Trainer to gather a batch of training and evaluation examples and prepare them in a format that is acceptable by the underlying model.
+
+import torch
+
+
+def collate_fn(examples):
+    pixel_values = torch.stack([example["pixel_values"] for example in examples])
+    labels = torch.tensor([example["label"] for example in examples])
+    return {"pixel_values": pixel_values, "labels": labels}
+Train and evaluate
+trainer = Trainer(
+    model,
+    args,
+    train_dataset=train_ds,
+    eval_dataset=val_ds,
+    tokenizer=image_processor,
+    compute_metrics=compute_metrics,
+    data_collator=collate_fn,
+)
+train_results = trainer.train()
+Cloning https://huggingface.co/sayakpaul/vit-base-patch16-224-in21k-finetuned-lora-food101 into local empty directory.
+WARNING:huggingface_hub.repository:Cloning https://huggingface.co/sayakpaul/vit-base-patch16-224-in21k-finetuned-lora-food101 into local empty directory.
+Download file pytorch_model.bin:   0%|          | 8.00k/330M [00:00<?, ?B/s]
+Download file runs/Feb07_02-43-38_319afa680fd7/1675737843.2328734/events.out.tfevents.1675737843.319afa680fd7.…
+Download file runs/Feb07_02-50-30_319afa680fd7/events.out.tfevents.1675738246.319afa680fd7.10047.0: 100%|#####…
+Download file runs/Feb07_03-56-51_319afa680fd7/1675742273.001745/events.out.tfevents.1675742273.319afa680fd7.2…
+Download file training_args.bin: 100%|##########| 3.50k/3.50k [00:00<?, ?B/s]
+Download file runs/Feb07_03-56-51_319afa680fd7/events.out.tfevents.1675742272.319afa680fd7.27769.0: 100%|#####…
+Clean file runs/Feb07_02-43-38_319afa680fd7/1675737843.2328734/events.out.tfevents.1675737843.319afa680fd7.718…
+Download file runs/Feb07_02-50-30_319afa680fd7/1675738246.1183074/events.out.tfevents.1675738246.319afa680fd7.…
+Clean file runs/Feb07_02-50-30_319afa680fd7/events.out.tfevents.1675738246.319afa680fd7.10047.0:  10%|#       …
+Download file runs/Feb07_02-43-38_319afa680fd7/events.out.tfevents.1675737843.319afa680fd7.7189.0: 100%|######…
+Clean file runs/Feb07_03-56-51_319afa680fd7/1675742273.001745/events.out.tfevents.1675742273.319afa680fd7.2776…
+Clean file training_args.bin:  29%|##8       | 1.00k/3.50k [00:00<?, ?B/s]
+Clean file runs/Feb07_03-56-51_319afa680fd7/events.out.tfevents.1675742272.319afa680fd7.27769.0:   9%|9       …
+Clean file runs/Feb07_02-50-30_319afa680fd7/1675738246.1183074/events.out.tfevents.1675738246.319afa680fd7.100…
+Clean file runs/Feb07_02-43-38_319afa680fd7/events.out.tfevents.1675737843.319afa680fd7.7189.0:  10%|#        …
+Download file runs/Feb07_02-50-30_319afa680fd7/events.out.tfevents.1675738403.319afa680fd7.10047.2: 100%|#####…
+Clean file runs/Feb07_02-50-30_319afa680fd7/events.out.tfevents.1675738403.319afa680fd7.10047.2: 100%|########…
+Clean file pytorch_model.bin:   0%|          | 1.00k/330M [00:00<?, ?B/s]
+Using cuda_amp half precision backend
+/usr/local/lib/python3.8/dist-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning
+  warnings.warn(
+***** Running training *****
+  Num examples = 4500
+  Num Epochs = 5
+  Instantaneous batch size per device = 128
+  Total train batch size (w. parallel, distributed & accumulation) = 512
+  Gradient Accumulation steps = 4
+  Total optimization steps = 45
+  Number of trainable parameters = 667493
+ [45/45 04:44, Epoch 5/5]
+Epoch	Training Loss	Validation Loss	Accuracy
+1	No log	0.506871	0.896000
+2	2.162700	0.189141	0.946000
+3	0.345100	0.144759	0.960000
+4	0.211600	0.150886	0.958000
+5	0.171100	0.149751	0.958000
+
+***** Running Evaluation *****
+  Num examples = 500
+  Batch size = 128
+Saving model checkpoint to vit-base-patch16-224-in21k-finetuned-lora-food101/checkpoint-9
+Configuration saved in vit-base-patch16-224-in21k-finetuned-lora-food101/checkpoint-9/config.json
+Model weights saved in vit-base-patch16-224-in21k-finetuned-lora-food101/checkpoint-9/pytorch_model.bin
+Image processor saved in vit-base-patch16-224-in21k-finetuned-lora-food101/checkpoint-9/preprocessor_config.json
+Image processor saved in vit-base-patch16-224-in21k-finetuned-lora-food101/preprocessor_config.json
+***** Running Evaluation *****
+  Num examples = 500
+  Batch size = 128
+Saving model checkpoint to vit-base-patch16-224-in21k-finetuned-lora-food101/checkpoint-18
+Configuration saved in vit-base-patch16-224-in21k-finetuned-lora-food101/checkpoint-18/config.json
+Model weights saved in vit-base-patch16-224-in21k-finetuned-lora-food101/checkpoint-18/pytorch_model.bin
+Image processor saved in vit-base-patch16-224-in21k-finetuned-lora-food101/checkpoint-18/preprocessor_config.json
+Image processor saved in vit-base-patch16-224-in21k-finetuned-lora-food101/preprocessor_config.json
+***** Running Evaluation *****
+  Num examples = 500
+  Batch size = 128
+Saving model checkpoint to vit-base-patch16-224-in21k-finetuned-lora-food101/checkpoint-27
+Configuration saved in vit-base-patch16-224-in21k-finetuned-lora-food101/checkpoint-27/config.json
+Model weights saved in vit-base-patch16-224-in21k-finetuned-lora-food101/checkpoint-27/pytorch_model.bin
+Image processor saved in vit-base-patch16-224-in21k-finetuned-lora-food101/checkpoint-27/preprocessor_config.json
+Image processor saved in vit-base-patch16-224-in21k-finetuned-lora-food101/preprocessor_config.json
+***** Running Evaluation *****
+  Num examples = 500
+  Batch size = 128
+Saving model checkpoint to vit-base-patch16-224-in21k-finetuned-lora-food101/checkpoint-36
+Configuration saved in vit-base-patch16-224-in21k-finetuned-lora-food101/checkpoint-36/config.json
+Model weights saved in vit-base-patch16-224-in21k-finetuned-lora-food101/checkpoint-36/pytorch_model.bin
+Image processor saved in vit-base-patch16-224-in21k-finetuned-lora-food101/checkpoint-36/preprocessor_config.json
+Image processor saved in vit-base-patch16-224-in21k-finetuned-lora-food101/preprocessor_config.json
+***** Running Evaluation *****
+  Num examples = 500
+  Batch size = 128
+Saving model checkpoint to vit-base-patch16-224-in21k-finetuned-lora-food101/checkpoint-45
+Configuration saved in vit-base-patch16-224-in21k-finetuned-lora-food101/checkpoint-45/config.json
+Model weights saved in vit-base-patch16-224-in21k-finetuned-lora-food101/checkpoint-45/pytorch_model.bin
+Image processor saved in vit-base-patch16-224-in21k-finetuned-lora-food101/checkpoint-45/preprocessor_config.json
+Image processor saved in vit-base-patch16-224-in21k-finetuned-lora-food101/preprocessor_config.json
+
+
+Training completed. Do not forget to share your model on huggingface.co/models =)
+
+
+Loading best model from vit-base-patch16-224-in21k-finetuned-lora-food101/checkpoint-27 (score: 0.96).
+In just a few minutes, we have a fine-tuned model with 96% validation accuracy. Also, note that we used a very small subset of the training dataset which is definitely impacting the results.
+
+trainer.evaluate(val_ds)
+***** Running Evaluation *****
+  Num examples = 500
+  Batch size = 128
+ [4/4 01:48]
+{'eval_loss': 0.14475855231285095,
+ 'eval_accuracy': 0.96,
+ 'eval_runtime': 3.5725,
+ 'eval_samples_per_second': 139.958,
+ 'eval_steps_per_second': 1.12,
+ 'epoch': 5.0}
+Sharing your model and inference
+Once the fine-tuning is done, we can share the LoRA parameters with the community like so:
+
+repo_name = f"sayakpaul/{model_name}-finetuned-lora-food101"
+lora_model.push_to_hub(repo_name)
+Uploading the following files to sayakpaul/vit-base-patch16-224-in21k-finetuned-lora-food101: adapter_config.json,adapter_model.bin
+Upload 1 LFS files:   0%|          | 0/1 [00:00<?, ?it/s]
+adapter_model.bin:   0%|          | 0.00/2.69M [00:00<?, ?B/s]
+CommitInfo(commit_url='https://huggingface.co/sayakpaul/vit-base-patch16-224-in21k-finetuned-lora-food101/commit/64e17d1cda300041cbc67428242a3136060772a3', commit_message='Upload model', commit_description='', oid='64e17d1cda300041cbc67428242a3136060772a3', pr_url=None, pr_revision=None, pr_num=None)
+When we call push_to_hub() on the lora_model, only the LoRA parameters along with any modules specified in modules_to_save are saved. If we take a look at the trained LoRA parameters, we see that it's only 2.6 MB! This greatly helps with portability especially when we're using a very large model to fine-tune (such as BLOOM).
+
+Next, we see how to load the LoRA updated parameters along with our base model for inference. When we wrap a base model with PeftModel that modifications are DONE in place. So to mitigate any concerns that might stem from in place modifications, we newly initialize our base model just like we did earlier and construct our inference model.
+
+from peft import PeftConfig, PeftModel
+
+
+config = PeftConfig.from_pretrained(repo_name)
+model = model = AutoModelForImageClassification.from_pretrained(
+    config.base_model_name_or_path,
+    label2id=label2id,
+    id2label=id2label,
+    ignore_mismatched_sizes=True,  # provide this in case you're planning to fine-tune an already fine-tuned checkpoint
+)
+# Load the Lora model
+inference_model = PeftModel.from_pretrained(model, repo_name)
+loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--google--vit-base-patch16-224-in21k/snapshots/1ba429d32753f33a0660b80ac6f43a3c80c18938/config.json
+Model config ViTConfig {
+  "_name_or_path": "google/vit-base-patch16-224-in21k",
+  "architectures": [
+    "ViTModel"
+  ],
+  "attention_probs_dropout_prob": 0.0,
+  "encoder_stride": 16,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.0,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "apple_pie",
+    "1": "baby_back_ribs",
+    "2": "baklava",
+    "3": "beef_carpaccio",
+    "4": "beef_tartare",
+    "5": "beet_salad",
+    "6": "beignets",
+    "7": "bibimbap",
+    "8": "bread_pudding",
+    "9": "breakfast_burrito",
+    "10": "bruschetta",
+    "11": "caesar_salad",
+    "12": "cannoli",
+    "13": "caprese_salad",
+    "14": "carrot_cake",
+    "15": "ceviche",
+    "16": "cheesecake",
+    "17": "cheese_plate",
+    "18": "chicken_curry",
+    "19": "chicken_quesadilla",
+    "20": "chicken_wings",
+    "21": "chocolate_cake",
+    "22": "chocolate_mousse",
+    "23": "churros",
+    "24": "clam_chowder",
+    "25": "club_sandwich",
+    "26": "crab_cakes",
+    "27": "creme_brulee",
+    "28": "croque_madame",
+    "29": "cup_cakes",
+    "30": "deviled_eggs",
+    "31": "donuts",
+    "32": "dumplings",
+    "33": "edamame",
+    "34": "eggs_benedict",
+    "35": "escargots",
+    "36": "falafel",
+    "37": "filet_mignon",
+    "38": "fish_and_chips",
+    "39": "foie_gras",
+    "40": "french_fries",
+    "41": "french_onion_soup",
+    "42": "french_toast",
+    "43": "fried_calamari",
+    "44": "fried_rice",
+    "45": "frozen_yogurt",
+    "46": "garlic_bread",
+    "47": "gnocchi",
+    "48": "greek_salad",
+    "49": "grilled_cheese_sandwich",
+    "50": "grilled_salmon",
+    "51": "guacamole",
+    "52": "gyoza",
+    "53": "hamburger",
+    "54": "hot_and_sour_soup",
+    "55": "hot_dog",
+    "56": "huevos_rancheros",
+    "57": "hummus",
+    "58": "ice_cream",
+    "59": "lasagna",
+    "60": "lobster_bisque",
+    "61": "lobster_roll_sandwich",
+    "62": "macaroni_and_cheese",
+    "63": "macarons",
+    "64": "miso_soup",
+    "65": "mussels",
+    "66": "nachos",
+    "67": "omelette",
+    "68": "onion_rings",
+    "69": "oysters",
+    "70": "pad_thai",
+    "71": "paella",
+    "72": "pancakes",
+    "73": "panna_cotta",
+    "74": "peking_duck",
+    "75": "pho",
+    "76": "pizza",
+    "77": "pork_chop",
+    "78": "poutine",
+    "79": "prime_rib",
+    "80": "pulled_pork_sandwich",
+    "81": "ramen",
+    "82": "ravioli",
+    "83": "red_velvet_cake",
+    "84": "risotto",
+    "85": "samosa",
+    "86": "sashimi",
+    "87": "scallops",
+    "88": "seaweed_salad",
+    "89": "shrimp_and_grits",
+    "90": "spaghetti_bolognese",
+    "91": "spaghetti_carbonara",
+    "92": "spring_rolls",
+    "93": "steak",
+    "94": "strawberry_shortcake",
+    "95": "sushi",
+    "96": "tacos",
+    "97": "takoyaki",
+    "98": "tiramisu",
+    "99": "tuna_tartare",
+    "100": "waffles"
+  },
+  "image_size": 224,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "apple_pie": 0,
+    "baby_back_ribs": 1,
+    "baklava": 2,
+    "beef_carpaccio": 3,
+    "beef_tartare": 4,
+    "beet_salad": 5,
+    "beignets": 6,
+    "bibimbap": 7,
+    "bread_pudding": 8,
+    "breakfast_burrito": 9,
+    "bruschetta": 10,
+    "caesar_salad": 11,
+    "cannoli": 12,
+    "caprese_salad": 13,
+    "carrot_cake": 14,
+    "ceviche": 15,
+    "cheese_plate": 17,
+    "cheesecake": 16,
+    "chicken_curry": 18,
+    "chicken_quesadilla": 19,
+    "chicken_wings": 20,
+    "chocolate_cake": 21,
+    "chocolate_mousse": 22,
+    "churros": 23,
+    "clam_chowder": 24,
+    "club_sandwich": 25,
+    "crab_cakes": 26,
+    "creme_brulee": 27,
+    "croque_madame": 28,
+    "cup_cakes": 29,
+    "deviled_eggs": 30,
+    "donuts": 31,
+    "dumplings": 32,
+    "edamame": 33,
+    "eggs_benedict": 34,
+    "escargots": 35,
+    "falafel": 36,
+    "filet_mignon": 37,
+    "fish_and_chips": 38,
+    "foie_gras": 39,
+    "french_fries": 40,
+    "french_onion_soup": 41,
+    "french_toast": 42,
+    "fried_calamari": 43,
+    "fried_rice": 44,
+    "frozen_yogurt": 45,
+    "garlic_bread": 46,
+    "gnocchi": 47,
+    "greek_salad": 48,
+    "grilled_cheese_sandwich": 49,
+    "grilled_salmon": 50,
+    "guacamole": 51,
+    "gyoza": 52,
+    "hamburger": 53,
+    "hot_and_sour_soup": 54,
+    "hot_dog": 55,
+    "huevos_rancheros": 56,
+    "hummus": 57,
+    "ice_cream": 58,
+    "lasagna": 59,
+    "lobster_bisque": 60,
+    "lobster_roll_sandwich": 61,
+    "macaroni_and_cheese": 62,
+    "macarons": 63,
+    "miso_soup": 64,
+    "mussels": 65,
+    "nachos": 66,
+    "omelette": 67,
+    "onion_rings": 68,
+    "oysters": 69,
+    "pad_thai": 70,
+    "paella": 71,
+    "pancakes": 72,
+    "panna_cotta": 73,
+    "peking_duck": 74,
+    "pho": 75,
+    "pizza": 76,
+    "pork_chop": 77,
+    "poutine": 78,
+    "prime_rib": 79,
+    "pulled_pork_sandwich": 80,
+    "ramen": 81,
+    "ravioli": 82,
+    "red_velvet_cake": 83,
+    "risotto": 84,
+    "samosa": 85,
+    "sashimi": 86,
+    "scallops": 87,
+    "seaweed_salad": 88,
+    "shrimp_and_grits": 89,
+    "spaghetti_bolognese": 90,
+    "spaghetti_carbonara": 91,
+    "spring_rolls": 92,
+    "steak": 93,
+    "strawberry_shortcake": 94,
+    "sushi": 95,
+    "tacos": 96,
+    "takoyaki": 97,
+    "tiramisu": 98,
+    "tuna_tartare": 99,
+    "waffles": 100
+  },
+  "layer_norm_eps": 1e-12,
+  "model_type": "vit",
+  "num_attention_heads": 12,
+  "num_channels": 3,
+  "num_hidden_layers": 12,
+  "patch_size": 16,
+  "qkv_bias": true,
+  "transformers_version": "4.26.0"
+}
+
+loading weights file pytorch_model.bin from cache at /root/.cache/huggingface/hub/models--google--vit-base-patch16-224-in21k/snapshots/1ba429d32753f33a0660b80ac6f43a3c80c18938/pytorch_model.bin
+Some weights of the model checkpoint at google/vit-base-patch16-224-in21k were not used when initializing ViTForImageClassification: ['pooler.dense.weight', 'pooler.dense.bias']
+- This IS expected if you are initializing ViTForImageClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
+- This IS NOT expected if you are initializing ViTForImageClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
+Some weights of ViTForImageClassification were not initialized from the model checkpoint at google/vit-base-patch16-224-in21k and are newly initialized: ['classifier.bias', 'classifier.weight']
+You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
+Downloading (…)"adapter_model.bin";:   0%|          | 0.00/2.69M [00:00<?, ?B/s]
+Don't worry about the warnings, they're harmless.
+
+Let's now fetch a sample for inference.
+
+from PIL import Image
+import requests
+
+url = "https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/beignets.jpeg"
+image = Image.open(requests.get(url, stream=True).raw)
+image
+
+We first instantiate an image_processor from the underlying model repo.
+
+image_processor = AutoImageProcessor.from_pretrained(repo_name)
+loading configuration file preprocessor_config.json from cache at /root/.cache/huggingface/hub/models--sayakpaul--vit-base-patch16-224-in21k-finetuned-lora-food101/snapshots/fa2503cc7d91e0dd69728c1dc66ed80d7bd3289b/preprocessor_config.json
+Image processor ViTImageProcessor {
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "ViTImageProcessor",
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "resample": 2,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "height": 224,
+    "width": 224
+  }
+}
+
+We then prepare the sample for inference.
+
+# prepare image for the model
+encoding = image_processor(image.convert("RGB"), return_tensors="pt")
+print(encoding.pixel_values.shape)
+torch.Size([1, 3, 224, 224])
+And run inference!
+
+import torch
+
+# forward pass
+with torch.no_grad():
+    outputs = inference_model(**encoding)
+    logits = outputs.logits
+
+predicted_class_idx = logits.argmax(-1).item()
+print("Predicted class:", inference_model.config.id2label[predicted_class_idx])
+Predicted class: beignets