---
language: en
license: mit
tags:
  - pytorch
  - causal-lm
  - language-model
  - flash-attention
datasets:
  - Salesforce/wikitext
pipeline_tag: question-answering
---

# PurelyUnfunctionalAI/GibberishGPT

A lightweight decoder-only transformer language model trained with Flash Attention on the WikiText dataset. This is a version used for learning about training LLMs and ML pipelines. The model does not actually output coherent text, although serves as a good starting point for learning more about LLMs
<a  href="https://github.com/PUFAI/GibberishGPT"> <img  alt="GitHub"  src="https://img.shields.io/badge/GitHub-Repo-blue?logo=github&style=flat-square"> </a>

## Model Details

- **Model Type:** Causal Language Model
- **Architecture:** Decoder-only Transformer
- **Embedding Size:** 512
- **Hidden Layers:** 8
- **Attention Heads:** 8
- **Context Length:** 512
- **Flash Attention:** Enabled
- **Training Data:** Salesforce/wikitext

## Usage

```python
import torch
import tiktoken
from transformers import AutoModelForCausalLM

# Load the tokenizer
tokenizer = tiktoken.get_encoding("gpt2")

# Load the model
model = AutoModelForCausalLM.from_pretrained("PurelyUnfunctionalAI/GibberishGPT")

# Encode input
input_text = "Your prompt here"
input_ids = tokenizer.encode(input_text)
input_tensor = torch.tensor([input_ids], dtype=torch.long)

# Generate
output = model.generate(input_tensor, max_length=100)
generated_text = tokenizer.decode(output[0].tolist())
print(generated_text)
```

# Limitations

- The model has a context length of 512 tokens
- It was trained on WikiText data which may not cover specialized domains
- As a lightweight model, it may not perform as well as larger LLMs on complex tasks

# Citation
If you use this model in your research, please cite:

```
@misc{GibberishGPT,
  author = {Gathara, Michael and Menon, Vaishak and Liu, Jason},
  title = {GibberishGPT: A Lightweight Language Model with Flash Attention},
  year = {2025},
  publisher = {Hugging Face},
  journal = {Hugging Face model repository},
  howpublished = {\url{https://huggingface.co/PurelyUnfunctionalAI/GibberishGPT}}
}
```