Pruned MoEs (Mixtral-8x7B-Instruct-v0.1)

Na0s 's Collections

Depth pruned and fine tuned Llama-3.1-8B

updated Nov 18, 2024

Pruned experts from Mixtral-8x7B-Instruct-v0.1 with respect to the paper "A Provably Effective Method for Pruning Experts in Fine-tuned Sparse MoEs"

Upvote

mistralai/Mixtral-8x7B-Instruct-v0.1

47B • Updated 27 days ago • 350k • 4.53k
Na0s/Mixtral-8x7B-Instruct-v0.1-LoRA-on-Gates

Text Generation • 47B • Updated Sep 26, 2024 • 4
Na0s/Mixtral-8x7B-Instruct-v0.1-exhaustive-LoRA

Text Generation • 47B • Updated Nov 12, 2024 • 1
Na0s/Mixtral-8x7B-v0.1-instruct-pruned-random-1-experts

Text Generation • 41B • Updated Oct 6, 2024 • 3
Na0s/Mixtral-8x7B-v0.1-instruct-pruned-random-2-experts

Text Generation • 35B • Updated Oct 6, 2024 • 3
Na0s/Mixtral-8x7B-v0.1-instruct-pruned-random-3-experts

Text Generation • 30B • Updated Oct 6, 2024 • 2
Na0s/Mixtral-8x7B-v0.1-instruct-pruned-random-4-experts

Text Generation • 24B • Updated Oct 6, 2024 • 3
Na0s/Mixtral-8x7B-v0.1-instruct-l2-norm-post-Gates-LoRA-SFT-pruned-1-expert

Text Generation • 41B • Updated Oct 6, 2024 • 3
Na0s/Mixtral-8x7B-v0.1-instruct-l2-norm-post-Gates-LoRA-SFT-pruned-2-experts

Text Generation • 35B • Updated Oct 5, 2024 • 3
Na0s/Mixtral-8x7B-v0.1-instruct-l2-norm-post-Gates-LoRA-SFT-pruned-3-experts

Text Generation • 30B • Updated Oct 6, 2024 • 3
Na0s/Mixtral-8x7B-v0.1-instruct-l2-norm-post-Gates-LoRA-SFT-pruned-4-experts

Text Generation • 24B • Updated Oct 6, 2024 • 3
Na0s/Mixtral-8x7B-Instruct-v0.1-exhaustive-LoRA-SFT-pruned-1-expert

Text Generation • 41B • Updated Nov 18, 2024 • 1
Na0s/Mixtral-8x7B-Instruct-v0.1-exhaustive-LoRA-SFT-pruned-2-experts

Text Generation • 35B • Updated Nov 18, 2024 • 2
Na0s/Mixtral-8x7B-Instruct-v0.1-exhaustive-LoRA-SFT-pruned-3-experts

Text Generation • 30B • Updated Nov 18, 2024 • 1
Na0s/Mixtral-8x7B-Instruct-v0.1-exhaustive-LoRA-SFT-pruned-4-experts

Text Generation • 24B • Updated Nov 18, 2024 • 1

Upvote