Would there ever be a GGUF for the gemma-3n models ?
Understand that Gemma 3N is specifically for edge devices, but a good efficient, highly optimized, open, lightweight model could be useful for non-edge cases as well. So wondering if we might expect to see a GGUF for these family of models anytime soon ?
Yes, this repository is just a preview. Stay tuned!
would really love to try it on my macbook locally
Yes want to run this on ollama. This gemma 3n seems like a perfect fit! Thanks team :)
Gemma 3n is based on the MatFormer architecture so the inference engine probably need to be significantly updated instead of just having this in GGUF format
would really love to try it on my macbook locally
You can, using MediaPipe. Built an iOS/MacOS app yesterday. It downloads both models (2B and 4B) from Hugging Face (auth first to get token and accept Google TOS) and it runs very well in Mac and high-end iPhones. https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference/ios
would really love to try it on my macbook locally
You can, using MediaPipe. Built an iOS/MacOS app yesterday. It downloads both models (2B and 4B) from Hugging Face (auth first to get token and accept Google TOS) and it runs very well in Mac and high-end iPhones. https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference/ios
So while documentation says iOS, it works on MacOS too ? Interesting option. I'd still prefer GGUF (if feasible) as it allows me to use it as an option in my current tooling and workflow.
would really love to try it on my macbook locally
You can, using MediaPipe. Built an iOS/MacOS app yesterday. It downloads both models (2B and 4B) from Hugging Face (auth first to get token and accept Google TOS) and it runs very well in Mac and high-end iPhones. https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference/ios
So while documentation says iOS, it works on MacOS too ? Interesting option. I'd still prefer GGUF (if feasible) as it allows me to use it as an option in my current tooling and workflow.
Sorry, Mac (designed for iPad), not MacOS. It works great for testing, btw.
Anyone know about how much we must wait until gguf?