John Leimgruber III

This is a good question, Ed. As we've discussed I'm still developing an intuition for these kinds of things.

In my limited experience there tend to be two more common scenarios:

1. The quantization doesn't damage the model too much and so is not immediately noticeable during inferencing. Probably 𝜌PPL is over 95%
2. The model barely works, can't form sentences, repeats small phrases forever and is very damaged.

Rarely have I seen a situation that is kind of in-between where the model is obviously acting different, but still is somewhat coherent though makes a lot of mistakes. That might be an interesting place to explore for this "cut-off" so to speak. I wish I had more stats on it. Specifically it happened on my first exllamav3 exl3 quantization of a "faile" ParetoQ QAT of a 1B model quantized to 2bpw lol:

https://gist.github.com/ubergarm/9d560bab80241b90dac802e91b656743#references

The drop down there shows the model is somewhat coherent, but definitely goofed up pretty good haha...

Anyway, I'll keep my eye on 𝜌PPL more closely as I'm running a lot of KLD comparisons lately. Cheers!