I agree. Further to that I believe protocol is that when speaking, one must address their comments to the speaker as opposed to any member in the chamber. It is mostly decorum and intended to keep a lid on tempers and direct personal exchanges that can flare up into a hubbub. Speaker is intended to be the impartial head master of the school he oversees and keep the house in order and apply the rules fairly. As you well will have seen - there are varying degrees of success in that pursuit.