Faulkner is speaking through the mouth of Anse (Pa). It is a horribly dysfunctional family (like most) with layers and layers of psychological drama. When one son seemed to go more nuts than the rest and another son raised the question of the sanity of his brother (after his brother burned down their host's barn and then was picking a fight when someone commented that his dead mother was stinking) his father said the following (in his Mississippi dialect):
Sometimes I aint so sho who's got ere a right to say when a man is crazy and when he aint. Sometimes I think aint none of us pure crazy and aint none of us pure sane until the balance of us talks him that-a-way. It's like it aint so much what a fellow does, but it's the way the majority of folks is looking at him when he does it.