I'd say the build quality of the S3 or S4 is the same as the One X and One, maybe even better since the One doesn't use anodized aluminum and the One X is made of the same material as all of the Galaxy's (polycarbonate), but has a unibody construction. I've read somewhere that devices with a removable back allows the shock of dropping it to be absorbed better, since the energy gets transmitted to the back cover as it flies off along with the battery. This causes less shock to be absorbed by the more important internals.
So I don't think you mean build quality
at all. I think the complaints in reviews and such mostly stem from Samsung deciding to use a shiny finish.
The One and is the better looking device though. However, Touchwiz is a tad better than Sense. Also, I like using my 64GB microSD card and the lack of a microSD card slot and non-removable battery are pretty big turn offs for me and many others. Plus, you can always change that glossy back cover with an aluminum or less shiny one. Personally, I'm waiting for the Note 3.