I would use "You're not wrong" in response to an observation that is totally correct but probably shouldn't be said out loud. Hope this helps!