We explore the possibility that the distinct patterns observed in fermion masses and mixings may be due to a minimally-broken $U(2)$ flavor symmetry giving rise to an accidental $U(2)^5$ symmetry in the Yukawa couplings of the charged fermions. In model-independent analysis we find that the selection rules of the $U(2)$ symmetry enhances the importance of charge lepton flavor violation as a probe for new physics. I will also present an economical realization of the framework with a renormalizable model, which links all flavor hierarchies to a single hierarchy of scales at high energies. The model has a rich phenomenology with promising signatures, especially in the context of $K$ and $B$ meson physics.