Pair breaking due to orbital magnetism in iron-based superconductors