Quantum billiards with surface scattering: Ballistic sigma-model approach