On-call duty teaches what three months of development do not

One night on call teaches you more about the system than three months writing it. It is expensive, it is unfair, and it is the most efficient method that exists. The least you can do is treat whoever takes the on-call shift as someone who knows where the problems are.