Fetching details from the dictionary...
to make matters worse; besides; what's more; in addition; on top of that
疲れて、お負けに風邪も引いた。
I'm tired, and to make it worse, I caught a cold.
失敗して、お負けに怒られた。
I failed, and on top of that, got scolded.
遅れて、お負けに道に迷った。
I was late, and on top of that, got lost.