Does someone need a reason to issue you a challenge?
In a team fight, they are offering you the opportunity for an uninterrupted and fair one on one fight. If you think you have a chance, accept. If you think you don't, then stay with your team and hope they will be deterred.
In many team fights, personal scores amount and demand settling. In some team fights, by taking two of the protagonists away you will change the fight significantly.
If you don't want to die to your challenger, go and hide. If your attacker succeeds, then their success is all the more having given you the warning. The issuing of a public challenge is an honourable thing.
Treat it as such and your honour is upheld. But please don't just complain because the stakes have been raised.
Written by my hand on the 18th of Eleuthral, in the year 1045.