Someone twice your size should be able to kill you with little trouble.
As you point out you can stand against a Loremaster for an age, whereas a Sorcerer can bring all their skills to a successful conclusion against you quite easily.
Ignoring the fact that it may just be a personal weakness of yours versus a certain profession and that Bards are generally seen as the antithesis of Sorcerers, perhaps what you're arguing for is an increase in Loremaster power?
After all, if one profession twice your size cannot kill you in even roughly the same timescale as another there is something wrong, no?
I look forward to the new Bard specific Loremaster skills that shall address this weakness you've brought up.
Perhaps a simple solution to this dire problem is that dwarfdom could constrict the vocal chords as well as the limbs, rendering you unable to sing or perform elaborate dramatic flourishes.
Written by my hand on the 22nd of Midsummer, in the year 1126.