You'll never guess who a weaker Sommers is screaming "Come on then" as a battle cry to, while having one arm.
Sommers is not afraid of Conqueror Haki users nor does he backs down from those who know about his hearts.
His fear stem from how Powerful Zoro became.
Talking to Zoro fans is hard sometimes
The difference in Sommers reaction is because Zoro touched his heart, he still freaks out and acts stupid and desperate when faced with anyone with conquerors his fight with Gaban is literally him spamming attacks to try an keep distance