It could be afraid of being harmed. Perhaps if you show that there's nothing to fear, it'll calm down?
It's definitely worth a shot.
Nino: Put away your sword, but get ready to draw it in case of attack.
You drop your sword close by on the ground. In the worst case scenario, you can just grab it quickly and defend yourself!
The beast stops howling.
Nino: Put your arms up. Show the ghost dog it has nothing to fear from you.
You give it the most friendliest silly-looking waving gesture and step away from your sword. See, you're completely harmless and trustworthy! It's got nothing to fear now!
The beast stops cowering.
Nino: Pet it. Good dog. Best friend.
Now that the beast seems no longer scared, you attempt the most obvious next course of action: Petting it!!!
You give a friendly BECKON to the GHOST DOG.
It complies with a friendly hug. Perhaps way too friendly. Ow.
Ow, it's too OVER FRIENDLY GOSH WHAT A SILLY DOGGY OUCH OVER FRIENDLY DEFINITELY IS THE MOST APPROPRIATE WORD HERE
You successfully end the battle, earning a new achievement for your efforts in the process: THE DOG WHISPERER.
You're not quite sure if achievements are tangible things. Best not to think too much about it.
Five and the GHOST DOG are both being excitedly silly.
tl;dr: Nino notices Ghost-wolf's fear and throws away his sword, and then makes vaguely friendly looking gestures to it. Ghost-wolf embraces his inner Ghost-doggo, which then embraces Nino. Five is happy to have a pet.