The fan socket is implemented from VESC AUX port:
Enable it on the VESC tool PC edition:
The MOSFET of AUX control is a low-side switch, so connect your fan's positive wire to '12V' pin, and the negative wire to the 'FAN-', check them when the fan not work properly. And the fan should be 12V spec.