Not an expert, so happy to be corrected, but for practical purposes you want to have larger gauge cable and connectors on the battery compared to the motor.
The motor shares the current over three phases so is less by a factor of approx 0.58. Ie if the total motor current is 100A the total per phase would be 58A.
The other reason is that the motor cables are just a continuation of the windings. Lower gauge will have little effect to the over all system.
The battery cables on the other hand have to handle any back EMF from the speed controller and you want them as short and as thick as possible to prevent damage to the vesc.