Penalties for technical violations have exploded, prompting the high-speed trader Virtu to litigate rather than pay.