A vote to leave wasn’t always vote to leave the single market