I'm going to guess that this is an issue with DNS and DC locator. If you are having to use an IP address to access the server, it is likely that DNS is not working to resolve this AD from the client and therefore DC locator will not work either. The fact that it works on the VM (where DNS likely does work) would seem to support this theory.
Setting up AD test environments can be a little tricky to ensure you get all the dependencies working correctly.
It may be possible to get this to work by changing the client machine's DNS server to be the IP address of the AD VM.