Interesting you should mention this as I've been having this discussion at work on a similar topic just recently. :)
The problem is that the URL bookmark character, the # sign, is NOT sent by the client to the server when it submits a GET or POST request. The browser remembers that client side and adds it back in as needed, but it is never sent to the server.
Any forms-based authentication method relies on a URL retention mechanism to return you to the URL you requested after you login. However, because this part of the URL is never sent to the server at all in the GET request, the ADFS agent cannot retain it and thus it is not "there" when you are redirected back. This is what you are seeing.
The only solution is to ensure that you use URLs that ARE fully sent to the server, at least during the initial request where you will be authenticated. A query string might work instead. However, it will be up to you to figure out how to deal with this. You might not be able to use the navigation framework you are using now or you might need to implement some type of a workaround. However, you can't fix this on the ADFS side or with any other forms-based auth mechanism because the server simply never sees this in the request.
I hope this doesn't turn out to be too disruptive for you.