I originally thought that you had to have SSL end to end as well and that the agent and fed server actually checked for this in code, but I think I heard from one of the other forum participants that you can actually get this to work with SSL termination at the load balancer. You'll need SSL on the web servers themselves when you do the initial install but I think it does still work if you turn it off.
Please remember that from a protocol requiremments perspective, fed passive (what ADFS uses) REQUIRES SSL for security because the SAML tokens that are used do not have proof of possession qualities and are subject to replay attacks otherwise. As such, if you do not have SSL at some point in the chain, you need to be absolutely certain that the network segment where plain HTTP is used is totally secure.