Depends on the effect that the LDS temple has on the community center surrounding it. Yes, I am aware of their policies. Their huge temple here in DC was built with the promise that it would be open to the public, and it was. For a day. Then they closed it, and tore out all the carpets. They have a visitor center that unbelievers can go into.

In the case at hand, neither the community center nor the mosque requires that you be a muslim to enter. If they pull a switcheroo and change that policy, I will happily say "you told me so" to the people who object, and agree with them, but until then...