Not all of my answers are sourced, some are based on my general reading of ethics courses, they are still accurate.
I want to prove it is immoral.
You can never prove that it is immoral. Immorality is only relative to the ethical system chosen. You can however argue, that even for a consequentialist, slavery can still be immoral.
What more could I say?
You can't say anything: A hardened consequentialist might simply bite the proverbial bullet and say that, yes indeed, a small amount of slave labor is justified for the greater good. This for example was one of the justifications that pro-slavery "intellectuals" in the American South advanced in the lead up to the American Civil war.
You can argue that in theory a consequentialist should accept a small amount of slavery if it benefits the greater good, but that in practice it is impossible to perform the necessary utility calculus that allows her/him to determine that the greater good achieved by the building is indeed higher than the harm inflicted on the slaves. Similarly, using probabilistic reasoning, you can argue that the harm coming from slavery is predictable and certain, while the benefit coming from the building is uncertain and liable to change (if for example a fire or an earthquake destroys the building right after it is built). Once you factor such probabilities/risk analysis into your utilities calculus, then slavery becomes an immoral choice. This what an Act Utilitarian would do (See IEP article on Utilitarianism, Act Utilitarianism vs Rule Utilitarianism)
You can argue that the harm of slavery is incomparable to the harm caused by the lack of such a building, or that even if it was numerically comparable, it so much greater that it counts as infinitely greater harm, and therefore can never be justified. See Alastair Norcross, “Comparing Harms: Headaches and Human Lives”, Philosophy and Public Affairs, 1997. Sections I and II.
You can argue that the long term harm of slavery due to its detrimental effects on society far outweigh any benefit coming out of the building. For example in a society where slavery is permitted, those who are enslaved or who fear enslavement might resort to violence and terrorism to avoid enslavement. Such a society would be living in constant fear of the violence that might erupt because of slavery, and this cancels any beneficial effects that cheap and efficient slave labor might provide. This would be a variation on Rule Utilitarianism, again see IEP.
You can argue along the lines of G.E. Moore's ideal utilitarianism("Principia Ethica", 1903), that although a small amount of slave labor does allow us to achieve a greater good from constructing the building, there is a scenario of even greater good, where the same outcome is achieved without using slave labor. Per Moore, if such a scenario is possible, then using slave labor to complete the building is immoral. You can then argue that there will always be situations where the outcome is achievable without resorting to slave labor, hence the scenario you describe is immoral.