In the B-edition of the first Critique, Kant asserts the existence of the world when we're not looking at it in two main stretches. The more famous section is "The Refutation of Idealism":
I am conscious of my own existence as determined in time. All determination in regard to time presupposes the existence of something permanent in perception. But this permanent something cannot be something in me, for the very reason that my existence in time is itself determined by this permanent something. It follows that the perception of this permanent existence is possible only through a thing without me and not through the mere representation of a thing without me. Consequently, the determination of my existence in time is possible only through the existence of real things external to me. Now, consciousness in time is necessarily connected with the consciousness of the possibility of this determination in time. Hence it follows that consciousness in time is necessarily connected also with the existence of things without me, inasmuch as the existence of these things is the condition of determination in time. That is to say, the consciousness of my own existence is at the same time an immediate consciousness of the existence of other things without me.
Remark I. The reader will observe, that in the foregoing proof the game which idealism plays is retorted upon itself, and with more justice. It assumed that the only immediate experience is internal and that from this we can only infer the existence of external things. But, as always happens, when we reason from given effects to determined causes, idealism has reasoned with too much haste and uncertainty, for it is quite possible that the cause of our representations may lie in ourselves, and that we ascribe it falsely to external things. But our proof shows that external experience is properly immediate, that only by virtue of it—not, indeed, the consciousness of our own existence, but certainly the determination of our existence in time, that is, internal experience—is possible. It is true, that the representation "I am," which is the expression of the consciousness which can accompany all my thoughts, is that which immediately includes the existence of a subject. But in this representation we cannot find any knowledge of the subject, and therefore also no empirical knowledge, that is, experience. For experience contains, in addition to the thought of something existing, intuition, and in this case it must be internal intuition, that is, time, in relation to which the subject must be determined. But the existence of external things is absolutely requisite for this purpose, so that it follows that internal experience is itself possible only mediately and through external experience.
The immediate consciousness of the existence of external things is, in the preceding theorem, not presupposed, but proved, be the possibility of this consciousness understood by us or not. The question as to the possibility of it would stand thus: "Have we an internal sense, but no external sense, and is our belief in external perception a mere delusion?" But it is evident that, in order merely to fancy to ourselves anything as external, that is, to present it to the sense in intuition we must already possess an external sense, and must thereby distinguish immediately the mere receptivity of an external intuition from the spontaneity which characterizes every act of imagination. For merely to imagine also an external sense, would annihilate the faculty of intuition itself which is to be determined by the imagination.
He also says earlier, regarding the third analogy of experience:
Things are coexistent, when in empirical intuition the perception of the one can follow upon the perception of the other, and vice versa—which cannot occur in the succession of phenomena, as we have shown in the explanation of the second principle. Thus I can perceive the moon and then the earth, or conversely, first the earth and then the moon; and for the reason that my perceptions of these objects can reciprocally follow each other, I say, they exist contemporaneously. Now coexistence is the existence of the manifold in the same time. But time itself is not an object of perception; and therefore we cannot conclude from the fact that things are placed in the same time, the other fact, that the perception of these things can follow each other reciprocally. The synthesis of the imagination in apprehension would only present to us each of these perceptions as present in the subject when the other is not present, and contrariwise; but would not show that the objects are coexistent, that is to say, that, if the one exists, the other also exists in the same time, and that this is necessarily so, in order that the perceptions may be capable of following each other reciprocally. It follows that a conception of the understanding or category of the reciprocal sequence of the determinations of phenomena (existing, as they do, apart from each other, and yet contemporaneously), is requisite to justify us in saying that the reciprocal succession of perceptions has its foundation in the object, and to enable us to represent coexistence as objective. But that relation of substances in which the one contains determinations the ground of which is in the other substance, is the relation of influence. And, when this influence is reciprocal, it is the relation of community or reciprocity. Consequently the coexistence of substances in space cannot be cognized in experience otherwise than under the precondition of their reciprocal action. This is therefore the condition of the possibility of things themselves as objects of experience. [emphasis added]
Granted, when Kant returns to the moon much later, he says:
That there may be inhabitants in the moon, although no one has ever observed them, must certainly be admitted; but this assertion means only, that we may in the possible progress of experience discover them at some future time. For that which stands in connection with a perception according to the laws of the progress of experience is real. They are therefore really existent, if they stand in empirical connection with my actual or real consciousness, although they are not in themselves real, that is, apart from the progress of experience.
Finally, he does exercise a bit of speculative doubt, if you will, at one point:
We may assume that this life is nothing more than a sensuous representation of pure spiritual life; that the whole world of sense is but an image, hovering before the faculty of cognition which we exercise in this sphere, and with no more objective reality than a dream; and that if we could intuit ourselves and other things as they really are, we should see ourselves in a world of spiritual natures, our connection with which did not begin at our birth and will not cease with the destruction of the body. And so on.
We cannot be said to know what has been above asserted, nor do we seriously maintain the truth of these assertions; and the notions therein indicated are not even ideas of reason, they are purely fictitious conceptions. But this hypothetical procedure is in perfect conformity with the laws of reason. Our opponent mistakes the absence of empirical conditions for a proof of the complete impossibility of all that we have asserted; and we have to show him that be has not exhausted the whole sphere of possibility and that he can as little compass that sphere by the laws of experience and nature, as we can lay a secure foundation for the operations of reason beyond the region of experience. Such hypothetical defences against the pretensions of an opponent must not be regarded as declarations of opinion. The philosopher abandons them, so soon as the opposite party renounces its dogmatical conceit. To maintain a simply negative position in relation to propositions which rest on an insecure foundation, well befits the moderation of a true philosopher; but to uphold the objections urged against an opponent as proofs of the opposite statement is a proceeding just as unwarrantable and arrogant as it is to attack the position of a philosopher who advances affirmative propositions regarding such a subject.
A deeper, or at least sideways, reason why Kant finds it difficult-bordering-on-easily-dismissable to suspect the outer world of being a hallucination, is a moral one. The maxims of reason, in which its dual practical and theoretical unity are provided for, include a call for us to reason with others, whoso if they did not exist would not be there to reason with:
Wisdom, as the idea of a practical use of reason that conforms perfectly with the law, is no doubt too much to demand of human beings. But also, not even the slightest degree of wisdom can be poured into a man by others; rather he must bring it forth from himself. The precept for reaching it contains three leading maxims: (1) Think for oneself, (2) Think into the place of the other [person] (in communication with human beings), (3) Always think consistently with oneself. (7:200; cf. 228f) [emphasis added]
In some sense, Kant seems to think that we have some sort of intuitive (in his sense) relationship with the existence of others; in The Metaphysics of Morals, he even goes so far as to deny that we have direct, objective duties to God, regardless of whether God possibly exists or we have faith in such a possibility:
The relation [of the moral law] in terms of rights of men toward a being that has only rights but no duties (God) [is philosophically vacuous because] such a being is not an object of possible experience. [Cambridge ed., 66/241-42]
In sum, or in other words, for Kant, a denial of the existence of materially subsistent other human beings would be tantamount to a denial of one of the conditions of all reasoning whatsoever, on every accessible level. And, to boot, it would be to fail to accept the existence of obligations, grounded in (or as) this faculty of reason itself, towards those other people, which would be unjust of us to do.