minds are one kind of thing, among many other kinds of things, located in the environment external to one's own mind?
There is a basic flaw here that assumes 'everything is a thing'. When you try to wedge the mind (or the set of integers, for that matter) into that model, lots of the model of what 'a thing is' falls apart.
To most folks shy of "We are all God" thinkers like Berkeley or Leibniz it is obvious other minds are outside my own mind. If the environment is just 'everything', your position becomes trivially true as a deduction from a non-sequitur -- physicality has nothing to do with it. At the same time, if each mind is a 'thing', as others here have already pointed out, your own mind is hardly external to your own mind, so if this is the notion of minds in general, your own mind is not a mind.
Minds are not that easily separated, anyway. Not only are there many competing thought processes going on at once in your own composite experience, but thought processes are not, by nature, individual.
If the environment isn't just 'everything' then things need some way that they are affixed to it. Usually this is places, but that clearly does not fit our notion of minds. There is clearly 'mind-stuff' that is just kind of nowhere and everywhere, in that it spans minds and does not belong to any individual mind: things that only exist in a composite of minds. Take for instance the English language. Its side-effects, like various sound patterns in the environment, or inkblots on a page are clearly outside the mind.
But the English language itself cannot be said to exist out side the collection of minds. So even if all minds are physical, there are still things that do not exist outside of minds. Likewise 'hope' or 'pi' are things that exist only within minds. They describe patterns of behavior and observations of physical reality, but they cannot be said to exist 'in' or 'as' those things.
So the notion of mind, as a thing has the TARDIS problem, it is bigger on the inside. I can conceive of English, which spans a whole range of minds, including my own. It is inside me, and I am one of the things inside it. Referentiality does not work for 'things in the environment'.
As process-oriented thinking like Daniel Dennett's both points out and pretends not to notice, if the mind is physical, it does not really exist, or at least it is not the sort of thing we think it is.
From this point of view, the mind is a useful concept, but at root it is metaphor for something else too complex for us to discuss easily. And as a metaphor, it immediately becomes one of those things like English or hope, that do not have an independent existence outside a mind.
Then, 'mind' is a metaphor I use to make sense of other people, and ultimately myself. But those metaphors cannot be entirely shared. My exact theories and assumptions about your mind don't exist outside of mine. It is dubious whether they exist inside of mine, to the extent they are not entirely clear.
Flipping the perspective over, all of our minds, as you see them, are not outside your mind. They are models of something that is clearly out here with us, and not in there with you, but the models aren't out here, they are just in there. You can communicate part of them, and you can act on the logic they are made of, but you cannot put them out here in their complete reality.
Even if you decide everything is physical, names and metaphors have to be a different kind of thing, just patterns, physical only in their effects. Then discussing them becomes like trying to hold water in a net. The effects are all there are, and together they specify the thing they make up, but many things have the same effects and many effects have the same cause. The grouping is too fluid to be a thing.
So rejecting idealism is practical in theory, but you are endlessly drawn back to it. You can declare references to be some kind of thing, but that thing has only some kind of statistical existence, and the place where statistics are, is very hard to imagine as having a physics.
