We sometimes pair objects with unnecessary ideas/senses. However, this is rare because we often encounter objects many times and thus refine our identification based on what stays stable across those encounters.
Let's say you don't know what a pencil is...
One day you hear someone ask for a pencil and get a long, yellow object with an eraser. Why would you associate a temperature, sound or other property with this object?
But maybe you think pencils must all be yellow and long? Well, multiple encounters should change that. For instance, later you hear a teacher tell the students to pick up their pencils, and you notice some students picking up red, thick objects. Now you've got some contradictory experience and so you ask what does this experience have in common with your previous one? Clearly all pencils don't need to be yellow and long, so what should they have?
With more encounters, should keep narrowing the stable properties until your definition of pencil is good enough to fulfill all the social and relational needs of that word.
However, there's another angle here. Senses are not the full story and may not even be that important; purpose may be more important. For instance, whenever you encounter pencils, it's usually in the role of writing and erasing, so you may think of a pencil as that which writes and erases. In fact, if you had a model of a pencil (which looks like an ideal pencil) and a very unorthodox pencil and someone asks you for a pencil, you're likely to hand them the real (unorthodox) one rather than the model, since a pencil is tied with its purpose. This means you're less likely to conflate ideas/senses since purpose may trump sense.
This process doesn't always work. Sometimes people don't get enough representative encounters to properly identify things. Think of people who suffer from red/green color-blindness. They can distinguish red/green, but they see those colors differently. Yet, the kinds of relationships that red/green enter into are such that they may never know they suffer from this.
Also, quite a few misunderstandings may be chalked up to this. When kids are growing up, they'll often use the wrong words until they get more examples. For instance, some children will end up calling all men "daddy" due to a mis-classification, or they may call dogs and cats "dogs" for similar reasons. Even adults will sometimes get into arguments or misunderstandings over this sort of thing, and from my experience it occurs a lot more with abstract objects (ones in which there isn't sensory confirmation which serves as corrective feedback). When you encounter something that's contrary to your past experience, do you update your concept, or do you think a mistake was made?
So when we talk of things, as long as our words satisfy the linguistic relationships of others, then we could be talking about different things and no one would ever know. For instance, if someone identifies "pencil" as any object that comes to a point, then this person can get corrected the moment s/he calls a knife a pencil. But if this person only ever happens to use that term in class, where the only pointed objects happen to be pencils, then how would anyone ever know that this person is talking about an entirely different thing?
The line between object, sense and language is blurred, and necessarily so. To this end, it may not hurt to read Wittgenstein's Private Language Argument.
Yes, you asked about Berkeley, and I pointed to Wittgenstein :)
