The text you quoted from the Notebooks most probably does refer to the tension, in the Tractatus Logico-Philosophicus, BETWEEN the aim of building an ideal perfect language, in accordance with an austere extensional scheme governed by the necessary laws of logic, AND the evidence that many linguistic expressions which have indubitably sense don't comply with that scheme.
This tension was very probably one of the reason why he changed his philosophy in his intermediate period, which culminated in the Philosophical Investigations, and stopped thinking in terms of syntax and semantics, focused on MEANING (1): he started thinking in terms of pragmatics, focused on USE of language (2).
(1) First Wittgenstein: Syntax and Pragmatics look for the MEANING of linguistic sentences.
What is the relation if there any between a linguistic entity and
things in the world?
the relation between a linguistic entity and things in the world is the theory of a proposition as a picture (the picture theory, considered the main theory in the Tractatus Logico-Philoosophicus): propositions, for being projections in words of situations which can be thought of (/logically depicted), share with the thoughts the status of being picture of the reality, in other terms they represent the world.
More specifically, each sentence, in order to have a sense, must represent a fact, which is a given disposition of object in the world. It can do that because it shares with reality the pictorial form, therefore each one of the constituents of the meaningful declarative sentence must be proxy for each one of the component of the depicted situation. It is equivalent to say that each name which occurs in the sentence must have a reference (bedeutung).
Just to be clear, if I'm looking at a dog and a chair in this moment and I say "The dog is under the chair", it is a meaningful declarative sentence, since both "dog" and "chair" have a reference, which are the existent objects I'm looking at.
It can be either true or false (since the fact that the dog is under the chair it's contingent, not necessary), but the point is that I AM ABLE to establish if it's either true or false (i.e. its truth-value) by virtue of this correlation between the disposition of the names in the proposition and the disposition of objects in the fact.
On the other hand, for instance, the quote from Nietzsche "If you look long enough to the abyss, the abyss will look back to you" is senseless in the tractarian perspective, because the sense which I grasp doesn't depend on a correlation between names and objects, so it is nonsensical (unsinnig sätze).
The backround of this assumptions is the solder and austere scheme governed by the necessary / tautological laws of logic (and mathematics, which is a logical method). This syntactical aspect is closely connected with the SEMANTICS because what corresponds to the general form is MEANINGFUL (and also Wittgenstein was basically concerned with propositional logic since it is decidable and therefore it complies with the requirements of a pure truth-functionality, since the understanding of the meaning is based on the knowing the truth-conditions).
This has implications on the theory of knowledge as well: in the tractarian viewpoint I don't understand wether a sentence is true and THEN decide what it means, BUT I FIRSTLY understand its MEANING and then decide if it's true or not by virtue of mechanical decisions (truth-tables)
This importance assigned to MEANING in accordance with logical crieteria belongs to the part of his philosophy which is contextualized in the so called "Early/first Wittgenstein" phase.
But, as you sayed,
later he abandoned this approach
in fact ->
(2) Second Wittgenstein: Pragmatics look for USES of language
The second phase of "later/second Wittgenstein" is mainly represented by the Philosophical investigations (1953), in which the working out of all the consequences of a total rejection of dogmatism culminated. The second Wittgenstein criticized his own Tractatus itself for being dogmatic, in the sense that it pretended to reduce all the linguistic expression to an abstract general form (([p-, ξ-, N(ξ-)]), according to which they are all equal.
But let's considere again the quote from Nietzsche:
"If you look long enough to the abyss, the abyss will look back to you"
The tension was: How to justify the fact that such kind of sentences hold a sense, although they don't have a meaning in logical terms? Or on the other hands how to deny that they have a sense, although it is undeniable that they have it?
The answer provided by the second Wittgenstein can be summed up somewhat like the following: "dont' look for the MEANING, look for the USE".
So such a sentence has a sense depending on its uses in a specific context (/on the rules of a "language-game") instead than on a logical decisional procedure: the linguistic context in this case is that of a group of philosophers who are talking not about Wittgenstein but about Nietzsche, and then we have an interpretation of it. I'm no longer worried about the references (bedeutung) of its names, since the rule of calculus-correctness doesn't hold in this other linguistic context.
To answer your last question synthetically,
If the function of a linguistic entity is not to refer to something in the world then what is it's function?
The relation between a linguistic entity and the world is fundamental but with regards to technical linguistic problems only, if we analyze language in the perspective of calculus-correctness. In the wider perspective of Philosophical Investigartions there are other "perspectives" (infinite language-games) we are allowed to refer to in order to be able to understand the sense of a sentence, in a sense.