"Genius is the talent that gives the rule to art", says Kant, what does "giving rule to art" means?
In the quote below, from Kant's Critique of Judgement, Kant is saying beautiful art isn't produced by standard production rules, concepts, design. He is saying the determining ground for beautiful art is sublime: the rule comes from inspiration, from out of nowhere, 'genius'. I believe this is analogous with Kant's determination of things in transcendental idealism; the determining ground of a discerned thing isn't another discerned thing, it is a foundational intuition [Anschauung], or the unconscious, or apparently nothing, a noumenon.
§46. Beautiful Art is the art of genius.
Genius is the talent (or natural gift) which gives the rule to Art. Since talent, as the innate productive faculty of the artist, belongs itself to Nature, we may express the matter thus: Genius is the innate mental disposition (ingenium) through which Nature gives the rule to Art.
Whatever may be thought of this definition, whether it is merely arbitrary or whether it is adequate to the concept that we are accustomed to combine with the word genius (which is to be examined in the following paragraphs), we can prove already beforehand that according to the signification of the word here adopted, beautiful arts must necessarily be considered as arts of genius.
For every art presupposes rules by means of which in the first instance a product, if it is to be called artistic, is represented as possible. But the concept of beautiful art does not permit the judgement upon the beauty of the product to be derived from any rule, which has a concept as its determining ground, and therefore has at its basis a concept of the way in which the product is possible. Therefore beautiful art cannot itself devise the rule according to which it can bring about its product. But since at the same time a product can never be called Art without some precedent rule, Nature in the subject must (by the harmony of its faculties) give the rule to Art; i.e. beautiful Art is only possible as a product of Genius.
To recap, Kant is saying ordinary art comes from conceptualised rules while sublime art comes from pre-conceptual rules—from the unconscious foundation as it were—via what he calls 'genius', inspired by Nature.
beautiful art does not permit the judgement ... to be derived from any rule, which has a concept as its determining ground
The wider analogy is that in phenomenology things exist by conceptualisation and cognition, i.e. Critique of Pure Reason A598/B626:
[the] Being [of things] ... is merely the copula of a judgement.
The foundation of things cannot be a thing or there would be an infinite regress of things. The determining ground, the foundation, must be pre-conceptual, e.g. unconscious. Or as Heidegger formulates it in Being & Time, H.38
[foundational] Being, as the basic theme of philosophy, is no class or genus of entities; yet it pertains to every entity. Its 'universality' is to be sought higher up. Being and the structure of Being lie beyond every entity [thing] and every possible character which an entity may possess.
Since beautiful art is inspired by Nature it comes directly from the nouminal determining ground, not procedural or by rules of design. Likewise the judgements that reify all discerned things come from the nouminal determining ground.