I'm not sure about the Stoic attitude generally towards rhetoric and poetry, certainly Cicero had no problem with rhetoric. But the classical case against rhetoric and poetry is most famously made by Plato in a number of his dialogues, and these are certainly in the Stoic lineage.
The objection is quite relevant to our own times. Plato was concerned, first, with the dramatic oral tradition represented by Homer. Of course, he admired Homer, but the tradition had no element of "science" or truth seeking. It provided only the reckless example of the Homeric heroes by way of models. A modern equivalent might be young people who imbibe all their moral standards from Western movies or rap videos.
Similarly, he was concerned about the political influence of the many renowned teachers of rhetoric at the time, such as Gorgias. They taught ambitious young men like Menos how to argue and persuade, but this was not accompanied by any deeper considerations of justice or truth.
The result was endless argument and a polis led into disastrous actions and eventual ruin by self-seeking demagogues. Not hard for us to sympathize there, alas. The scholar Eric Havelock has described Plato as developing new models for a self-reflective culture of literacy to replace the older culture or oral tradition and forceful persuasion.
I apologize that my answer is somewhat off topic, in terms of Stoicism itself, but I thought the background might be helpful. For many classical thinkers poetry and rhetoric could be not only frivolous, but politically and personally injurious.