Mentally disabled adult humans, human children and non-human animals have no ability to discern "right" from "wrong"
Can they be held accountable for their actions?
One can judge, blame, and hold accountable one's computer chair for being too uncomfortable; or one can do the same to a plant for being poisonous or prickly. However, one's honesty and wisdom may be questionable here. Perhaps one bought the cheapest chair available, yet now is blaming the company for making cheap products. Perhaps prior generations of that plant were not poisonous but were eaten into extinction by some animal.
Many if not most moral systems hold some idea of a greater good. The tricky thing is that the scope of this greatness may vary in size and shape. What if, for example, the scope includes only one's tribe, so that anyone external to that tribe is excluded from consideration? What if within the tribe some members decide, for convenience, that certain other members are not of the same substance or inherent value? Perhaps honest attributions and factors of agency apply only to members of the in-group.
A purely self-serving philosophy would presumably judge as bad anything which fails to give service in the way wanted, regardless of agency. From the perspective of might makes right, the outcome for a party may determine the morality of that party; so agency is not necessarily considered. In social Darwinism, "lesser" beings are seen as meant to falter or perish, to make way for the deserved ones. If these members were meant to fail, what incentive is there to question their agency?
Ultimately, the system of attribution is what decides whether agency is relevant. And this system is generally driven by the values of the ones making the judgement. For example, those who value empathy and compassion are likely to recognise that kind people can sometimes fall victim to trickery or coercion. In this case, agency is going to matter. On the contrary, those who value cunning and trickery might recognise victims of such as unworthy and hence deserving of their fate.
Peter Singer in his book The Expanding Circle develops a capacities framework for thinking about moral concern. This kind of thinking has led to a movement for great ape personhood, a degree of rights and considerations intermediate between humans, and what we currently grant other animals. Basically moral responsibility on a spectrum, by degrees. The capacities of animals will evolve, and come to be understood better, so it's important to keep things updated.
It would seem wrong to put a chimpanzee on trial for violence, because we can't see the impacts from that we expect with humans, individual behaviour change, and maintenance of social norms. We could compare the situation to that of a child before the age of criminal responsibility, or to a mentally impaired human, where steps to manage violence and limit it's impacts would be used, instead of punishment. Pigs are one of the most intelligent animals we eat, with no signs humans will stop eating them, at least for now. Though they have been given better conditions & rights in some places like the UK. It underlines how these issues are as much about culture and convenience, as moral reasoning. Singer identifies expanding the circle of moral concern as the direction of moral progress, which seems a useful way to think about these tensions.
For inanimate objects, we look for where the autonomy is, to apply responsibility. Say a self-driving car, if it causes accidents we hold a company responsible, and their chain of command means for the most serious causes of death their CEO is directly accountable to shareholders. Boeing's Dreamliner crashes are an example of that in practice.
Lethal autonomous weapons are another edge-case. The major powers most likely to use them, refuse to accept the jurisdiction of the Hague War Crimes Court, and other international systems of justice, so in practice if they chose to use them real accountability would be impossible until they lost a war, like happened with the Nazis. Sanctions and exclusion from international bodies for violating norms could have some limited impacts, like it has with cluster bombs and land mines, and torture equipment.