tl;dr- We'll be able to construct better models for human minds by reducing general AI to human-like simplifications. However, since human intelligences are hive-minds which seek many different objectives, it seems almost certain that we won't find a single objective shared throughout a human mind.
Optimization is the field of minimizing/maximizing a value.
I know that the Artificially intelligent agent we have been putting the most hopes into recently for gaining human like capacities, uses techniques such as Reinforcement Learning where the AI is trying to make a "reward" numerical variable increase by performing actions.
Optimization is about extremizing a numerical variable called an "objective function". Artificial intelligence is a subset of optimization.
Note that some sources will talk about maximizing an objective function while others will talk about minimizing it. It's the same thing either way; add a negative sign to it, and maximization turns into minimization and vice-versa.
We'll be able to describe humans as reductions of general intelligence.
Is there a specific term refering to and/or field of research in philosophy where people are trying to compare how our brains are built with how we build our AIs in order to determine exactly whether we function identically to them, and if so, how our "reward variables" would be encoded?
Once we have super-human general AI, we could try to describe human intelligences as reductions of it. Once we find reduced variants of general AI that seem pretty human-like, then we can say that humans are like general AI operating with such simplifications.
As hive minds, humans don't have single objective functions.
That said, it seems pretty obvious that humans don't have singular objective functions; rather, parts of our brains act independently.
For example, consider your country as an intelligent agent: what's its objective function? Is it particularly meaningful to see your country as a singular intelligent agent that seeks a single thing?
The United States of America does exist as a conscious entity. It can be said to be sentient; it thinks, it has desires, it's self-aware, it has an identity, it seeks to maintain its own existence, etc.. But, it doesn't seem too reasonable to say that the USA has a single objective function that it seems to extremize.
Instead, it seems more reasonable to look at the USA as being composed of many subordinate intelligences with varying agendas. Exactly what those subordinate intelligences are depends on how far down you want to step – I mean, you can first consider individual regions of the US, or political parties, or human individuals, or human neurons, etc.. Let's pick US political parties, e.g. the Republican and Democratic parties.
Then, what does the Republican party want? And what about the Democratic party? Again, no single thing – while it's more informative to consider the parties individually, they're still not singular intelligences with singular goals.
We can then divide up the Republican and Democratic parties into, let's say, individual humans within them. Again, it's now more informative, but my point's basically that there's still not really a singular objective function.
Analogy: Companies are hive minds of employees.
Think about how a commercial organization (company) operates: many people have individual duties, right? The sales staff will keep trying to sell while the programmers will keep trying to program and the janitors will keep trying to clean.
In principle, the CEO is responsible for organizing everything such that all of these moving parts meet the organization's objectives. They delegate tasks down the chain-of-command, where subordinate executives have more focused responsibilities, down to the workers who perform tasks.
In reality, it's not so perfect. For example, the sales people might be incentivized to sell for commissions, perhaps making decisions that optimize their personal objective functions that don't precisely align with the company's own objective function.
Hive-minded thinking is appropriate for parallelization.
Obviously, it's undesirable for a company to have employees acting out-of-alignment with the company's overall objective function. So why do major companies do it anyway?
Because they have to! Because, if a CEO had to make every little decision constantly, nothing would ever get done. Instead, executives have to learn to delegate work to their subordinates. Equivalently, executives have to tweak their subordinates' objective functions to better align them with the company's overall objective function.
Ditto for the human brain.
This is, the little parts of the human brain aren't observing singular objective functions, but rather lots of 'em. Different parts of the human mind truly want different things. And this is necessary to have a reasonable level of overall intelligence, as a singular line of thought can't reasonably manage all decisions the human mind must make in a practical timeframe.
Conclusion: There's not one objective function human minds seek.
Point of all this being that human minds aren't seeking a single end. Rather, humans have a huge mix of objectives that different parts are seeking.
Of course, we can try to analyze people to determine specific objectives – just, my point's that there's not going to be an appreciably singular answer.
