What's Wrong with AI-Generated Everything?

A case for welcoming imperfection in our quest for truth

Twenty years ago, Michael Sandel published an article in The Atlantic called, “The Case Against Perfection.” In it he argues that, though we will almost inevitably be able to genetically engineer future children with our preferred genetic traits—traits that code for more perfect athletic abilities, musical talents, facial symmetry, intelligence and perhaps even creativity—this would be a mistake. But the mistake lies neither in what terrible unintended consequences might come about if we were to create such perfect designer babies, nor does it lie in using these perfected people as a means for our own wish-fulfillment. Rather, the mistake is one of failing to regard things in the way they ought to be regarded. In aiming to create these perfected humans, we fail to properly appreciate children as gifts and the gifted nature of our talents.

Consider the case of athletic achievement. Sandel argues that part of what we appreciate when we appreciate athletic skill in sports just is the display of natural talents, which are, by their very nature, gifts. And while he admits we appreciate hard work and the honing of athletic talent, it is not all that we admire. If it were, then the aim to genetically enhance oneself as an athlete wouldn’t undermine the very aim of sports. Yet it does. (Similar thinking applies in the case of performance-enhancing drugs like anabolic steroids.) He writes:

“The real problem with genetically altered athletes is that they corrupt athletic competition as a human activity that honors the cultivation and display of natural talents.”

Or, consider the case of parental love:

“The ethic of giftedness, under siege in sports, persists in the practice of parenting. But here, too, bioengineering and genetic enhancement threaten to dislodge it. To appreciate children as gifts is to accept them as they come, not as objects of our design or products of our will or instruments of our ambition. Parental love is not contingent on the talents and attributes a child happens to have.”

Here, Sandel is showing us how the drive toward genetic enhancement undermines the kind of regard we ought to have or cultivate in another context: parenting. Parental love, properly understood, seeks not to perfect the object of one’s love, and it would be a contortion of that love to attempt it. This is a natural thought, as it echoes a modern platitude of romantic love, namely, that one shouldn’t strive to perfect one’s partner in accordance with one’s own image of perfection.

In short, Sandel is arguing that in some cases, aiming for perfection constitutes an ethical failure, a failure of proper regard. This failure arises in the case of genetically engineered athletes and in the case of genetically engineered children because competitive sports and parenting both make certain practice-dependent requirements on us. Parenting is the kind of activity that ethically demands acceptance of one’s child as it comes into the world, a kind of gift, imperfect as it may be relative to what is technologically possible with the advent of genetic enhancement.

Whether we agree with Sandel’s verdict in the domains of sports and parenting or not, we can see that some norms are just built in to our practices. Friendship is the kind of human practice that demands a certain kind of loyalty, whereas being an employee is not. After all, we scorn personal betrayal of a friend yet praise whistleblowing, and rightly so. Storytelling calls for certain narrative structures, whereas merely conversing does not. We take the marital promise to be binding in a way that other romantic commitments are not. And so on.

I am going to argue that the practice of inquiry—whether scientific or philosophical—is one that makes certain demands on us too, such that over-reliance on AI models to attain new knowledge undermines that very practice. Even if generative AI systems were perfectly trustworthy, secure, and reliable, a world with AI-generated everything could end up being a worse world than one in which what we believed was a little less likely to be true, less certain, less systematic.

The Promise of Generative AI

Generative AI (genAI) holds the promise of what’s come to be known as artificial general intelligence (AGI), a state of AI development in which these models have capabilities to generate outputs comparable to the outputs of human thinking, creativity, and problem-solving.¹ AGI models could help us generate the best scientific hypotheses and design ways to test them in the most efficient ways. They could design the complex machines required for the most cutting edge research. For example, they could design machines like the Hadron Collider that had to be built before the discovery of the Higgs boson.

Already, existing AI models (which are generative but not yet generally intelligent) are solving heretofore unsolved mathematical problems. The cap set problem is the problem of finding the largest set of points such that no straight line could be drawn intersecting any three of the points within the set, and a large language model solved it. It has been unsolved since it was proposed in 1972. Famously, DeepMind’s AlphaGo model solved the protein folding problem, one of the most important longstanding challenges in modern biology. That’s a pretty good track record, and the predicted exponential growth of computational power that results in these AI capabilities (and perhaps even higher growth rates) should foreclose for us the idea that these systems couldn’t outperform us on all reasoning and truth-tracking measures one day.

The gains in our intellectual discoveries, which we value both for their own sake and for the sake of progress and the well-being of others, could be unimaginably immense. It would therefore be surprising if we also stood to lose something quite significant, if this future came to pass. Is there any reason we should be wary of outsource our thinking and creative problem-solving to these models?

Roadblocks to Safe and Trustworthy AI

One reason we should be wary of AI-generated outputs (for the foreseeable future) has to do with alignment. There will be a whole upcoming series on the alignment problem here on Gavagai, but for now we’ll focus on the high-level facts. Given the way in which large language models and other cutting edge foundation models work, we cannot understand the basis on which any particular output was generated. This is partly due to the mathematical complexity of the models and partly due to the fact that we can’t tell what properties of the inputs are salient to these models. For example, although we have foundation models that can diagnose cancerous skin lesions with near perfect accuracy, we don’t know which properties of the lesions (or skin lesion images) these models are sensitive to, and so we cannot tell why a model (correctly) identifies any one lesion as cancerous. These models are ‘black box’ systems in this sense. Famously, genAI models also have emergent capabilities. This means that they have abilities to execute tasks that would not have been predictable given their training data set. (It is not settled yet in what sense these properties ‘emerge’—stay tuned for forthcoming proposals about this on Gavagai.)

The fact that these models have properties we have trouble predicting and the fact that we cannot understand their inner workings give rise to the alignment problem: how can we create AI models that will generate outputs which don’t conflict with human values and well-being? A model that merely detects skin cancer lesions seems pretty innocuous. But for an AI model that can usurp valuable resources for its own purposes, self-replicate and spread dangerous and false ideas, or control a nuclear arsenal, the problem is much more pressing. We want AI models to behave in accordance with our aims and values, yet because we cannot tell why they do what they do, it is currently impossible to be sure that they won’t go and do something very bad indeed. (Of course, we’re granting here that there is something like a relatively coherent set of human values to get the problem off the ground. If we can’t even specify such a set, then the problem becomes even worse, because we won’t even have clear standards with which to ‘align’ the models we create.)

Another related problem with current models has to do with trust. Much ink has been spilled about how biased the outputs of AI systems can be. The focus in popular news articles has been on social bias, especially on racially biased content. But the problem of bias is more general than this: insofar as we cannot grasp the basis on which a particular output was generated by, say, a large language model, we will also have a difficult time telling the ways in which the system is biased toward or away from getting at the truth. In situations where we can check how accurate the system is, the issue of trust dissipates. But when our own knowledge places limitation on us so as to make this bias undetectable, we should heed our tendency to rely on these systems. (My own take is that we rely on human reasoners whose bias is in principle just as opaque to us. That’s also for a future post.)

The problems of creating trustworthy and safe AI are deep and in my view require our best technical and philosophical skills to resolve. But suppose we could solve these problems and create future AI models that are both trustworthy and safe. How should we think about the nature of discovery if AI systems were, on the whole, significantly more reliable than human reasoners at getting at the truth? If they were safe enough to be relied on in most contexts and less biased than us, should we have any further concerns about relying on them in our inquiry practices? My own view is that there would remain at least one significant reason to be cautious in how we use these models for discovery, and that this reason is so significant that we will have to carefully consider the ways in which we ought to employ AI models in our inquiry practices.

Interestingly, there is at least one case study where the existence of superhuman AI has not usurped our engagement with that activity: chess. Can we, by explaining why we continue to have an interest in chess competitions between humans, come to understand why we might want to hold out on relying on AI models to answer our unanswered questions in mathematics, medicine, philosophy, and other domains?

Why We Won’t Stop Playing Chess

One of the most public instances of AI success occurred in 1997, when the IBM-developed chess supercomputer DeepBlue beat one of the greatest chess players of all time, then-world champion Garry Kasparov. After a series of human-computer matches over the years, some resulting in draws, some in wins for humans, and some in wins for machines, we finally reached a point at which humans could no longer outperform our best chess engines. The last year on record when a human beat a chess engine was in 2005, almost twenty years ago. Yet professional chess is alive and well, and the game continues to increase in popularity among the general public. Why do we persist, if we cannot beat the machines?

As viewers of chess matches and competitions, we seek to marvel at human talents (innate and honed) as we do in the context of sports. Compare: if we devised robots that could suddenly play baseball more skillfully than any human players, we would perhaps want to watch some of these games as a way to appreciate the engineering success, but not as a way to enjoy baseball. The point of enjoying baseball as an audience member is to see others in the human community engage in it. And so it is with chess: it is our own engagement with the game, and our mastery of it, that we seek. This is true even though we can appreciate the success of DeepBlue’s triumph over Kasparov. This appreciation, though, amounts to an appreciation of something different than when we appreciate the triumph of one grandmaster over another in a match.

I think all of this suggests that AI use in a particular activity can run counter to the spirit of that activity—the spirit of the game. We might be tempted to say that, in the context of sports and chess, our interest in human competition is what explains why AI successes don’t undermine the importance of the practice in question (e.g., baseball, chess). It’s not interesting to watch a human play a chess engine anymore because the chess engines have such an unfair advantage. Tempting as this explanation is, however, it may not provide the right diagnosis, since the same problem arises in non-competitive contexts, as I show below. AI dominance in a particular domain should concern us not because AI models have an unfair advantage over us, but because in outsourcing the task to them, we lose out on ownership of that activity. It’s our own engagement and mastery we sacrifice.

AI & Democracy

If I had to make the least controversial statement I possibly could about democracy, it would be that the spirit of that particular game, as it were, is self-governance. We can use this fact to see why over-reliance on AI models to create policies would run counter to our democratic goals.

By outsourcing policy-creation to AI systems we could gain better policies, decreased polarization, and increased trust in our in institutions. But we would be losing out on something important because we would be giving up the endeavor of weighing the benefits and tradeoffs of potential futures for ourselves and creating policy proposals from this rational and creative process. In that way, though we gain something quite valuable, we also give up on the spirit of democracy.

I am not imagining that we would give up on self-governance because we would not have the chance to vote on policies that were AI-generated. My claim is rather that in coming up with solutions to public issues we exercise a kind of ownership over the proposed solutions, such that they belong to us (the crafters of these solutions). Since elected officials represent the voting public, the products of their reasoning are in a sense also our own. To outsource the task of solving public problems through policy proposals is to give up on self-governance because it is to give up ownership over a key part of this task, even if we ultimately get to vote on these policies. Where the spirit of the game is self-governance, over reliance on AI models to problem-solve for us is a self-undermining endeavor.

Owning Our Knowledge

Where does this leave us with respect to new knowledge generate by AI models? My contention is that, although AI systems have increased our knowledge beyond what we might have achieved without them, our reliance on them in scientific, mathematical, or even philosophical discovery can run counter to the spirit of those inquiries, just as their deployment in the context of chess or policy creation undermines the aims of those practices. To increase our knowledge is not just to add to a disembodied list of truths somewhere out there. It is to strengthen our relationship to—our grasp of—the varieties of truths.

We don’t engage in inquiry so that there be knowledge about the world—that hardly sounds like English, let alone like a natural description of our goal. We seek to know the world for ourselves. If this is our goal, then we have to pay attention to the ways in which we implement AI models in our process of discovery. We may one day have models that can develop and test theories in every domain of inquiry humans have ever engaged in, and they may well outpace our own intellectual historical development. A world like this could bring many benefits—therapies for medical diseases and disorders we would take decades to develop could be created in a matter of weeks, and many of the mysteries of philosophy could finally be solved. And there is of course something awe inspiring about this picture. At the same time, I can’t help but turn away from it. A body of knowledge like this would ultimately be so disconnected from us so as not to be ours at all.

Image Credit: AlesiaKan - stock.adobe.com

This post is part of an ongoing research project on AI ethics and AI education policy. If you’d like to support this project and others like it, share it with someone you know or donate your subscription below.

1. Here, I am being careful about saying things like, “AGI systems could think and learn like humans can,” even though you see this kind of thing all over. Mostly because it’s not clear that neural networks really can think or learn, let alone in the ways that we can, given how they work. This is a topic I’ll return to in a future post.