Quotes from "Orthodoxy" 

But the cross, though it has at its heart a collision and a contradiction, can extend its four arms for ever without altering its shape. Because it has a paradox in its centre it can grow without changing. The circle returns upon itself and is bound. The cross opens its arms to the four winds; it is a signpost for free travelers.  ("Orthodoxy" by G. K. Chesterton, Chapter 2)

The Christian permits free will to remain a sacred mystery; but because of this his relations with the housemaid become of a sparkling and crystal clearness. He puts the seed of dogma in a central darkness; but it branches forth in all directions with abounding natural health. As we have taken the circle as the symbol of reason and madness, we may very well take the cross as the symbol at once of mystery and of health. Buddhism is centripetal but Christianity is centrifugal: it breaks out. For the circle is perfect and infinite in its nature; but it is fixed for ever in its size; it can never be larger or smaller.  ("Orthodoxy" by G. K. Chesterton, Chapter 2)

 Quotes from "The Everlasting Man"

Christ founded the Church with two great figures of speech; in the final words to the Apostles who received authority to found it. The first was the phrase about founding it on Peter as on a rock; the second was the symbol of the keys. About the meaning of the former there is naturally no doubt in my own case; but it does not directly affect the argument here save in two more secondary aspects. It is yet another example of a thing that could only fully expand and explain itself afterwards, and even long afterwards. And it is yet another example of something the very reverse of simple and self-evident even in the language, in so far as it described a man as a rock when he had much more the appearance of a reed. ("The Everlasting Man" by G. K. Chesterton-Part 2, Chapter 4)

But the other image of the keys has an exactitude that has hardly been exactly noticed. The keys have been conspicuous enough in the art and heraldry of Christendom; but not everyone has noted the peculiar aptness of the allegory. We have now reached the point in history where something must be said of the first appearance and activities of the Church in the Roman Empire; and for that brief description nothing could be more perfect than that ancient metaphor. The Early Christian was very precisely a person carrying about a key, or what he said was a key. The whole Christian movement consisted in claiming to possess that key. It was not merely a vague forward movement, which might be better represented by a battering-ram. It was not something that swept along with it similar or dissimilar things, as does a modern social movement. As we shall see in a moment, it rather definitely refused to do so. It definitely asserted that there was a key and that it possessed that key and that no other key was like it; in that sense it was as narrow as you please. Only it happened to be the key that could unlock the prison of the whole world; and let in the white daylight of liberty. ("The Everlasting Man" by G. K. Chesterton-Part 2, Chapter 4)

The creed was like a key in three respects; which can be most conveniently summed up under this symbol. First, a key is above all things a thing with a shape. It is a thing that depends entirely upon keeping its shape. The Christian creed is above all things the philosophy of shapes and the enemy of shapelessness. That is where it differs from all that formless infinity, Manichean or Buddhist, which makes a sort of pool of night in the dark heart of Asia; the ideal of uncreating all the creatures. That is where it differs also from the analogous vagueness of mere evolutionism, the idea of creatures constantly losing their shape. A man told that his solitary latchkey had been melted down with a million others into a Buddhistic unity would be annoyed. But a man told that his key was gradually growing and sprouting in his pocket, and branching into new wards or complications, would not be more gratified. ("The Everlasting Man" by G. K. Chesterton-Part 2, Chapter 4)

Second, the shape of a key is in itself a rather fantastic shape. A savage who did not know it was a key would have the greatest difficulty in guessing what it could possibly be. And it is fantastic because it is in a sense arbitrary. A key is not a matter of abstractions; in that sense a key is not a matter of argument. It either fits the lock or it does not. It is useless for men to stand disputing over it, considered by itself; or reconstructing it on pure principles of geometry or decorative art. It is senseless for a man to say he would like a simple key; it would be far more sensible to do his best with a crowbar. And thirdly, as the key is necessarily a thing with a pattern, so this was one having in some ways a rather elaborate pattern. When people complain of the religion being so early complicated with theology and things of the kind, they forget that the world had not only got into a hole, but had got into a whole maze of holes and corners. The problem itself was a complicated problem; it did not in the ordinary sense merely involve anything so simple as sin. It was also full of secrets, of unexplored and unfathomable fallacies, of unconscious mental diseases, of dangers in all directions. If the faith had faced the world only with the platitudes about peace and simplicity some moralists would confine it to, it would not have had the faintest effect on that luxurious and labyrinthine lunatic asylum. What it did do we must now roughly describe; it is enough to say here that there was undoubtedly much about the key that seemed complex, indeed there was only one thing about it that was simple. It opened the door.  ("The Everlasting Man" by G. K. Chesterton-Part 2, Chapter 4)