30 Years After "The Cathedral and the Bazaar": Thinking About the Era of AI Hegemony Through the "Law of Spiral Development"

The Cathedral, the Bazaar, and the Philosophy of Palantir’s CEO

Recently, during a 1on1 with a colleague, we had a discussion about the intense philosophy of Alex Karp, CEO of Palantir—the giant player in data integration and defense AI.

Karp boldly asserts that "AI is a geopolitical weapon equivalent to nuclear weapons to defend Western democracy." His worldview, along with his stance on overwhelmingly integrating and controlling national-level data centrally, exhibits the sheer intensity of what could be called "ultimate centralization." Engaging with this concept suddenly brought back to mind a book I read years ago.

During my university days, when I was completely captivated by hacker culture and open-source philosophy—devouring technology books—I was deeply struck by Eric S. Raymond's classic essay, The Cathedral and the Bazaar (1997).

In the "Cathedral" model, a select group of elite developers designs a perfect blueprint top-down, building a robust product over years under strict management. In IT, the ultimate symbol of the "traditional Cathedral" is the mainframe development typified by IBM in its prime, or waterfall-style system integration by massive vendors. There, based on rigid specification documents, source code was thoroughly hidden as proprietary corporate assets, and only a privileged few engineers held the rights to develop and maintain it. An absolute reliance on vendors (vendor lock-in), coupled with a closed, orderly "perfection through control," was the supreme righteousness that supported the early computer industry.

In contrast, the "Bazaar" model is driven by countless developers collaborating flatly across the internet. They frequently release unfinished code, operating under Linus's Law: "Given enough eyeballs, all bugs are shallow." In this model, software evolves amidst the passion and chaos of a community. Represented by Linux and Apache, this approach is fueled by the free, emergent creativity of developers.

Thus, the essence of the "traditional Cathedral" versus the "traditional Bazaar" in IT history lies in the sharp contrast between "perfection through control" and "emergence through openness." As Linux came to dominate the server market and everyone became free to modify and share code, we long believed without a doubt that "in the world of technology, the Bazaar eventually wins over the Cathedral." This narrative spanned from Web2.0 to the cloud ecosystem, and eventually to the decentralized worldview championed by blockchain.

(As a slight tangent, both Microsoft and Apple are undoubtedly massive "Cathedrals" in terms of their control and platform closedness. Yet, for some reason, there are moments when Apple still evokes a sense of Bazaar-like excitement and free creativity. This illusion might stem from their cleverness in creating a secure "walled garden" while simultaneously staging the "bustle of a marketplace" where countless creators run wild.)

Returning to the main point, what my colleague's conversation forced me to reconsider is where our long-held narrative of the "Bazaar's victory" stands today. Witnessing the explosive evolution of generative AI, is the world now undergoing a fierce swing back toward a "massive Cathedral"?

From "Liberal Platforms" to a "Cathedral Holding Existential Sway"

The structure of "a few platformers building the foundation (the Cathedral) and third parties dancing on top of it (the Bazaar)" is not new in itself. It is the very platform economy established by the likes of GAFA (Apple's App Store, Google, AWS, etc.) and Shopify.

However, there is a decisive qualitative difference between the worldview of that era and today's AI hegemony.

During the GAFA era, the themes were mainly "information search," "social media and advertising," and "e-commerce enablement." There existed "well-behaved reins" such as liberal social norms, privacy regulations, and platform content moderation. It was a relatively gentle worldview where selfish abuse was restricted by terms of service.

But as symbolized by Alex Karp's remarks mentioned earlier, the current struggle for AI hegemony presents themes as monumental as the invention of nuclear weapons: the existential survival of humanity, the monopoly of intelligence itself, and geopolitical shifts. Most peculiarly, supranational decisions that could dictate the future of humanity—such as "safety guardrails" or the "release of AGI (Artificial General Intelligence)"—lie in the hands of a tiny group of private corporations (and their executives), rather than democratically elected leaders. This is precisely where the overwhelming "eeriness" of modern AI lies, representing an inescapable, massive Cathedral.

The "Law of Spiral Development" in E-Commerce and Retail

When contemplating this seemingly discouraging return to a "massive Cathedral," a certain philosophical concept comes to mind: the "Law of Spiral Development."

Derived from Hegelian "aufheben" (sublation) and frequently cited in business contexts, this law accurately describes the evolution of history and systems. Things do not simply swing back and forth between state A and state B. Even if a system appears to revert to an old state, it actually evolves and re-emerges at a higher level, much like climbing a spiral staircase.

We can see this clearly by looking at the familiar world of e-commerce and retail.

Historically, shopping was an analog, face-to-face service where you consulted a skilled store clerk, explaining your specific needs. With the spread of the internet, this was replaced by e-commerce websites. We began typing keywords into search bars, comparing specifications, and choosing products ourselves—a shift to digital self-service.

Today, however, the advent of AI agents is reviving the experience of "grasping context from vague requests and proposing the most suitable product through conversation," right within the digital space of e-commerce.

Of course, we may not expect the warmth of human-to-human connection in this interaction (or perhaps it is intentionally bypassed). However, focusing strictly on the function of customer service—understanding customer intent and aiding decision-making—this is a clear return to "personalized customer service," but on a distinctly higher dimension powered by global data and analytical capabilities.

Deepening Roles: "Intelligence" as Infrastructure and the Cathedral Absorbing the Bazaar

When we unpack this structure, the true nature of the unease felt at the beginning—the sense that the world is regressing into an eerie Cathedral—becomes clear.

The core structure of "platform and third parties" established by GAFAM has not changed. What has changed is that the infrastructure layer provided by the Cathedral has evolved from "servers, payments, and features" to "general intelligence itself," expanding to a national scale.

"Foundation intelligence and security infrastructure," which requires massive capital and energy on par with national budgets, must inevitably be handled by the giant Cathedrals of private enterprises. On top of that layer, powered by this unlocked, overwhelming intelligence, an "ultra-Bazaar" is expanding. Here, merchants and developers worldwide are building custom-context apps, AI agents, and novel commerce experiences at a breakneck pace.

Even more fascinating is that the Cathedral and the Bazaar do not exist in a simple top-down relationship; they function as a cyclical ecosystem feeding into each other.

Looking at Shopify's technical ecosystem provides a clear example. Within the Bazaar of the open-source community, developers create new web development tools and frameworks like Remix (React Router) or Tailwind CSS, iterating through trial and error. Eventually, the Cathedral (Shopify) absorbs and integrates these outstanding projects and technologies that have proven their market value—often through acquisition or sponsorship—incorporating them into the standard foundation of its own platform.

The Bazaar is an experimental ground that generates innovation with high uncertainty but at overwhelming speed. The Cathedral is a safe zone that acquires, standardizes, and guarantees the continuity and quality of those outcomes.

"Spiral Development" as a Thinking Framework

What started as a casual chat during a 1on1 led to reflections on Alex Karp's modern intensity and the 30-year-old concepts of The Cathedral and the Bazaar. Perhaps the greatest insight to be gained here is the perspective of deciphering patterns of change from history.

When faced with rapid technological changes and shifts in social structure, we tend to either panic, thinking "something completely unprecedented has occurred," or cynically dismiss it, believing "we have simply reverted to old structures."

However, holding onto the framework of "spiral development" drastically changes how we perceive these shifts. We can look back at past history and philosophies (such as The Cathedral and the Bazaar) and ask: "At what layer, and at what higher dimension, is that old structure re-emerging today?"

History does not simply repeat. However, it certainly cycles to a similar rhythm on a higher plane. In this turbulent era of AI, to cultivate an independent design philosophy rather than just drifting with the tides, deciphering the "spiral trajectory" from past wisdom might just be a surprisingly powerful weapon.