Innovation presents several distinct problems: forming an idea, finding a worthwhile problem, reaching the right users, and turning use into the intended benefits. Six books observe different parts of that process. Johnson examines creative environments; Ridley emphasizes gradual combination and markets; Drucker develops systematic search and entrepreneurial management; Christensen and Raynor explain disruption and organizational capability; Rogers examines adoption and diffusion; and Perez places technology within long-run financial and institutional change. Their accounts illuminate one another and also disagree about market incentives, organization, and direction. The frameworks below follow the decisions a reader needs to make. Shortform’s additions are attributed separately, and source errors are corrected where they affect the argument.
1. How ideas form: platforms, connections, and experimentation#
Whether an idea can be realized depends on the knowledge, tools, and people it can connect. Begin with formation, then ask how to make room for exploration.
Platforms let successors start somewhere#
Matt Ridley uses recombination to explain innovation in How Innovation Works: atoms, molecules, and data are organized into new functional systems. He places CRISPR, the rise of dairy farming, and the evolution of lactose tolerance alongside one another to illustrate the emergence of new patterns and the role of energy use in sustaining order. This is an analogy between technological and biological change, not a claim that innovation overturns entropy.
Steven Johnson calls the knowledge and technologies available to later creators platforms. A platform may be physical, such as a steam engine or computer, or conceptual, such as a language, theory, or body of writing. It saves successors from reinventing a writing system or the internet, while providing parameters they can follow, bend, or break. Artistic genres supply such parameters: a writer uses a language, pen and paper or a computer, and earlier works, then may combine conventions from children’s literature, romance, and coming-of-age stories to imagine a form such as young adult literature. This is an illustration of recombination, not an exclusive historical account of a genre’s origin.
Ridley’s historical survey treats technological change as a major force in human development and describes most innovations as products of sustained work, trial and error, and incremental improvement. The Newcomen steam engine built on earlier technology; aviation pioneers likewise refined designs by learning from successes and failures. These cases illustrate a common way of working without excluding sudden insight or breakthroughs that change a paradigm.
Shortform adds: Innovation builds on prior knowledge platforms in several ways. Studying how an idea has developed over time deepens understanding of its underlying concepts and can reveal further innovations even after the idea seems complete. This approach is emphasized by Edward Burger and Michael Starbird in The 5 Elements of Effective Thinking.
Johnson writes for outlets including The New York Times and The Wall Street Journal and is the author of Everything Bad is Good for You, Future Perfect, and Enemy of All Mankind. Where Good Ideas Come From examines the environments and connections ideas require.
Sources for this section: how innovation works, where good ideas come from, diffusion of innovations
The adjacent possible, independent discovery, and play#
Platforms of accumulated knowledge define the boundaries of the adjacent possible — the realm of new ideas that are reachable given a society's current knowledge, resources, and capabilities. While good ideas can originate outside the adjacent possible, all successfully implemented ideas fall within it: ideas developed inside the adjacent possible are understandable and achievable at the time they are created, whereas ideas conceived beyond it — however strong they may be — cannot be realized in their own era due to insufficient knowledge or resources, though they may be successfully developed later once the necessary conditions emerge.
Johnson illustrates the constraint with Charles Babbage’s Analytical Engine, conceived in the 1830s. It was to use mechanical parts to execute programmed operations, remained unfinished in Babbage’s lifetime, and would have faced mechanical limits on speed. A conception can precede the knowledge, resources, and engineering capacity needed to realize it. The source notes overstate the connection to modern computing: storage, operations, and program control provide functional analogies, not grounds for saying modern electronic computers used “the same blueprints.” The Science Museum’s Babbage account supports the narrower distinction.
The adjacent possible also helps explain “the multiple”: independent researchers working with a shared stock of knowledge and tools may reach the same answer at roughly the same time. Johnson’s example is an electrical storage device independently developed in the mid-eighteenth century. The notes mistakenly call it a battery; Shortform’s own correction identifies it as the Leyden jar.
Stuart Kauffman introduced the adjacent possible in evolutionary biology, and Johnson applied it to innovation. Shortform describes each advance opening further combinations as “exponential” expansion. The useful explanation is that combinations generate further combinations; the material does not establish a universal measured exponential growth rate for technology.
Evolution also supplies a distinct version of the “multiple” described above. Different mutations can produce the same beneficial trait, meaning that organisms can reach the same feature through different paths. Scientists suspect that traits with several possible genetic origins are more likely to recur in different organisms and spread through their descendants than traits that depend on one specific, complex mutation.
Shortform adds: Multiple discovery challenges the heroic theory that assigns invention wholly to isolated genius. Shared platforms help explain why breakthroughs cluster at particular historical moments, while individual work still matters in turning a possibility into a result. The historical cases support that explanation without measuring the respective contribution of either factor.
The Leyden jar was independently developed in work associated with E. Georg von Kleist and, in Leiden, Pieter van Musschenbroek; the summary’s “Cuneus” should not be treated as another name for Musschenbroek. A jar stores static electricity; it is not Volta’s later electrochemical battery. Shortform also connects the jar’s subsequent uses with Franklin’s kite experiment and Volta to illustrate how later electrical work drew on earlier technology. The case concerns independent discovery and cumulative use, not an identity between distinct devices.
Shortform adds: In a TED Talk, Steven Johnson adds further context to Charles Babbage’s invention and the platforms that led up to it, arguing that invention often stems not from necessity but from a spirit of playfulness. He traces the origins of the programmable computer back thousands of years to the invention of the flute: a device that produced sound by blowing air through a tube. That invention led to the organ, which used a keyboard to generate sound; centuries later, the organ keyboard was adapted into the typewriter keyboard. The organ also gave rise to the music box, which was programmable in the same way Babbage’s computer was—you could swap out the box’s cylinder, which carried the musical “code,” to make it play a new song.
Sources for this section: where good ideas come from
Combining fields and repurposing functions#
Innovation frequently cross-pollinates across disciplines. Insights and techniques from one field often inspire breakthroughs in another, and the practice of combining and reworking existing concepts, technologies, and methods to create something entirely new is central to a wide range of creative pursuits. The application of agricultural genetics principles to medical technologies illustrates how major breakthroughs often emerge from the convergence of distinct fields; scientific advances across domains — from using chemicals to stop fungal spread in agriculture to deploying gene-editing techniques to alter DNA — reveal the deep interconnection between scientific discovery and technological progress. The development of the computer, a landmark achievement made possible by the collaborative work of specialists across many fields, underscores the value of cross-field idea exchange. Similarly, Bosch and his team drew inspiration from fields unrelated to nitrogen fixation to devise cost-effective methods for the mass production of ammonia, demonstrating the benefits of interdisciplinary collaboration.
Johnson frames this kind of cross-field work as the combination or repositioning of knowledge platforms, which opens up new avenues of discovery. He offers the resolution of Darwin's Paradox as a key example: Charles Darwin had observed that vast biological systems like coral reefs could thrive in ecosystems with very little nutrition and resources, a puzzle that long defied explanation. The question was finally solved when scientists from multiple fields — including biology, chemistry, and systems science — combined their expertise to discover that a range of distinct biological platforms work together to recycle resources and nutrients, sustaining the entire system. This cross-disciplinary synthesis gave rise to an entirely new field: ecosystems ecology.
A closely related driver of innovation is exaptation, a term borrowed from evolutionary biology describing when a tool, trait, or structure originally developed for one purpose is repurposed in a different context to serve a new function. Feathers, for example, initially evolved for insulation but later proved perfectly suited to enable gliding and flight when animals took to the air.
Dense, connected networks such as cities are natural hotbeds of exaptation, because they bring together people with diverse ideas and cultural backgrounds. The collision of different perspectives generates greater innovation and more creative ideas. The coffeehouse is a classic example of a physical space undergoing exaptation: though originally founded as commercial venues selling food and drink, coffeehouses were quickly repurposed into public meeting places where people could exchange ideas on a larger, more diverse scale than was possible in private homes or workplaces.
Exaptation is also a powerful force in media and art. Repurposing a narrative device across mediums can change its function and produce genuine artistic innovation — for instance, adapting the stream-of-consciousness technique from prose fiction into film creates a new mode of cinematic storytelling.
When different knowledge platforms are combined, the result is a form of transdisciplinary thinking. This sits on a spectrum: multidisciplinarity approaches a topic from several separate perspectives; interdisciplinarity treats those perspectives as interrelated; and transdisciplinarity dissolves boundaries between disciplines entirely to form a cohesive new whole, such as a new scientific field or media genre.
Shortform extends the exaptation lens from technology to human culture. Some experts argue that art itself emerged by repurposing communication: neural networks that evolved to support humans’ innate use of symbols were recruited for creating and appreciating art.
The scientific status of exaptation as a distinct evolutionary concept is contested. Some researchers question whether exaptation is meaningfully different from adaptation, arguing that if you trace any evolutionary trait far enough back, you will find it arose by coopting an existing structure for a function different from its original one—making the line between adaptation and exaptation arbitrary and irrelevant. Others note that the term is rarely used in mainstream scientific literature, possibly because its definition remains unclear. Despite this, Shortform points out, exaptation has gained traction in fields such as the history of technology and innovation studies, where it resonates with the human tendency to consciously apply one idea across very different contexts.
Sources for this section: how innovation works, where good ideas come from
Slow hunches need time and a way to survive#
Most ideas take a long time to develop and require a great deal of patience. They begin as suspicions, hunches, or incomplete notions, and often need to be combined with other people's unfinished ideas to take shape — which is part of why networks are so important. Such ideas can brew for years or decades before they encounter the complementary ideas that complete them and push them into the realm of the adjacent possible.
Johnson distinguishes these unfinished ideas from gut instincts. While both can act as underlying seeds that lead to something greater, gut instincts are immediate and fast. Unfinished ideas or hunches start with an inexpressible suspicion that there is something deeper at play in a situation or problem you are considering; unlike gut instincts, they do not lend themselves to quick decisions. Instead, they linger in the back of your mind for long periods, slowly connecting to new things you learn and eventually coalescing into a whole idea greater than the sum of its parts. Because these long-term partial ideas can be nebulous, ill-defined, and easy to forget, Johnson recommends that you record every one you have.
Shortform identifies a risk for slow hunches: the creative cliff illusion, in which a slowdown in idea production is mistaken for exhaustion of worthwhile ideas, prompting premature abandonment. Its research summary reports that many good ideas can emerge late in an ideation period. The implication is to avoid equating slower production with exhausted quality, not to assume every task must produce its best ideas at the end.
The nature of early-stage hunches is also debated. Some experts draw only a weak distinction between hunches and gut instincts, arguing that the “inexpressible suspicion” that sparks a half-formed idea may come from intuition, which plays a vital role in the judgment and decision-making needed to develop and refine unfinished ideas. Others caution against overreliance on gut instinct, however, because it can reinforce hidden biases and narrow thinking.
Practically, recording every idea immediately is not always feasible, since many insights strike during routine activities like showering. To retain them, you can improve your memory through exercise, sleep, and mnemonic devices. If you have already forgotten an idea, you can try retracing the original thought process or even physically revisiting the environment where it occurred to jog your memory.
Sources for this section: where good ideas come from
Make connections without treating insight as proof#
Sudden insights are yet another pathway. To cultivate them, Johnson recommends practices that help shift the mind into a more associative, receptive state. Activities such as taking a walk or a bath draw attention away from daily tasks and allow the unconscious mind to make important connections. Another approach is to absorb as much new information as possible through reading or other means. Johnson notes a flaw in casual, daily information intake: it is easy to forget what has been learned when only a small amount of information is processed each day. He points to figures like Bill Gates, who takes annual reading vacations of a week or more to deep-dive into accumulated books, absorbing and processing new information in a concentrated period. Johnson suggests that workplaces could benefit from allotting time for such immersive learning, as it can be highly productive for generating new ideas.
Shortform adds: The experience of sudden insights during low-focus, routine activities like bathing or walking is known as the shower principle. The feeling of an idea clicking into place through epiphany can be deeply satisfying and often leaves people grateful for the insight. Experts recommend practicing introspection and staying open to personal change as ways to invite more epiphanies.
At the same time, there are important caveats. The excitement of a sudden insight can cause people to overlook practical limitations and fail to notice that an idea may be unfeasible or unachievable, so ideas generated in these moments should be evaluated carefully rather than accepted uncritically.
Shortform recommends extended, uninterrupted reading to support these connections. It favors physical books because online hyperlinks repeatedly invite decisions about switching pages, while email and other tasks interrupt processing. The transferable practice is to reduce those interruptions and allow time for absorption and retention; the material does not establish that paper itself always outperforms undistracted digital reading.
Johnson also offers a neural explanation. The summary describes alternation between synchronized “phase-locking,” characterized as networks firing at the same frequency, and less synchronized activity thought to permit unusual associations. It cites a study of children in which time spent in the two states varied and longer “chaotic” periods correlated with higher IQ. This is an observation cited in the book, not evidence that deliberately increasing desynchronization raises intelligence or creativity; nor does it justify treating the whole brain as a simple two-mode switch.
Shortform’s supplementary material discusses possible roles for phase-locking in communication between brain regions, focus, and learning, and describes exploratory work on music intended to induce it, including speculation about future ADHD applications. Those materials do not establish music as a substitute for medication. It also connects dreaming and daydreaming with the default network, active during rest or passive activity: the cited literature discusses associations, network boundaries, and activation conditions, not a settled finding that daydreaming causes creativity. The story that August Kekulé discovered the benzene ring in a daydream is also disputed; some historians point to the work of two other researchers as a possible source.
Sources for this section: diffusion of innovations, how innovation works, where good ideas come from
Tolerating error, thought experiments, and genuine dissent#
Mistakes are another powerful driver of better ideas and greater innovation. Many landmark inventions in history emerged from accidents, including penicillin, the daguerreotype, and the pacemaker. Making mistakes and then investigating why they happened makes people smarter: errors open up new ways of thinking and push people to explore additional possibilities within the adjacent possible. Organizations that try to eliminate all error cut themselves off from the larger pool of ideas in the adjacent possible. The goal should not be to seek out mistakes, but to embrace and allow them, because only then can people truly understand and learn from them.
Johnson draws on Charlan Nemeth’s research to examine how error can interrupt a group’s habitual associations. In a colored-slide task, participants usually offered familiar associations such as green and grass; when an actor in the group called a blue slide green, others produced more varied associations. The source also mentions similar findings in simulated juries. These are findings about how anomalous views can alter thinking in particular tasks, not a recommendation to distribute inaccurate information at work or proof that varied associations produce successful innovation.
Johnson also draws an evolutionary analogy: DNA changes produce variation; most mutations are neutral or harmful, while some beneficial variants may spread through selection and open new possibilities. Inheritance and selection must be distinguished: a mutation does not have to be beneficial to be inherited, and mutations do not occur for the purpose of improving a species. Johnson invokes the billions of years between early single-celled life and later biological diversity to illustrate the immense timescale over which variation, inheritance, and selection accumulate; this is not a process driven by one beneficial mutation or directed toward a predetermined goal.
Shortform adds: How much people learn from mistakes depends heavily on mindset. Those who believe they can learn from errors gain more from them than people who view intelligence as fixed and see mistakes as personal failures. This means learning from error is less about the act of making a mistake itself than about being willing to examine what went wrong.
Shortform cites Edward Burger and Michael Starbird’s The 5 Elements of Effective Thinking for an exercise more deliberate than Johnson’s tolerance of error: make a logical leap known to be incorrect, follow it through, and look for new possibilities. They advocate rewarding exploration and learning from error instead of punishing people for mistakes. This differs from deliberately causing losses in real operations; exploration should be able to explain what was learned, rather than treating error itself as an accomplishment.
Group settings also benefit from genuine disagreement. While it may seem reasonable to assign someone to play “devil’s advocate” in idea-generation or decision-making groups, Charlan Nemeth’s research shows this approach is less effective than having someone who genuinely holds a minority view. A devil’s advocate who personally agrees with the majority but argues the opposite can actually reinforce the majority position rather than sparking real dissent.
Sources for this section: where good ideas come from
Networks multiply combinations and consume attention#
In Ridley’s account, Innovation thrives in cultures of free idea exchange and open information distribution, and it flourishes in environments where people can share insights and lived experience. Innovators rarely succeed in isolation; instead, they thrive as part of communities, contributing to and benefiting from shared wisdom. Historical examples include Fibonacci, who introduced the Indian numerical system to Europe and thereby strengthened commercial activity, and the Wright brothers, who drew on the work of a diverse network of early flight innovators. Joint initiatives have also driven major progress in agriculture: the collaboration between Vogel and Borlaug, which successfully incorporated genes from Japanese dwarf wheat strains into other wheat varieties to reduce plant height, demonstrates the power of international cooperation in delivering transformative advances. Ideas shared freely across borders can spark transformative innovations that reshape entire industries and revolutionize daily life.
Johnson identifies networks of interconnected ideas as a hallmark of the most successful ideas. Networks can take many forms: physical communities, online spaces, or any structure that facilitates collaboration between people — they even include the neural networks within the human brain. Larger networks are generally more effective at spawning and propagating new ideas.
Johnson treats cities as important historical innovation networks: compared with smaller groups with more limited contact, agricultural settlement and urban growth widened encounters, while shared culture and written records let ideas be stored, transmitted, and improved. This is a historical account of communication scale, not a claim that hunter-gatherer societies lacked knowledge transmission. The internet adds rapid connections and information access across distance, but the number of connections still needs to be considered alongside filtering, trust, and the ability to collaborate.
Johnson compares social networks with brain networks, using rough source-era estimates of about 100 billion neurons and 100 trillion connections to emphasize relationships rather than isolated nodes. These are approximate figures from the account, not updated counts. Neural plasticity—the ability of connections to form new patterns—is relevant to learning, but it does not independently prove that enlarging an organization’s network increases creativity.
Shortform adds: Networks are widely recognized as beneficial to innovation, but larger size does not automatically translate into higher productivity: participation is often uneven, and a small number of people can end up carrying a disproportionate share of the work, leading to collaborative overload that actually reduces output. Experts therefore advise employers to respect employees’ time and energy limits and encourage only a manageable level of collaboration rather than pushing for maximum interaction.
Networks also enable innovation through the division of labor. In agricultural societies, communal living meant not everyone had to produce food; as communities grew, more people gained free time to pursue other work. This same principle appears in modern organizations through job specialization, where labor is delegated so some people keep operations running while others focus on innovation.
Digital tools have extended networked collaboration. During the Covid-19 pandemic, when in-person work was disrupted, tools such as Google Docs, Zoom, and Slack helped remote teams continue to function and their workplace networks to thrive.
At a cognitive level, neural networks follow a similar pattern: the strongest connections form between neurons that are functionally similar or physically close, just as people tend to connect with others who are like-minded or nearby. Weaker connections, however, are essential to brain plasticity, allowing the brain to adapt to new input much faster than if it had to build entirely new pathways from scratch.
Shortform adds: The convergence of different cultures is one of the key strengths of exaptational hubs such as cities and coffeehouses, where ideas from varied backgrounds recombine in unexpected ways. However, cultural mixing also carries risks: research suggests it can sometimes suppress good ideas when people clash over cultural differences. Notably, people who consider themselves open-minded may be particularly prone to this kind of ideological shutdown. Participants in these creative environments are therefore advised to avoid being overly critical of others’ differing perspectives, so they do not cut themselves off from the very recombinations these spaces are meant to foster.
Sources for this section: how innovation works, where good ideas come from
Exploration needs real capacity#
Johnson argues that long hunches need practical time and support. Berners-Lee’s conception of the World Wide Web developed over years of accumulated interests, traced by the source to childhood; room to explore during his work at CERN helped turn it into a working project.
Google’s “20% time” is another historical example Johnson discusses: employees could explore interests beyond their formal roles; most ideas were not adopted, while a few might produce high returns. He names Gmail and AdSense and cites the then vice president of Search Products and User Experience as saying more than half of new products originated in that time. This illustrates organizational room for exploration, not the current policy or a predicted return for every employer.
There is also an open-access dimension to Johnson’s Berners-Lee example. Shortform describes his work through the World Wide Web Foundation to promote access to the web, connecting it to the time, resources, and prior knowledge that others need in order to innovate. This is an account of the activity recorded in the source.
Shortform records a contrary experience: some employees said regular assignments remained intact, turning 20% exploration into a 120% workload. Critics cited in its account claimed in 2020 that no major new product had emerged from the practice since 2014 and questioned whether it continued, while the company said it did. These are claims in that period’s debate.
Shortform nevertheless recommends trying structured innovation time. Citing a secondary report that the average worker is productive for about three hours a day, it speculates that exploration might replace web browsing or social-media time. That average does not establish spare capacity in a particular company. Whether regular assignments are reduced, who absorbs unfinished work, and what happens to output determine the cost; allocating 20% on paper does not automatically create additional capacity.
Sources for this section: where good ideas come from, innovation and entrepreneurship
How tools change the path to discovery#
Technology can act as a collaborative partner in generating sudden insights, rather than being a passive tool. Johnson cites his own use of DEVONthink, an indexing program whose algorithm retrieves not just verbatim matches for a search term but also semantically related results — for example, returning entries containing the word "waste" when the user searches for "sewage." He reports that this tool has yielded insights he does not believe he could have reached on his own, and he frames those insights as the product of collaboration between two intelligences: his human intelligence and the software's artificial intelligence.
Google search offers a similar collaborative function, but with the far larger information base of the entire internet at its disposal. Johnson addresses the common criticism from writers that internet search has replaced the old practice of going to a library, browsing the shelves, and pulling out books that catch your eye. He counters that this kind of serendipitous library browsing was never a regular habit for everyone, or even for most people; internet filters, which he argues are actually beneficial because they screen out irrelevant results, now enable far more people to explore information than ever browsed libraries in the way critics lament.
Shortform adds a distinction to that defense: library browsing is filtered by a reader’s interests, whereas search also actively ranks results. Its secondary account lists factors such as search history, site authority, and traffic, and speculates that Google may favor YouTube because it owns it. The distinction helps explain different discovery paths; the ownership explanation remains speculation in the supplied material, not a verified ranking mechanism.
In supplementary material from 2022, Shortform describes another form of collaboration: employees use AI to retrieve information and suggest applications, while generative models produce images and text from human prompts, described by some commentators as “artificial imagination.” The material still treats humans as the main creative source and cites a proposal to recognize computers that contribute to patentable inventions as inventors. These are source-era observations and a normative proposal, not a statement about present capabilities or legal eligibility.
Sources for this section: where good ideas come from
