(Written In response to a suggestion on THE SEEDS OF INNOVATION thread that improvisation is a vital part of innovation.)

The claim is that the "next great innovations in music will come from musicians who can further integrate rich complex structures and organization of classical music with the "in the moment" uniqueness of improvisation."

Improvisation is a vital part of the arts, but is this the key? Let's look at this in-depth.

Robert Jourdain goes into this topic in MUSIC, THE BRAIN, and ECSTASY:
“ There is nothing extraordinary about improvisation. We all improvise constantly, but in words, not tones.” Yet, if improv were the key to innovation, we’d all be innovators, yet we’re not. Jourdain explains that improv does not contribute heavily to compositions with “deep structure.” “…[I]provisation can proceed by association from moment to moment, free of any preconceived form, as is the case of ‘free jazz.’ Deep structure is inevitable lacking in this style…Yet even in its apparent chaos, ‘free’ improvisation is constrained….Much of what is ‘new’ follows pathways that have worked in the past. Like all music, it is largely ready-to-play. It cannot be otherwise, for even a well-prepared mind can work only so quickly.
• The classic line about classical music being “stiff” and and lacking spontaneity is partly due to the “lack of improv.” It’s also often used in support of “the noble savage” theory, that the white man has grown too academic and theoretical (though a ring of truth is lent to this when Rand talks about how scientists like “Robert Stadler” argue for pure science with no real-world application) while the primitive natives are “more spontaneous” and “in tune with nature.” There’s also a saying that goes “Marines don’t plan, they improvise.” But there is the tradeoff of spontaneity for long-term planning, or what Phil Coates calls “long chain thinking.” Anything that develops beyond short-term goals with have to require a plan. You don’t see contractors and workers "improvising" a skyscraper.
• In addition to the “noble savage” defense, there is also an element of “second-handedness” involved in the anti-score improv argument, born of a Communist ideology. Communists argue that “bourgeoisie” institutions like the orchestra are a product of an hierarchal approach that stifles the workers (the workers being the musicians and the composer being the “factory owner.” The composer has, in reality, applied his logic and will to bring the piece into fruition, while the individual musicians are more like technicians, whose performance depends on the product of the composer’s mind. Of course, we can’t have that, say the proletariat, and argue for the technicians to run the factory via “improvisation.” The other example is seen in THE FOUNTAINHEAD, when other architects want to “express themselves” as well on another man’s work.
• Because of the need for planning in complex endeavors, it’s safe to say that most innovations will require the same. It requires more than a mindless spewing of random notes, or “noodling,” as you rightfully call it. The exact reference escapes me, but Rand had commented somewhere, in response to a question similar to our discussion, that “thinking in pictures” is not the same as full-fledged thought. Not that it’s wrong, but more is required for a fully conscious understanding.
• Improvisation usually brings to mind “blues” jamming or “jazz”, often a repetitive vamp while the soloist spews forth a string of notes. This is less “spontaneous” then it seems, because those riffs are usually standard issue for the genre, requiring that the musicians have studied the existing script, even if it’s not formally written out, as in classical. If innovation requires new abstractions, it’s hard to say where the new innovation is in such jams.
• The idea of “improv” as self-expression. All art is, in a sense, born of improv, the way that most conversation is “improvised.” (And when it’s not, we can usually tell: “You sound like you’re reading from a script.”) The innovation comes in when a new abstraction is presented, or as a vehicle for that new abstraction.
None of this is meant to disparage improvisation; it’s a vital tool in the artist’s arsenal. No great works of art were created from a “paint-by-numbers” approach; it requires a mind that is not afraid to experiment and stretch out. And Jourdain rightfully notes that as performance, "improvisation can be a marvel-a conjuction of physical technique, musical understanding, and creative flair."
But it only really works for the mind that is ready for it. Jourdain touches on improv by composers, and the image of savant composers. But usually those composers lose their gift, and never having understood the process to begin with, cannot continue. He writes that “A brain simply can’t generate effective deep structures quickly enough. Rather, composers mine their improvisations for ideas and then develop the ideas methodically. For them, improvisation at an instrument at an instrument is merely an extension of the improvisations they spawn through auditory imagery all day long.”