I wrote the following as a brainstorming/warming-up excercise to develope my own understanding after being invited by Computer Music Journal to write guest editorial about Microsoft's recently obtained algorithmic composition patent. The following is the extended rough draft from which CMJ's editor, Doug Keislar, culled the final much briefer version that will be published in the Winter 1998 issue of CMJ.

Thoughts on Microsoft's Algorithmic Composition Patent

by Laurie Spiegel, July, 1998

When Doug Keislar suggested the possibility of my writing something on Microsoft's new algorithmic composition patent, we briefly discussed either a straight forward editorial or a statement to be put forth for comment by others in the manner of several previous Computer Music Journal Editors Notes series. We never fully decided which way to go, so I wrote the following hoping to combine those aproaches by throwing the topic out to the entire community for response via CMJ's Letters to the Editor section, instead of soliciting responses from a preselected group to publish simultaneously with my own thoughts as is the usual practise.

That difference aside, the purpose of this ("(Guest) Editor's Notes"?) column remains the same as for previous CMJ Editors Notes projects: to present a topic for public discussion among participants in the field, and to solicit interesting written responses to the questions raised for publication, in order to further our discussion and thought, and to document the range of attitudes, opinions, and philosophies that currently exist.


Patent Shock: (or) The Criminalization of Algorithmic Composition?

by Laurie Spiegel


Upon learning that Microsoft was recently (May 19, 1998) granted a patent (#5,753,843) for a "System and Process for Composing Musical Sections" (including 47 claims), this patent clearly pertaining to the algorithmic generation of music, my personal reaction went through a series of stages, each of which constituted an alternative perspective, and perhaps an alternative ideology. A narrative of this somewhat-philosophical journey follows:


Perspective 1:

My first reaction was subjective and somewhat fearful. Immediately, I felt personally sold out and potentially under attack, as though my personal freedom of expression had been compromised, as though any future algorithmic compositional work I might do could get me hauled into court for violating the rights of artificially constructed legal entities ironically devoid of all ability to comprehend others' personal sonic self-expression or to subjectively understand why humans feel a need to make music at all. It felt as though a significant part of individual human sovereignty had been inadvertently lopped off by a giant reaping machine designed for altogether unrelated purposes.

A patent may either create or prevent the establishment of a standard practise, by imposing top down on a society through force of law a legal monopoly. It is quite different from the establishment of common practises culture-wide through gradually evolving concensus among practitioners of whatever art or craft is affected.

What would be the problem with patents? Afterall, there have been patents for musical inventions such as piano action parts for many years without apparent detriment?

A major problem results from recent changes to the concept and nature of artistic creative processes, the designing of creative processes and techniques in themselves having throughout our century increasingly come to be considered an integral part of an artist's creative work, instead of the province of a separate being functioning as a tool builder to the artist.

If specific compositional techniques are now to be privately owned, must each composer, especially composers of computer-based interactive process pieces (such as are often presented as installations in the arts community where I live or distributed as computer software), now stop at every stage of the designing of each such composition to do a patent search or else risk being taken to court? With the floodgates now open for a gold rush of corporate claims to very specific compositional techniques, how are we composers supposed to preserve our sense of freedom, our exhileration at exploring, and our deep psychological immersion in following musical ideas wherever they may lead us, while knowing that we cannot ever be sure any more of the simple legality of any new refinement we make in our own process-based works?

Are we composers going to end up having to pay a royalty to the owner of each technique we use when the royalties we receive from the music we compose using these techniques typically would not even pay for the paperwork of just keeping track of them? Must the education of every composer who wants to make process-based music now include courses in what techniques are exclusively owned by whom and for how long, what the fees and methods will decriminalize their use, and the penalties for unauthorized use? Will process-based composing or composer construction of interactive algorithmic tools now become so complex legally (and possibly dangerous too) that such approaches will simply die out? How am I to keep the question of who owns a compositional technique I find myself wanting to use from interfering with my personal sense of creative freedom?


Perspective 2:

Wishing to approach the topic less subjectively, more analytically. I try to generalize this new development, to extrapolate, to map it to other better known contexts in order to test my understanding of this new realm of application of the concept of private ownership. I want to figure out how such monopolies on creative processes are supposed to benefit the art of music, for the basis of intellectual property law is supposedly to provide some benefit back to the creative artists or art or art-using society at large by use of these laws. I find myself asking how the use of patents as a method of motivating technical development for compositional techniques would have worked in historical contexts, in hope of finding this system not such a counterproductive development for music as I feared,

To hypothesize an instance, what if someone had patented the replaying of a musical theme at a time delay to itself early in the era of Bach? By what scenario might the result still give us the most important imitative countrapuntal repetoire of all time, the cannons, fugues, et cetera that inspired so many lives and so much later work? If that patent had existed in Bach's time, could that music still have been written? if not, would other music of comparable quality have been written instead, or were the kinds of mental processing that motivic imitation embodied the appropriate, natural, or perhaps even the necessary musical art of the times such that no other method of structuring music would successfully match or express the dominant mental processes of that era and place?

If so, if no other techniques would have been equally right for the times, would Bach have been able to afford, and also willing to pay, royalties (or - perish the thought: legal defense fees) to use or build new techniques based on the patented imitative contrapuntal processes his works required? Or would composing the way he did have made him a criminal, as Galileo and others came to be considered criminals for their scientific work?

What if composers such as Bach relied instead on the patent holders building and selling them the only composing and editing tools they could legally use? They would at least now have legal access to means of writing imitative counterpoint. But could any patent-holding tool designer possess sufficient understanding of the working requirements of composers of Bach's artistic caliber to be able to create adequately fine-tuned procedural tools for their use? Could any degree of generality and power built into a system provided by someone else have been sufficient to give Bach and each of his contemporaries the exact and complete set of technical methods each would have needed for his unique musical approach?

Even very successful composers of the highest level of quality would be unlikely to be able to design creative tools that could properly and fully serve each other's artistic needs. Would the exact same techniques work for both Handel and Bach? Mozart and Haydn? Chopin and Liszt? Debussy and Ravel? Debussy and Stravinsky? Stravinsky and Schoenberg? Schoenberg and Kurt Weil? Could any tool builder whose products had been fine tuned for Haydn's methods have anticipated the ways in which Beethoven would need to break out beyond their scope or why this would be important or how to accommodate the change in advance?

Then why does our own society assume that the definition, implementation, and provision of any such creative technique should be done non-competitively, non-pluralistically, under the complete control of any single corporate monopoly? Why should the monopoly we call "patent" include artistic methods, tools, or techniques within its domain? Simply because these can now be constructed within the medium of computer software and the law now allows the patenting of processes that are implemented as computer software?

Even aside from the effect such monolpolies and anti-pluralism might have on artists, potential artists, and the art medium per se, what might this change do to other technology developers for the art? If a patent owner has the power to bundle a specific set of musical techniques with or integrate them into the most dominant computer operating system of our times (one already so dominant that the government was already investigating multiple possible antitrust law violations prior to the granting of this new patent-based monopoly), how are small developer-publishers of musical tools supposed to compete successfully and stay in business in order to furnish variety?

The only arguably successful scenario I was able to envision that took as a premise the existence of an 18th century patent on time-delayed repetition techniques and that would still allow us the Well Tempered Clavier, the Musical Offering, and the Art of the Fugue was one in which Bach got himself hired as Microsoft's official court (well, company) composer. But probably Teleman would have gotten the job instead.


Perspective 3:

My third reaction was judgmental, prescriptive and activist. I found myself passionately wishing to be able to rouse a global community of creative artists to protest. But, were we to decide to act against such practises, other than adopting patent infringement as a new form of civil disobedience, what recourse might we have?

I conjured visions of performing arts organizations lobbying government for legislative protection, possibly for the exemption from private ownership of all techniques of individual artistic expression, perhaps on First Amendment grounds. Or maybe a cartel could be organized consisting of all the largest music technology corporations, these customarily competing companies collaborating to challenge this patent or to help achieve such a legal exemption for arts processes. Now that a common enemy had entered the field to their probable mutual detriment, in an unexpected alignment of all musicdom ­ music and music technology companies, their customers and the arts world at large could act as a single political force.


Perspective 4:

In my fourth stage of reaction, I spontaneously found myself looking for a way to understand why an individual inventor would assign the exclusive rights to his own musical methods to one of the world's largest, most powerful, and increasingly most feared corporations. I hoped to find hooks for accepting this new status quo through use of empathy, employing my imagination to answer this burning question: Why would any individual computer-musician give any such giant corporation the exclusive right to any artistic technique he or she valued - ever?

Protecting an invention from being copied is not the only reason for obtaining a patent. Patenting a technique also adds exchange value to its existing use value. The technique can now be bought, sold, or licensed as a commodity.

But barring extremely dire circumstances and lack of all other possible recourse, how could money alone could not provide sufficient reason for an artist or inventor to give over to a giant corporation the complete right to control one's creation? Perhaps as part of a trade-off, such as if the inventor had sold this monopoly to Microsoft in order to support what he perceived as a greater good of some other kind with the procedes.

Might he then actually believe that Microsoft would create such a profoundly wonderfully general, accessible, powerful, adaptable, and individualizable implementation of his techniques that no one would ever have any reason to want to explore or use alternatives to the patent holding corporation's implementation? Such optimism is possible but success in achieving this would be unlikely in music, where individuality is paramount such that there can never be any single best to do anything. Wouldn't any one set of methods inevitably have limits and make assumptions potentially precluding some other way of working? Can there exist any musical model sufficiently general that it would be mutually exclusive to no other model? Certainly someone could believe so enough to try.

The most likely other reason I came up with that made at least practical sense was that an artist-inventor would want to secure support for the implementation and channels for the distribution of his personally preferred methods if he really believed in them and their technical and financial requirements exceeded the resources available to him by other means. This is a comprehensible gamble even given the built-in risk that the patent assignee might fail to make publicly available the desired result while continuing to prevent anyone else from even trying to.

Much to my surprise, I had come up with multiple ways that a person could conceptualize giving to one of the earth's largest corporation a monopoly over a range of specific creative techniques without seeing it as overly detrimental to the art.

Ultimately, though, I did not manage to find any reason that would have been sufficient to persuade me to sell such a monopoly over my own preferred creative techniques to such a company myself. At this point I decided to write to the person listed as the "inventor" on the Microsoft patent to learn his take on this directly. (An excerpt from his response is at the end of this essay.)


Perspective 5:

My next stage of reaction was quite unexpected. I woke after 3 hours of sleep seeming to view the entire matter as though from a great distance, as a scholar from an alien world might do while visiting a distant planet. I awoke feeling as though I were perceiving with quite an unaccountable objectivity and a complete emotional detachment some far and completely foreign culture.

In the process of searching online for information on this patent, I had discovered perhaps two dozen other algorithmic music patents already assigned to a small number of other large corporations. (Many of these concerned methods of deriving accompaniments from existing melodies, possibly Karaoke-related since most belong to Japanese firms.)

I had initially assumed the Microsoft patent to be a rare instance of deviation from a normal long-held view that musical techniques are to be shared among, teachable to, and usable by all. The current case might not after all be an anomaly. Ignornant of the extent to which the patent system had been applied to my field in the last very few years, I had been too ready to see the Microsoft occurance as a highly questionable departure from well-established normal humanistic practises, as the result of an unanticipated transposition of legal methods from the early industrial culture for which they were designed to an aesthetic domain for which they had never been intended.

But now my new less obvious perspective was framing a different view of this group of patents, seeing them as they might eventually be seen: as the first beginnings of a new common practise, presaging a vastly different post-millenial culture, at least insofar as the arts would be concerned. As I found myself mellowing into a more scholarly stance, my mind moved with relative detachment through a variety of sociological and anthropological perspectives. Among the questions I encountered in those travels were these:

What would be the characteristics of music in particular, and of the arts in general, in a society that allowed corporations to proscribe the use of specific creative artistic techniques as criminal?

How might these differ, aesthetically, qualitatively, from the results of nature's prohibition of specific creative techniques at various points in human evolution because such techniques were simply not feasible within the constraints of our then-current stage of technological development? How might we try to analyze the defining nature-enforced cultural constraints of the human paleolithic, bronze, pre-electronic industrial and pre-digital electronic periods?

What other aesthetic or methodological influences on artistic or expressive media have resulted from monopoly or territoriality pertaining to technique?

In what ways might technical limits correlate with aesthetic limits?

What would be the methodological considerations in trying to study such questions?

What qualitative differences might be identifiable between different sources of artificially imposed prohibitions against specific artistic practises? How might the outlawing of artistic methods by private corporations differ from historical artistic prohibitions enforced by religious and political institutions and socioeconomic situations?

We now take for granted the clearly audible differences between music produced by different kinds of common practise techniques. For example music made by imitative contrapuntal processes such as canon or fugue is easily distinguishable from music constructed by interpolation such as melismatic trope. In the same way, might private corporate ownership of specific artistic techniques result in easily perceptible corporate aesthetics? Might there come to be comparably distinct Microsoftian and (why not?) Apple-onian musics?

Would these correlate with different patterns of cultural and then individual cognitive processing? Might we find ourselves inhabiting a world in which not only specific aesthetic qualities and means, but also structurally related modes of thought, were owned by corporations? It is not a greap leap from there to see these artificial corporate entities as developing associated cultures in the same ways and to the same degrees that we now take for granted for a nationality, ethnicity, or historical period: that we would expect each to have an aesthetic style or musical personality and also a unique world view and way of thinking and perceiving. The difference would be that these would have the legal right to prohibit the sharing of such characteristics.


Perspective 6:

In the midst of asking such hypothetical questions, I notice how far my thinking has strayed by now from my first reaction to news of this patent. The hypotheses I now consider are becoming increasingly detailed and more farfetched at once, as though my mind is trying to sharpen it's focus on an increasingly fuzzy image. Both methodology and purpose need attention.

What separates scholarly hypotheses from speculative fiction (aka "science fiction")? What is the role of hypothesis in scientific study, anyway? Do criteria even exist for evaluating whether a question is too speculative to be of any actual value or use?

A methodology-based "useful" answer may be to ask how many unsubstantiable premises we have posited all at once. But the question of meaning or purpose may require that we ask if we can use the insights such thought experiments produce to actively affect the direction an anticipated evolution will take or to control the specifics of its form as it unfolds.

We cannot just swallow whole Star Trek's "Prime Directive" of non-interference. We are not visiting scholars from another world whose function is merely to report back to the home planet on a new world we have glimpsed. This is a one-and-only human world.

We are specialists in a field that we ourselves created from the intersection of our very own individual visions and desires. This field, our lifework, is increasingly influenced by differently motivated entities. The legal departments of large corporations or those more concerned with the price of corporate shares trading in international markets may govern whether or not the next Art of the Fugue will ever be made. This is neither without precedent nor easily under our control, but we do want to try to avoid oncoming disasters and to preserve whatever we value that might otherwise be lost. It is not wrong to try.


Perspective 7:

I took a break and checked my email.

Todor Fay, who is listed as the inventor on the Microsoft patent, had written me back most positively as follows:

>Would you or others in the composer community be interested in
>DirectMusic? It's far more extensible and pluggable than Bars&Pipes,* and it
>ships as part of the Windows OS. There's a pluggable authoring tool as well,
>so you can create music technologies and create the tools to edit the
>technologies hand in hand. One big objective is to make it easier to
>Developing this beast has been an incredibly gratifying experience for me.
>I'm very lucky to have this opportunity.

Have I been overly cynical, thinking in general terms as though trying to formulate a rational policy based on general principles, from fear of loss of freedom?

[*Bars&Pipes was a patchable modular music architecture which Todor Fay wrote for the Amiga in the late 1980s and which he and his wife Melissa Grey published independently under their company name "Blue Ribbon Bakery".]


Perspective 8:

The following anonymous sentence had also arrived via the Hackers Conference listserver:

>"The best thing about patents is that they eventually expire."*

Indeed, patents can be viewed as mere temporary glitches in the otherwise uninterruptable universal availability of all techniques to all creators.

[*This brief quote was later attributed to Phil Karn by its sender.]


Perspective 9:

This is an ongoing sequence of thoughts that can never be exhaustively transcribed, nor will following it further achieve the closure I had hoped to find. These are interim ideas in a potentially infinite speculative sequence about a future that ultimately no one can predict before it unfolds. By the time this text reaches print, the thought-stream above will doubtless be ancient conceptual history relative to wherever my mind will have taken these topics by then. Although my mind will continue riding such trains of thought as though with a mind of its own, I will impose an artificial stopping point here.


Comments, corrections, additions, alternatives, reductia ad absurda et cetera, and many more questions, are invited in response by CMJ, not by the author personally. Please send them to CMJ.

- Laurie Spiegel

Copyright ©1998 Laurie Spiegel. All rights reserved.
back to Laurie Spiegel's writings page | go to Laurie Spiegel's home page