WO2001048964A2

WO2001048964A2 - System and method for optimizing the performance of email and other message campaigns

Info

Publication number: WO2001048964A2
Application number: PCT/US2000/035404
Authority: WO
Inventors: Sanjay Ranka; Hitendra K. Wadhwa
Original assignee: Paramark, Inc.
Priority date: 1999-12-29
Filing date: 2000-12-27
Publication date: 2001-07-05
Also published as: WO2001048964A3; AU2602401A

Abstract

This disclosure describes the system, apparatus, methods, and computer program and computer program product for improving or optimizing the performance of messages including emails. The methods include parameterizing the message alternatives in a multi-attribute space. The methods also include sending messages for a set of alternatives, collecting response data for the alternatives (Fig. 7 #760), analyzing the response data to allocate messages to different alternatives and repeating this process from the beginning until all the messages have been sent or a business objective has been achieved. The sending of messages (Fig. 7 #710) may involve allocating unequal numbers of messages to different alternatives, and the unequal allocation may involve concentrating messages on better-performing alternatives. The optimizing apparatus (Fig. 7 #750) may generate different number of messages at different stages. The sending of messages may commence before a preceding analysis completes. Similarly, generation and sending of messages for a given stage may commence before response-data collection of the previous stages is complete. An optimizing apparatus may combine the message process with other business process for better overall optimization. Email type messages optimized according to the invention are described. These can be used for advertising and marketing as well as a variety of other purposes.

Description

SYSTEM AND METHOD FOR OPTIMIZING THE PERFORMANCE

OF EMAIL AND OTHER MESSAGE CAMPAIGNS

Related Applications This application claims the benefit of the filing date(s) of the following earlier application(s):

U. S. Patent Application Serial No. 60/173,689, entitled, "Optimizing the Performance of Emails," filed December 29, 1999, naming Sanjay Ranka et al. as inventors, with Attorney Docket No. P-68768/RMA/LM and under an obligation of assignment to Paramark, Inc., of Sunnyvale, California.

This application is related to the following applications:

U. S. Patent Application Serial No. 09/ , , entitled, "System, Method, and Business Operating Model for Optimizing the Performance of Advertisements and Other Messages in an Interactive Measurable Medium," filed 2000, naming

Sankrant Sanu et al. as inventors, with Attorney Docket No. A-69257/RMA, and under an obligation of assignment to Paramark, Inc., of Sunnyvale, California; and U.S. Patent

Application Serial No. 09/ , , entitled, "Method, Algorithm, and Computer Program or Optimizing the Performance of Messages Including Advertisements in an Interactive

Measurable Medium," filed 2000, naming Sanjay Ranka et al. as inventors, with

Attorney Docket No. A-69258/RMA and under an obligation of assignment to Paramark, Inc., Sunnyvale, California.

The applications to which benefit of priority is claimed and the other related applications are incorporated herein by reference.

BACKGROUND

A prior-art or conventional example of direct-mail advertising is now described. The goal of the direct-mail advertising campaign is typically to maximize responses to advertisements for a product A, sent to a population G. A typical direct-mail campaign for the product A is divided into two phases: a test campaign and a main campaign.

In the test campaign, an advertiser selects a set of advertisement alternatives for the product and mails them to a sub-population H of the population G. The advertisement alternatives number n, while the size of the sub-population H is |H| where (|H| « |G|). This sub-population H is divided into n groups: HI, H2, . . . , Hn such that |H1| + |H2| + . . . + |Hn| = |H|. All members of a group Hi receive the same advertisement alternative. (Typically, the sizes of the trial groups are about equal. That is to say, |H1| ≡ |H2| ≡ . . . ≡ |Hn|.)

The advertiser uses the data collected in the test campaign to determine which of the n advertisement alternatives generates the best response. For example, where the advertisement is in the form of a letter, the envelope of the letter may be varied n different ways to allow the advertiser to determine which of the n envelopes generates the best response.

Indeed, for many applications, the advertisement alternatives can be described in terms of a set of attributes. An "attribute" is an differentiating characteristic of an advertisement alternative. For the above example of a letter advertisement, the envelope might be described as having two attributes: a first, the color of the envelope, and a second, the design (such as text, graphic, or image) on the envelope. Herein, the term "level" refers to one of the choices along a given attribute. Assume that the color of the envelope has three levels: pink, white and blue. If there are two different designs that may be used (i.e., the design attribute has two levels), the total number of possible advertisement alternatives is equal to six.

The advertiser conducts trials (i.e. send out mailings) for each of the advertisement alternatives. Assuming that each of the alternatives is treated independently (i.e. there is no underlying multi-attribute structure), the total number of trials required is proportional to n, the total number of alternatives. However, if the advertisement alternatives can be described by a set of attributes, this underlying structure can be exploited to reduce the number of trials required substantially. For some cases, with a multi-attribute structure, the total number of trials required is proportional to the sum of the levels along each of the attributes. This is generally much smaller than n.

At the end of the test campaign, the advertiser collects the responses for the mailing of the n different advertisement alternatives to the sub-population H and determines which of the advertisement alternatives in the test campaign generated the best response. Let's say advertisement alternative k produces the best response from the subpopulation H. In the main campaign, the advertiser mails to the remaining population (G-H) advertisement alternative k.

As described above, the steps of sending of the advertisements, collecting the responses and determining the best alternative are each executed only once in the test campaign. A long delay between the design of the test campaign and the response to the campaign precludes repeating this process multiple times.

The number of alternatives over which the direct mail advertiser can learn in the test campaign is also limited because there is a large fixed cost or overhead for developing each alternative.

An advertiser desires to accelerate the pace of learning in the test campaign, while simultaneously investigating a broader set of alternatives along the way.

An advertiser further desires to design a large number of advertisement alternatives with multiple attributes and to test and analyze a campaign in a shorter time period and at a lower cost.

While conventional direct-mailing methods have been set forth as a basis for understanding embodiments of the invention, comparable problems and limitations are presented in the more general case of messages and messaging systems and methods.

In the messaging context, each message may have a goal which may succeed or fail, and where the characteristics or attributes of a message may be selected to improve the performance of the selected messages or of the messaging campaign overall. As for the advertising campaign, the message originator desires to accelerate the pace of learning in the test phase of the messaging campaign, while simultaneously investigating a broader set of messaging alternatives along the way. A message originator further desires to design a large number of message alternatives with multiple attributes and to test and analyze a messaging campaign in a shorter time period and at a lower cost. These and other goals of the invention will be readily apparent to one of skill in the art on reading the background above and the description below.

SUMMARY

Herein are described apparatus and methods for optimizing the performance of email based messaging campaigns, including advertising or marketing campaigns. The marketer (or other message originator or controller) provides the apparatus with a list of email or other message alternatives which may have an underlying multi-attribute structure. The method includes sending messages, such as emails, for the campaign, for different alternatives in multiple stages. At each stage, the method involves sending messages, such as emails, for a subset of alternatives, collecting response data for these alternatives and analyzing this response data to bias the allocation of messages (emails) to alternatives for the next stage. This process is repeated until a given objective is met or all the messages (emails) for the test campaign are used by the method.

The number of messages, such as emails, sent out to different alternatives in a given stage may be allocated non-uniformly. Further, the non-uniform allocation may involve concentrating messages (emails) on better-performing alternatives. The total number of messages (emails) sent out in each stage may also be non-uniform.

The method also allows for sending of messages, such as emails, for a stage before the response data collection is complete for all or a subset of the previous stages. In general, a message includes any communication, fact, idea, symbolic representation, or the like, that is communicated. Such messages may for example include but are not limited to advertisements for products and/or services, political campaigns, ballot measures and initiatives, media programming, lobbying, surveys, polling, news headlines, sports scores, as well as other directed marketing, promotions, surveys, news, information, other content generally, and the like. An email is a particular type of message.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a table showing the number of emails and the number of responses for an example marketing campaign with n = 9 email alternatives, each with k = 2 attributes.

Figure 2 shows the results for the first stage of a multi-stage campaign. Figure 3 A gives the number of emails and the number of responses collected for each alternative during stage 2 of the multi-stage campaign.

Figure 3B gives the cumulative results up to stage 2 for all the unpruned alternatives (i.e. all the alternatives that are allocated some emails) in the multi-stage campaign. Figure 4A gives the results collected in stage 3 of the multi-stage campaign.

Figure 4B the cumulative results up to stage 3 for the unpiuned alternatives in the multi-stage campaign.

Figures 5 and 6 present two possible allocations of emails for a two-stage campaign.

Figure 7 illustrates an optimizing system according to one embodiment of the invention.

Figure 8 illustrates a test-and-main-campaign strategy according to the prior art. Figure 9 is a flowchart illustrating a process for decomposing the learning/optimization process in the multiple stages.

Figure 10 illustrate sub-steps of the multi-stage process of Figure 9. Figure 11 illustrates components of the email server of Figure 7.

DESCRIPTION OF EMBODIMENTS OF THE INVENTION

Embodiments of the invention are now described relative to the figures.

Section headers are provided merely to assist the reader and are not intended to limit the description of the embodiments of the invention or the invention as a whole in any way.

Those workers having ordinary skill in the art will appreciate that aspects and features of the invention and of embodiments of the invention are described throughout the specification and in the drawings.

AN OPTIMIZING SYSTEM

Aspects and features of the invention are now described relative to email type messages as they represent a particularly useful form of message that is widely used.

It will be appreciated by workers having ordinary skill in the art in light of the description provided here and in the related applications incorporated by reference that the invention applies more generally to other message types and is not limited to email type messages.

Figure 7 is a diagram illustrating an optimizing system 700 according to one embodiment of the invention. The optimizing system 700 includes an outbound message or email mailer 710, a message or email-response collector 760, a database of message or email assignments, responses and response statistics 720, an intelligent message or email system 750 and a population P.

Communications links 730, 770 and 780 connect the database 720 to the outbound mailer 710, the response collector 760 and the email system 750, respectively. A communications link 740 connects the population P to the mailer 710 and the response collector 760. As Figure 11 illustrates, the email system 750 includes one or more of the following: a central processing unit ("CPU") 751, a memory 752, a user interface 753, a port 755, a communications interface 756 and an internal bus 757.

Of course, in an embedded system, some of these components may be missing, as is well understood in the art of embedded systems. In a distributed computing environment, some of these components may be on separate physical machines, as is well understood in the art of distributed computing.

The memory 752 typically includes high-speed, volatile random-access memory (RAM) 7522, as well as non-volatile memory such as read-only memory (ROM) 7521 and mass storage devices such as magnetic-disk drives. Further, the memory 752 typically contains or stores computer program software 7523. The software 7523 is layered: Application software 75231 communicates with the operating system 75232, and the operating system 75232 includes an I/O subsystem 75233. The I/O subsystem 75233 communicates with the user interface 753 and the communications interface 756 by means of the communications bus 757.

The port 755 provides access to the user interface 753, including some or all of the following devices common in computer systems: a display 7531, a keyboard

7532, a mouse 7533, a touch-screen overlying an LCD (not shown), a pen (not shown), a trackball (not shown), etc. The communications interface 756 includes at least one input/output port 7561.

The communication bus 757 communicatively interconnects the CPU

751, memory 752, user interface 753 and communications interface 756.

Any of the outbound message or email mailer 750, response collector 760 and database 720 may of itself be a complete computer system. A characteristic of the communications link 740 is that it allows the population P to respond quickly C if not nearly instantaneously C to communications from the email outbound mailer 710. Any of the communications links 730, 740, 770, 780 may be an Internet or other network, according to one embodiment.

The email system 750 is programmed to execute one or more of the methods described herein and in that sense is intelligent.

A SINGLE-STAGE TEST CAMPAIGN The rapid increase in the number of email users or other electronic messaging users, the low cost of its distribution and the ability to target individual consumers have made email an important medium for advertising and marketing, as well as to other directed or targeted messaging campaigns. Many businesses now readily use email to acquire new customers, build brands, advertise and promote products, measure consumer satisfaction and manage customer relationships. A typical email campaign involves sending emails to each address on a list of recipients, the list bought or otherwise acquired from an outside firm or collected internally over a period of time.

Email alternatives used for marketing or advertising may have several attributes: the subject field, the initial greeting and one or more offers, for example. A more general message type may have analogous attributes as well as other attributes. Each of these attributes may have multiple levels. The product of the number of levels of the attributes is the size of the pool of alternatives. Two different alternatives differ from each other by the levels assigned to at least one of attributes. This represents a feasible set of alternatives from which a marketing manager (or other action manager) may seek to find the best alternative to optimize business objectives such as:

Maximization of the number of responses received for an email (or other messaging) campaign. (The response may be clicking a link in the email to visit an Internet site or replying to an email or other message, as two examples.) Maximization of the revenues generated by customers buying a product based on the information and/or links in the email campaign. Analogous sets of alternatives from which a messaging originator or manager may seek to find the best alternative message slimilarly exist.

In pursuit of these objectives, marketing managers typically follow direct-mail practices, dividing the list of email recipients into two disjoint sub-lists: a list for the test campaign and a list for the main campaign. Figure 8 is a diagram illustrating a test-and-main-campaign strategy according to the prior art, and step 850 illustrates the step of dividing.

The marketing or other message manager then processes the test campaign list as follows:

Generate a list of email alternatives, step 810. ^• Send multiple emails (to different recipients on the list) for each alternative, step 820.

Collect response data for each alternative, step 830. Analyze the data to find the best alternative, step 840.

The test campaign is used to learn the performance of different email alternatives. Emails for the best alternative (as determined by the test campaign) are then sent to the list for the main campaign. This is similar to the testing phase and main phase of a direct-marketing campaign.

In the above described process, the steps of sending of the emails, collecting the responses and determining the best alternative are each executed only once in the test campaign — that is, the test campaign is a single stage process. This process is herein modified and improved for email-based marketing. These modifications and improvements are possible because a response for an email typically happens in a few hours to a few days. Further, the additional cost of generating emails for a large number of alternatives (as compared to small number of alternatives) is nominal. This allows for a larger number of alternatives to be used for a given cost.

These characteristics of fast response and flexible generation contrast with land-based (so called "snail-mail" based) direct-mail marketing where response times are typically on the order of 8-12 weeks and where the high cost of generating different alternatives limits the numbers of different alternatives available for experimentation.

Herein are taught methods for accelerating the prior-art single-stage process within email and other fast-response and flexible-generation environments. The methods may be applied along with a large number of established techniques from statistics, data mining and optimization.

The methods improve the prior-art single-stage process in any or all of several ways: First, a method parameterizes all email (or other message) alternatives with multiple attributes. Second, a method sends emails (or other messages) for the test campaign in multiple stages. Third, a method combines the optimization of other business processes in conjunction with emails (or other messages). Each of these improvements is discussed in turn below.

Multiple Attributes

Email or other message alternatives can generally be represented by multiple attributes. Example attributes include a subject field, an initial customer greeting, a logo or other picture and product offers. (Each offer typically has a clickthrough or hot-spot area. Clicking on the area is a customer response. Each offer corresponds to a different attribute.) In an example email application, let the alternatives have k attributes, each with multiple levels. Where each attribute i (1 <= i <= k) has level[i] number of levels, the resultant test space then consists of levelf l] x level[2] x ... x level[k] alternatives. (Some combinations of attribute levels may not be feasible and may be eliminated from consideration.)

Suppose the goal is to find an email alternative that has the highest performance for a given business objective. If each alternative is tested independently, the number of emails required is proportional to the total number of alternatives, which can be a very large number.

Exploiting the multi-attribute structure of the alternatives space can yield methods that require a significantly smaller number of emails to determine the best alternative. In many situations, correlation between the levels of one attribute and the levels of another attribute is negligible. This negligible correlation may allow for reducing the total number of emails required during the test campaign for determining the best alternative. The total number of emails, for such special cases, may be proportional to the sum of levels along each of the attributes.

(Experiment designs and strategies exploiting this structure are well-known in the art. See, for example, P. G. Evans, Business Statistics: Methods and Applications (R.D. Irwin Inc., IL, 1985), which is incorporated herein by reference.)

Figure 1 is a table of cells showing the number of emails sent and the number of responses for an example marketing campaign of n = 9 email altematives with k = 2 attributes. Each cell corresponds to one alternative. (The lower number in each cell gives the number of emails sent for a given email alternative and the upper number the number of responses.) Each of the attributes i (1 <= i <= 2) has three levels. For example, 100 emails were sent out for the alternative with attribute 1 at level 1 and attribute 2 at level 3. There were 5 responses for this email alternative.

Based on the responses received for each of the alternatives as given in

Figure 1, one may conclude that the 5 email alternatives corresponding to attribute 1 at level 3, or attribute 2 at level 3, are under-performing with the required degree of confidence. (In some situations, this conclusion may only be possible at attribute granularity. The information at the cell level may not be sufficient to reach the same conclusion.)

Multiple Stages

Assume that the marketer or other message manager would like to find the best email (or other message) alternative from a given list of alternatives. A method for decomposing the testing process into multiple stages is as follows:

(a) Generate emails (or other messages) for each alternative based on the current allocation of email alternatives, step 910. The allocation of emails to different alternatives determines the fraction of total emails to be sent out to each alternative.

(b) Send emails (or other messages) based on the above step to estimate the performance of each alternative, step 920.

(c) Bias allocation of emails (or other messages) to different alternatives, step 930. In a preferred embodiment, the biasing allocation includes the steps of collecting performance response data for each different alternative, analyzing the performance response data to estimate the performance of each alternative. (d) Repeat steps (a), (b) and (c) until the best alternative has been determined, step 940, or there are no more emails (or other messages) available to sent out in the test campaign. Decomposing the testing process into multiple stages allows for learning accrued from a given stage to be used in the generation of the emails (or other message types) in the next stage, thereby optimizing overall learning and/or performance.

The advantages of the inventive multi-stage process benefit from the fast responses to and flexible generation of emails or other electronic message types. While applicable, these methods are not particularly suitable for traditional surface mail-based direct marketing where the response times may be prohibitively large.

Sub-steps of this multi-stage process may include allocating unequal number of emails to different alternatives, step 923, or allocating unequal number of emails in each stage, step 926. Figure 10 illustrates these sub-steps, which are also described in turn below.

One approach sends a fixed fraction of emails in a test campaign followed by the remaining emails in the main campaign. The main goal of the test campaign is to leam the best alternative. This is then used to optimize the responses in the main campaign.

The inventive method can take advantage of the continuous nature of multi-stage testing and combine the test and main campaigns — i.e., there is only one campaign with multiple stages. The goal is then the maximization of the performance for the entire campaign.

In allocating unequal numbers of emails to different alternatives, the inventive method biases the number of emails sent for different alternatives in order to maximize learning, performance or both. Investments (in terms of sending emails or other message types) are desirably made in the earlier stages on all or most of the alternatives to discover high-performing email alternatives. However, for maximizing performance, a higher concentration of emails should be sent to better-performing alternatives. A marketer or other message manager has to balance this tradeoff to maximize performance for the entire marketing or other messaging campaign.

Consider a representative example of a multi-stage process with 9 email alternatives (n = 9). Each email alternative consists of two attributes (k = 2). For this example, the total number of emails to be sent in each stage is set to 900. For the sake of the clarity, at each stage, the allocation scheme stops allocating emails to some alternatives and allocates all the emails for a stage equally to all the remaining alternatives. Figure 2 shows the results for the first stage. Based on these results, an optimizing system operating according to the invention may conclude that the 5 email alternatives with attribute 1 at level 3 or attribute 2 at level 3 are under-performing. It prunes these 5 alternatives i.e., the system does not allocate any emails to these alternatives (in step 920) for the remaining stages.

In stage 2, the optimizing system allocates the 900 available emails among 4 alternatives. Two hundred twenty five emails are sent with each of these alternatives. Figure 3A gives the number of emails sent and the number of responses during stage 2.

Figure 3B gives the cumulative results up to stage 2 for all the remaining alternatives. Based on these results, the optimizing system may conclude that the email alternatives corresponding to attribute 2 level 2 are under performing and prunes them for the remaining stages.

In stage 3, emails are sent only for the two remaining alternatives. The optimizing system decides that 450 emails are to be sent to each alternative for this stage.

Figure 4 A gives the results collected in stage 3, and Figure 4B the cumulative results up to this stage for the unpruned alternatives. Based on these results, the optimizing system may conclude that email alternative with attribute 1 at level 1 and attribute 2 at level 1 is the best.

All the emails sent in the remaining stages are then sent to this email alternative. The same procedure may be applied when the message is of a type other than an email.

This example illustrates the effectiveness of a multi-stage process that biases the allocation of emails or other messages away from poorer-performing alternatives and focuses on better-performing alternatives. The performance gains of using a multi-stage method are dependent on the performance variation between the different alternatives. If all the alternatives have similar performance, the improvements may be negligible. However, if the variation is substantial, a multi-stage method will have the opportunity to observe the performance differences between poor- and high- performing alternatives at early stages. This may allow it to bias emails away from the poor-performing alternatives at early stages and hence, improve overall performance.

In allocating unequal numbers of emails in each stage, step 916, the taught method biases the total number of emails in different stages. The optimizing system may generate allocations for different alternatives for the next stage before it finishes collecting data on the alternatives for the current stage. Indeed, the optimizing system may generate allocations for the next stage even before it completes sending emails for the current stage.

The biasing of emails across multiple stages optimizes the following trade-off: On the one hand, sending a large fraction of emails in the earlier stages produces better learning for the later stages, but the advantages of this learning may benefit only a small fraction of emails. On the other hand, sending a small fraction of emails in the earlier stages may result in poor learning for later stages, but the advantages of this learning may benefit a larger fraction of emails. An optimal schedule balances the above trade-offs. It will be understood by workers having ordinary skill in the art in light of the description provided here that the use of such terms as optimal, optimum, or the like do not require optimal or optimum performance in an absolute sense, rather the terms imply an improvement or an attempt to identify or attain a high-level performance which may be better than the performance than would otherwise be achieved, such as a near-optimal if not in fact optimal performance.

Consider an example two-stage process with 9 email alternatives (n = 9), where each email alternative consists of two attributes (k = 2). Let the total number of emails to be sent over the two stages equal 2700. Figures 5 and 6 present two possible allocations of emails for the two stages. In the first allocation (Figure 5), the optimizing system sends 900 emails in the first stage and 1800 emails in the second stage. It uses the learning in the first stage to prune all but four altematives for stage 2. The optimizing system may then allocate all the emails sent in the second stage between these four alternatives. A total number of 201 responses are generated over the entire campaign.

In the second allocation (Figure 6), 1800 emails are sent in the first stage. This may result in better learning as compared to the first allocation and allows for pruning all but two alternatives. In the second stage, all the emails are sent equally to these two alternatives. A total of 189 responses are generated. Thus, although better learning may have been achieved using the second allocation, that better learning nonetheless resulted in lower overall success. This is because the number of emails from which the learning benefit could be accrued in the second stage was much smaller as compared to the first allocation.

The above is one illustrative example. Simulations on synthetic data sets demonstrate that on the average, an allocation of emails across different stages can lead to improved performance. Optimization with Other Business Processes

Emails are a vehicle for drawing a consumer in to participate in other business processes. The optimization of such business processes may be combined with emails or other message types. Examples of other processes where such combination improves decision-making include choosing the splash page (defined below) that acquires the most new customers and setting prices of different products on the web site to optimize revenues, sales or profits. Treating emails as an integral part of the overall process results in better optimization.

Consider the integration of emails with the splash-page process. When a customer clicks on an email, he/she is redirected to a web page using an internet protocol. This web page is also called the "splash page." Typically there are several splash-page alternatives.

The goal of the marketer is to maximize a business objective such as the total amount of revenue generated. One way to achieve this objective is to optimize the two independent (but concurrent) phases:

1) Maximize the total number of responses (visits to the splash page) using optimal allocation of emails. 2) Maximize the sales generated using optimal allocation of splash pages to all the visitors. Other message types may be directed to achieving other business directives, yet the business directives may be analogously achieved.

By integrating the optimization of email or other message alternatives with the optimization of splash-page alternatives, one can simultaneously optimize both elements of the system. This is superior to separately optimizing the two phases because it generates the best end-to-end results. Some example reasons for better results in an integrated process are: 1) An email alternative may generate a low response rate. However it may generate a large amount of revenue per email. In a non-integrated process, such emails alternatives may get smaller allocations and this will affect the overall objective negatively.

2) A splash-page alternative may generate poor revenue for the mix of visitors from the better email alternatives but may generate very high revenue for visitors from a poor-performing email alternative (in terms of response rate). In a non-integrated process, such splash pages may get a small fraction of total visitors and affect the overall objective negatively.

Just like the email alternatives, the splash-page alternatives may have an underlying multi-attribute structure. Integrated optimization can be achieved by combining the email attributes with the attributes of the splash-page process into a combined attribute space - the total number of attributes will be equal to the sum of the number of attributes for the emails and splash pages, respectively. The total number of alternatives in the combined space is proportional to the product of the number of email alternatives and splash-page alternatives.

If the splash pages do not have multiple attributes, the splash pages can be treated as a single attribute in the combined attribute space. The levels of this attribute will correspond to the different splash page alternatives.

Embodiments of the invention may advantageously be implemented as computer program or programs executing within a processor and/or associated memory in a general purpose computer. For example, the inventive methods and procedures may be implemented as one or more computer program on a server computer system. Such server computers and the execution of computer programs within such systems are known in the art and not described in detail here. When implemented as a computer program product, the computer program may exist and be communicated over any communication link such as the internet or be stored on a tangible medium such as on magnetic storage medium, optical storage medium, solid state storage medium, or any other form of storage medium.

The invention now being fully described, many changes and modifications that can be made thereto without departing from the spirit of the appended claims will be apparent to one of ordinary skill in the art. For example, as already mentioned in the detailed description, email is one type of message and the inventive system and method described here may be utilized with other message types.

Furthermore, the methods and procedures for optimization described in co- pending U.S. Pat. App. Ser. No. _/ , (Attorney Docket No. A-69258/RMA), filed May 2000 and entitled Method, Algorithm, and Computer Program for Optimizing the Performance of Messages Including Advertisements in an Interactive Measurable Medium, may be used in conjunction with the invention though alternative optimization methodologies and algorithms may be used as well. In addition, the invention may advantageously be practiced in conjunction with the system and methods described in co- pending U.S. Pat. App. Ser. No. _/__, (Attorney Docket No. A-69257/RMA), filed May 2000 and entitled System, Method, and Operating Model for Optimizing The

Performance of Messages In An Interactive Measurable Medium, though other systems and methods may alternatively be employed.

Claims

WHAT IS CLAIMED IS:

1. A method for optimizing the performance of messages, said method comprising: sending messages for each of a set of alternatives; collecting response data for said alternatives; analyzing said response data; allocating messages to the said alternatives; and repeating said method from the beginning until all the messages have been sent or the objective has been achieved.

2. The method of claim 1, wherein said step of sending messages comprises allocating unequal numbers of messages to different alternatives.

3. The method of claim 1, wherein said step of sending messages comprises sending unequal numbers of messages in different stages or loop of said method.

4. The method of claim 1, wherein said step of sending messages commences before a preceding step of analyzing completes.

5. The method of claim 1, wherein said step of sending messages commences before a preceding step of collecting response data completes.

6. The method of claim 1, wherein said message comprises an email message.

7. The method of claim 1 , wherein said step of sending messages for each of a set of alternatives comprises sending messages for each of a set of alternatives over a tangible measurable medium.

8. The method of claim 7, wherein said tangible measurable medium comprises a global network of interconnected computers or information appliances.

9. The method of claim 8, wherein said message is an electronic mail message communicated over the internet.

10. A method for parameterizing and exploiting message alternatives with multiple attributes.

11. A method that combines the optimization of other business processes with message optimization.

12. A method for optimizing the performance of messages communicated over an interconnected network of computers, said method comprising: sending email messages for each of a set of message alternatives; and allocating future messages among selected ones of said message alternatives based on performance of each of said sent messages.

13. The method of claim 12, wherein said selected ones of said message alternatives comprise all of said sets of message alternatives.

14. The method of claim 12, wherein said selected ones of said message alternatives comprise a subset of better performing ones of said sets of message alternatives.

15. The method of claim 12, wherein said step of allocating future messages among selected ones of said email message alternatives based on performance of each of said sent email messages further comprises the steps of: collecting response data pertaining to the performance of at least some of said sent message alternatives; and analyzing said response data to determine an allocation of future messages among said email message alternatives.

16. The method of claim 15, wherein said step of analyzing said response data to determine an allocation of future messages among said email message alternatives further comprises the steps of: estimating the performance of each message alternative; and biasing allocation of said email messages to different ones of said message alternatives.

17. A method as in claim 12, further comprising repeating said steps of sending and allocating in a plurality of stages until all said messages have been sent or the objective of sending said messages has been achieved.

18. The method of claim 17, wherein said step of sending messages comprises: allocating unequal numbers of email messages to different alternatives.

19. The method of claim 17, wherein unequal numbers of messages are sent for at least some of said repeated stages.

20. The method of claim 15, wherein said step of sending messages commences before a preceding step of analyzing completes.

21. The method of claim 13, wherein said step of sending messages commences before a preceding step of collecting response data completes.

22. The method of claim 12, wherein said message comprises an email message.

23. The method of claim 12, wherein at least one of said messages comprises an email message.

24. The method of claim 12, wherein each of said messages comprises an email message.

25. The method of claim 16, wherein at least one of said messages comprises an email message.

26. A computer program for use in conjunction with a computer system, the computer program comprising a computer program mechanism embedded therein, the computer program mechanism, comprising: a program module that directs the computer system to improve the performance of a message campaign in a multi-stage message campaign, the program module including instructions for: sending email messages for each of a set of message alternatives; and allocating future messages among selected ones of said message alternatives based on performance of each of said sent messages.

27. The computer program of claim 26, wherein said program module instruction for allocating future messages among selected ones of said email message alternatives based on performance of each of said sent email messages further includes instructions for: collecting response data pertaining to the performance of at least some of said sent message alternatives; and analyzing said response data to determine an allocation of future messages among said email message alternatives.

28. The computer program of claim 27, wherein program instruction for analyzing said response data to determine an allocation of future messages among said email message alternatives further includes instructions for: estimating the performance of each message alternative; and biasing allocation of said email messages to different ones of said message alternatives.

29. The computer program of claim 26, further including instructions for repeating said instructions for sending and allocating in a plurality of stages until all said messages have been sent or the objective of sending said messages has been achieved.

30. The computer program of claim 29, wherein said instruction for sending messages includes an instruction for allocating unequal numbers of email messages to different message alternatives.

31. The computer program of claim 26, wherein said messages comprise at least one email message.

32. The computer program of claim 31, wherein said email message is an advertising email message.