WO2011078646A1 - Client load balancing, power management, and mobility in hierarchical wireless mesh networks - Google Patents

Abstract

In one embodiment, a method for client load balancing is implemented in a wireless hierarchical mesh network. The method includes grouping a plurality of nodes in the network, broadcasting a load information message (LIM) from each of the nodes to provide a plurality of LIMs, each LIM comprising a node-to-stations mapping for stations detected by each respective node, computing a station-to-nodes mapping for each respective node using data collected from the plurality of LIMs, and determining a station-to-node assignment using rank and signal strength information obtained from processing the collected data. Other methods are provided for power management and client mobility in the network.

Description

CLIENT LOAD BALANCING, POWER MANAGEMENT, AND MOBILITY IN HIERARCHICAL WIRELESS MESH NETWORKS

DESCRIPTION

FIELD OF THE TECHNOLOGY

[001 ] This disclosure relates to communication systems in general, and more particularly, but not limited to, load balancing, power management, and client mobility in mesh networks. BACKGROUND

[002] WiFi access points have become extremely popular in the last decade and are seen everywhere - homes, offices, public hotspots, and other locations. The reason for their strong success is that they are secure, enable mobility, provide high bandwidth and are very simple to operate. Their main limitation however is their range, which is typically limited to 50-100 meters. A client station can benefit from mobility and high throughput offered by the access points only as long as it is within this range. Even though this limited range is sufficient to cover small offices and homes, it cannot be extended to cover large areas like campuses and townships. [003] To provide WiFi coverage to a large area like a campus, several access points may be deployed at various points on the campus in a grid-like pattern. A well known problem with this approach, and a difficult one to solve, is running power and Ethernet cables to the access points. While the power to the access points can be tapped from the nearest power outlet, the Ethernet cable connecting the access point to the central switch can be prohibitively long and expensive. It is often impossible when, for example, the cable has to cross streets and private properties. To overcome this problem, WiFi meshes have been proposed. [004] A WiFi mesh is an interconnection of nodes using wireless links in a topology that is more general than a star topology. The key idea behind a WiFi mesh is that an access point sends its traffic to a neighboring access point wirelessly instead of sending the traffic directly to the backhaul switch over an Ethernet. The neighboring access point then transfers the traffic to its neighbor, and so on to yet other neighbors, until the traffic reaches the special access point called the gateway, which, for example, connects the mesh to a wired network using Ethernet or other wired link technology. Since the gateway access point is typically the only access point using a wired connection to the backhaul switch, the problems associated with cabling is eliminated.

[005] WiFi meshes have their own share of challenges. As WiFi-based mesh networks grow larger and carry more load, their scalability and performance becomes more critical.

BRIEF DESCRIPTION OF THE DRAWINGS

[006] The embodiments in this disclosure are illustrated by way of example and not limitation in the figures of the accompanying drawings.

[007] Figure 1 shows client load balancing in a mesh network according to one embodiment.

[008] Figure 2 shows radio calibration in a mesh network according to one embodiment.

[009] Figure 3 shows signal level thresholds in a mesh network according to one embodiment.

[0010] Figure 4 shows a client mobility framework in a mesh network according to one embodiment. [001 1 ] Figure 5 shows a two-level mesh network according to an exemplary embodiment. [0012] Figure 6 shows a mesh network device, suitable for use in the mesh network of Figure 5, according to an exemplary embodiment.

[0013] Figure 7 illustrates a three-level mesh network according to an exemplary embodiment. DETAILED DESCRIPTION

[0014] Systems and methods to provide a mesh network are described herein. The disclosure below includes methods and apparatuses which perform these methods, including mesh networks and systems which perform these methods, and computer readable media containing instructions which when executed on computing devices or data processing systems cause the networks or systems to perform these methods.

[0015] The following description and drawings are illustrative and are not to be construed as limiting. Numerous specific details are described to provide a thorough understanding. However, in certain instances, well known or conventional details are not described in order to avoid obscuring the description. References to one or an embodiment in the present disclosure are not necessarily references to the same embodiment; and, such references mean at least one.

[0016] Reference in this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosure. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Moreover, various features are described which may be exhibited by some embodiments and not by others. Similarly, various requirements are described which may be requirements for some embodiments, but not other embodiments. [001 7] In United States Patent Application Publication No. 2008/0192713 (published August 14, 2008, and titled "Hierarchical Mesh Network for Wireless Access" by Mighani et al.), which is hereby incorporated by reference in its entirety, a hierarchical mesh network architecture for wireless access is described that scales well in size and capacity. This architecture and the systems and methods described therein may be used with various embodiments that are described below. Other architectures may be used in alternative embodiments.

[001 8] Three new general approaches are described below for use with a mesh network architecture that improve mesh network performance and/or capabilities. These approaches are as follows:

(1 ) Client Load Balancing,

(2) Power Management, and

(3) Client Mobility. CLIENT LOAD BALANCING

[0019] As client stations join and leave a mesh network, load imbalances often occur, where some nodes have significantly more clients connected to them than others. These imbalances are due to an inherent limitation of the 802.1 1 architecture, because in the 802.1 1 architecture both clients and access points do not take into account the number of clients already connected to the access point when establishing new connections. To improve performance of the mesh, these imbalances need to be rectified so that clients are more evenly distributed among the nodes.

[0020] The metric used to measure load on a node can simply be the number stations connected to the node, or can be a more complex metric such as, for example a metric based on the traffic generated by the clients. Without loss of generality, in one embodiment the number of clients is used as the load metric when the load balancing mechanisms are described herein. [0021] Client load balancing is a two stage process: (1 ) during the first stage, the load on the nodes is measured to determine overloaded nodes, and (2) during the second stage, certain clients disconnect from the overloaded nodes and reconnect to less-loaded nodes. [0022] Some previous approaches to client load balancing use a central controller to track load on the nodes and perform the load balancing tasks. For instance, U.S. Patent No. 7,177,649, which is hereby incorporated by reference in its entirety, presents a method where the central controller, an MSC (Mobile Switching Center), decides based on load which cell site in a cellular network should service a mobile client. Similarly, in U.S. Patent No. 7,554,962, which is hereby incorporated by reference in its entirety, the central controller uses physical signal characteristics and load on the channel as a criteria in its load balancing decisions.

[0023] U.S. Patent No. 7,406,319, which is hereby incorporated by reference in its entirety, describes a method for controlling incoming client associations. In this decision process, access points use load information transmitted from the mobile clients, containing a list of potential access points to which the mobile client can connect. It blocks further connections and terminates existing connections if needed to balance load. In U.S. Patent Application Publication No. 2003/0139197, which is hereby incorporated by reference in its entirety, beacons transmitted by the nodes carry load information. The client then uses this information along with signal strength in selecting the node to connect to.

[0024] Both the above approaches differ from our method in that the node neither considers load on the neighborhood nodes during the selection process, nor does it redirect the client to a less-loaded node.

[0025] Our approach differs fundamentally from previous approaches in that there is no central control device and the control decision is distributed among the nodes. To perform load balancing, the nodes in our approach exchange load information with a select number of other nodes in the neighborhood. As local load balancing is performed among a small number in a group rather than global load balancing, it may scale readily to larger meshes.

[0026] A significant aspect of our approach is to organize nodes in groups and to perform load balancing within each group, in one embodiment, independently of other groups. Thus, the nodes in the mesh are divided into smaller, disjoint groups in which a group's members are physically closer to each other. Each group may be identified with a group ID.

[0027] Figure 1 shows client load balancing in a mesh network according to one embodiment as described herein. [0028] The groups can be established based on different criteria. For example, they can be based on physical proximity determined via GPS (global positioning system) information, where the distance between any two nodes has a maximum limit. Note that the CLB (client load balancing) groups may be different than clusters in the mesh hierarchy. For example, the nodes in the CLB group can belong to different clusters.

[0029] Each node in a group periodically, every T seconds, broadcasts Load Information Messages (LIM) to other nodes of the same group. These messages carry the node-to-client mapping that specifies the list of clients detected by the node. The mapping also specifies if a particular client is connected to the node.

[0030] In one embodiment, an LIM message carries the following information among others:

1. The MAC address of the node broadcasting the message.

2. The node-to-clients mappings. Each mapping specifies the MAC address of the client the node detected along with the corresponding RSSI (Receive Signal

Strength Indication) and the connection status. RSSI is a measurement of power in the received radio signal, and connection status specifies whether the client is already connected to the node. [0031] In one embodiment, the broadcast of LIM messages is synchronized so that the messages are received within a certain deterministic time bound. For example, the synchronization may be readily achieved by equipping each node with a clock and GPS device. [0032] A node, after receiving the LIM messages from all members of its group, processes them to generate the following information:

1. Load Table: The table specifies the number of clients connected to each node, their MAC addresses and signal strengths (RSSI). It obtains this information from the LIM messages received from the members of the group. The load metric can simply be, for example, the number of clients connected to the node or a complex function of signal strength, traffic rate, error rate and type of traffic.

2. Potential Assignment Table: This table specifies the reverse map, that is the list of nodes that are visible from the client's perspective, and is computed from the information collected in the LIM messages. The list of nodes for each client are sorted in descending order based on signal strength. The number of nodes in the list is rank of the client.

3. Optimal Station Assignment Table: This table specifies the optimal station assignment, that is, the node each station should connect to achieve load balancing. In alternative embodiments, the station assignment might not necessarily be considered optimal, but rather be a desired assignment (e.g., as desired for other or external reasons).

[0033] The nodes process LIM messages and build the above tables locally and independently, and since nodes report the same information in the LIMs to every other node in the group, in a steady state, all nodes in the group will eventually have the same load information, and hence the tables built will be identical. [0034] We describe embodiments below using algorithms to perform the client assignment: one deterministic and the other non-deterministic.

[0035] Algorithm for Non-Deterministic Client Load Balancing

1. Every T seconds, the node builds an LIM message and broadcasts it to members of its group.

2. A node /, after receiving LIM messages from all nodes in the group for a particular period, determines the average number of clients per node, represented by G, by dividing the total client associations (connections) by the number of nodes in the group. The number of associations at the node is represented by N_i.

3. The excess load L_i at node /^' is calculated by the formula L_i - N_i - G. Similarly, the surplus S_/^' for each node / is calculated by subtracting the number of associations N_i at the node from the average, that is S_i = G - N_i.

4. If the number of associated clients N_i is greater than a predetermined threshold K, and there is a surplus S J on at least one other node j in the group, the following actions are triggered.

(a) Some clients, at most N_i - G, are blacklisted and forced to disconnect from the node. The blacklisted clients are then blocked from connecting again for a period of T seconds. (b) After T seconds, the blacklisted clients are removed from the blacklist. During this duration of T seconds, the blacklisted clients are expected to connect to another node in the group that has a surplus. To prevent a ping-pong effect, in which a client is bounced back and forth between nodes, a counter is maintained in the node for each client to limit the number of times a client is blacklisted.

[0036] Note that the non-deterministic algorithm only uses the load table and does not use the Potential and Station Assignment tables. [0037] Algorithm for Deterministic Client Load Balancing

Each node performs the following steps periodically and independently.

1 . Every T seconds, the node builds an LIM message and broadcasts it to members of its group. 2. Periodically, a node /^', after receiving LIM messages from all other nodes in the group, does the following computations:

(a) A reverse map for each station, which is a list of nodes the station can potentially connect to, is computed from the data collected from the LIM messages and stored in the Potential Assignment Table. This list can have one or more nodes, and the number of nodes in the list is the rank of the client. The nodes in the list are sorted in descending order of the signal strength.

(b) The stations are sorted in ascending order of their rank. The load of each node is initially set to zero.

(c) The node assignment for each station is then determined as follows and stored in the Optimal Assignment Table: i. If the client's rank is 1 , then there is only one node in the potential assignment table. So, it is assigned to the only node in its reverse map. The load of this node is incremented by one. ii. If the rank is more than 1 , the client is assigned to the node in its reverse map with the least load. If there are two or more nodes with same load, the node with the strongest signal is selected. The load on the selected node is incremented by one.

3. Finally, the newly computed optimal node assignment is compared with the current node assignment, and if there are differences the stations are reassigned according to the new assignment. Otherwise, no action is taken. The node performs the reassignment by first forcing the clients that are not assigned to it to disconnect and then allowing only clients that are assigned to it to connect.

POWER MANAGEMENT

[0038] The power levels for the radios in network nodes are typically set to their maximum at the time of the node installation in order to ensure connectivity, even for installation in an unfriendly radio frequency (RF) environment. However, transmitting at high power when not needed often results in unnecessary interference that can quickly degrade network performance. The power management system described below may be used to automatically set radio power to levels that are sufficiently low enough to minimize interference, but at the same time sufficiently high enough to be able to transmit at high data rates.

[0039] Our approach differs from the prior art in several ways. In U.S. Patent No. 7,366,202, which is hereby incorporated by reference in its entirety, and U.S. Patent Application Publication No. 2008/0274745, which is hereby incorporated by reference in its entirety, interference mitigation is through dynamic selection of the best channel among multiple channels that are available. The basic approach is that of interference avoidance by configuring radios on different channels. However, in these prior approaches, after dynamic channel selection there will still be situations where two or more radios must still use the same channel thereby causing interference. In contrast, our approach addresses interference mitigation even when two or more radios are on the same channel through calibration and power level adjustment.

[0040] In U.S. Patent No. 7,058,146, which is hereby incorporated by reference in its entirety, transmitting base stations (nodes) get signal strength feedback from clients and then all base stations mitigate interference by coordinating the timing of their transmissions. This is different from the method we propose in which power levels are adjusted to mitigate interference. [0041] U.S. Patent Application Publication No. 2009/0197590, which is hereby incorporated by reference in its entirety, discusses a power level adjustment technique to mitigate short-term interference, in which a base station first sends a probe message to the client. In response, the client checks the signal strength from all other base stations in the neighborhood and sends a reply message to the base station suggesting a power level to use. The base station then reduces its power level and sends a pilot transmission at the suggested power level. The client then determines the channel quality and sends feedback information indicating quality, based on which the base station may further adjust its power level. This differs from our approach in that it uses only signal strength measurement between base stations and clients and not between two base stations, and further there is no optimization at a global or other higher level.

[0042] In one embodiment of our approach, each level of the hierarchical mesh is defined by a specific radio in the node. At the lowest level (corresponding to the access radio), the node connects to the client devices, and at each of the higher levels (each corresponding to one mesh radio), the node connects to other nodes to form clusters. So, at each level in the hierarchical mesh, where nodes are organized in clusters, nodes belonging to the same cluster need to experience a strong signal while nodes belonging to different clusters must minimize signal interference. At the lowest level, there are no clusters and inter- nodal interference should always be minimized.

[0043] In various embodiments, the radios in the hierarchical mesh can use many technologies such as WiMAX, 3G, LTE (Long Term Evolution), Picocell and FemtoCell, as long as there are nodes that group together to form clusters and there is a notion of nodes belonging to the same cluster or a different cluster, and that nodes in same cluster have strong signal and nodes in different cluster have minimal interference. The power management approach described herein may be generally applied to any such network.

[0044] In general, a cluster is group of nodes that are on the same channel and connect to each other using wireless links. [0045] The general rules for power level setting are different for access and t mesh radios and are outlined below. The difference is due to the fact that access radios typically need to maintain a strong signal only to clients, but should not interfere with access radios of other nodes, where as a mesh radio needs to maintain strong signal to mesh radios of nodes in the same cluster, while mitigating interference with mesh radios of nodes of other clusters.

[0046] In one embodiment, the power setting rule for an access radio is as follows:

1. The power level for the radio is set high enough to cover as large an area as possible.

2. The power level for the radio is set low enough so as not to interfere with any other access radio also on the same channel. For this we use a signal threshold value defined ACCESS_RADIO_SIGNAL_THRESHOLD. A signal strength received by a node should not exceed this threshold. [0047] In one embodiment, the power setting rule for a mesh radio is as follows:

1. The power level for the radio is high enough to reach every other mesh radio in the same cluster. We define the signal threshold

INTRA_CLUSTER_SIGNAL_THRESHOLD. The signal received from a particular node should exceed this value.

2. The power level for the radio is low enough so as not to interfere with another mesh radio on the same channel in a different cluster. For this we define the signal threshold INTER CLUSTER SIGNAL THRESHOLD. A signal received from a node not belonging to the same cluster should not be greater than this value.

[0048] In one embodiment, the Power Management System consists of a Power Management Server and a Power Management Client. There is one instance of the Power Management Server that runs either on a dedicated machine that is connected to a node or a backhaul wired network, or runs on one of the nodes itself. An instance of the Power Management Client runs on each node in the mesh, which does radio calibration and reports the generated signal vectors to the Power Management Server. [0049] Radio Calibration is the process during which nodes measure the strength of signals received at various power levels from other nodes in the neighborhood. In one embodiment, the signal strength data, which is represented in signal vectors, collected during the radio calibration process is stored in a local data repository and then transmitted to the Power Management Server.

[0050] The Power Management Server processes the signal vectors and performs calculations to determine an optimal, or otherwise desired, power vector, which specifies the power level of each radio of each node. The Power Management Server then sends the optimal power vector to Power Management Clients on the nodes. Each node then extracts the power settings for each of its radios from the power vector and sets the power accordingly. Alternatively, to reduce communication overhead, the Power Management Server may send only a subset of the power vector that is relevant to each node.

[0051 ] Figure 2 shows radio calibration in a mesh network according to one embodiment. The algorithms for radio calibration and power level optimization in one embodiment are given below.

[0052] Algorithm for Radio Calibration

1 . Each node periodically broadcasts a sequence of Power Beacon Messages (PBM), each sent at a different power level starting from a minimum level (for example, 5db) and incrementing up to a maximum level (for example, 24db). The maximum power level is defined by the radio characteristics. Even though the theoretical minimum for the power level of a radio is zero, we start from some level higher because no communication can take place at zero power level. The Power Beacon Message contains, at the minimum, the node's MAC address, MAC address of the specific radio (mesh or access) and the power level at which the message is sent. Depending on the power level some or all nodes (for example, in a small network where nodes are physically close to each other) may receive the power beacon message sent by the node. 2. When a node receives a Power Beacon Message from another node it records the signal strength at which it received it. The collected signal strength information is compiled and reported to the Power Management Server in a Calibration Message. The Calibration Message at a minimum contains [Node_MAC, Sender_MAC] and a set of signal vectors each containing [RSSI, Powerjevel].

[0053] The Power Management Server stores the information received in the calibration messages in calibration tables, for example, on local storage. It periodically processes data from the calibration tables using a Power Level Optimizer (PLO) to determine the optimal power settings for each radio. The PLO may be, for example, implemented in software on the gateway node.

[0054] Figure 3 shows signal level thresholds in a mesh network according to one embodiment.

[0055] To measure the interference level in the mesh, in this embodiment the Power Level Optimizer uses a cost metric called "interference cost" represented by lnterference_Cost. The goal of the PLO is to find the power setting for which the lnterference_Cost is a minimum.

[0056] The algorithm to calculate interference cost for a given power vector is given below. In this embodiment, a separate cost for each of the access and mesh levels is computed and the optimization for each level is performed independently of the other levels in the mesh network. For the access level we use Access_lnterference_Cost to represent the interference cost, and for the mesh level we use Mesh_lnterference_Cost. Algorithm for Access Radio Interference Cost

For a given power vector,

For each node, if the RSSI from another node, represented by

Access_Receive_RSSI, is greater than

ACCESS_RADIO_SIGNAL_THRESHOLD, then the Access_lnterference_Cost is incremented by the following:

Weight x (Access_Receive_RSSI - ACCESS_RADIO_SIGNAL_THRESHOLD), where Weight is a constant used to scale the value.

Algorithm for Mesh Radio Interference Cost For a given power vector,

(a) For each node, if the RSSI from another node, represented by

Mesh_Receive_RSSI, in the same cluster is less than

INTRA_MESH_SIGNAL_THRESHOLD, then we set the cost for this power vector to infinity and invalidate the power vector. (b) For each node, if the RSSI from another node in a different cluster is greater than INTER_MESH_SIGNAL_THRESHOLD, then

Mesh_lnterference_Cost is incremented by the following:

Weight ^* (Mesh_Receive_RSSI - INTER_MESH_SIGNAL_THRESHOLD)

[0057] The above method is a specific instance of how total interference in the mesh can be determined. In a more general method, multiple thresholds may be defined with incremental cost associated with each threshold. In the above method the cost of each received signal can be represented by the function cost=W*(S-T), where W is the weight, S is the strength of the received signal and T is the threshold. In the most general case the cost can be continuous function of signal strength where a cost is defined at each signal level, such as the function cost-F(S), where F is a general function and S is the signal vector. Typically, the function F is represented by a table that specifies the cost for each level of signal strength. Thus, for each valid power vector, there is a corresponding total interference cost represented by total_cost=∑F(s) over all signal vectors s. [0058] The goal of the power optimization algorithm is to find a power vector that minimizes total interference cost. The power vector is set of P(n,r), where P(n,r) is the power for radio with MAC address r in node n. A brute force approach to finding the minimal power vector is to iterate through all valid power vectors and pick the one with the minimum cost. However, this can be prohibitively expensive, and more efficient methods based on heuristics can be used. A specific example of a heuristic is given below.

[0059] In this embodiment, we here define a heuristics-based algorithm as follows:

Algorithm for Power Vector Computation 1. Mesh Radio Power Vector:

(a) For each radio, increment the power of the mesh radio from a minimum value to a maximum value while checking the power vector is valid (i.e., the intra-cluster nodes record the minimum threshold signal strength

INTRA_CLUSTER_MESH_THRESHOLD). Otherwise, the power vector is discarded. If the power vector is valid, the MeshJnterference Cost is

calculated.

(b) After reaching the minimum threshold signal strength, if there are no inter-cluster signal violations, that is Mesh_lnterference_Cost is still zero, the power is further incremented step by step until a violation is eventually detected (i.e., the Mesh_lnterference_Cost turns non-zero). The power level before the first violation is then noted. 2. Access Radio Power Vector: For each node, increment the power of the access radio from a minimum to a maximum value while checking the calibration tables for each power vector to ensure that no violations occur (i.e., Access_lnterference_Cost is equal to zero). The power level before the first violation is then noted.

[0060] After obtaining the power vector P(n,r) as described in the above algorithm, the Power Management Server communicates it to the nodes in the mesh. The nodes then set the power level of their radio according to the values specified in the power vector. Thus a node k would obtain the power level of its access radio with MAC address M as P(k,M). To reduce overhead, the Power Management Server may send only a subset of the power vector that applies to each node.

CLIENT MOBILITY

[0061 ] In the embodiment discussed below, the mesh is implemented as a layer 3 network where each node with its clients is a separate subnet. So, when a client roams, it will be between two subnets. To support layer 3 mobility, Mobile IP has been proposed and most implementations of IP layer 3 mobility in wireless networks are based on Mobile IP (IETF RFC 3344 Mobility Support for IPv4), which is incorporated herein by reference in its entirety. [0062] In Mobile IP, the first node a client station connects to in the wireless network (or mesh network in our case) is called the home node of the station. When the client is at the home node, it might establish some application level connections, typically TCP/IP, to other hosts or servers on the backhaul network or the Internet or other clients connected to the mesh. These applications level connections are usually referred to as sessions. Subsequently, when the station moves and connects to another node in the network, the new node is called the foreign node, and the sessions established at the home node will generally be dropped if there is no coordination between the three entities - home node, foreign node and the client. The coordination process, called hand-off, involves setting up IP-IP tunnels and routes, for which the required IP address information is exchanged between the three using explicit messages.

[0063] In the new approach presented here, the hand-off is performed without exchange of any explicit IP messages between the client and the network, and the required IP information to set up IP-IP tunnels is directly generated from the Media Access Control (MAC) address of the mobile client. None of the systems or methods described in the prior art proposing modifications to Mobile IP have used the MAC address of a mobile client to generate IP addresses required to set up the tunnels. [0064] The disclosures in U.S. Patent No. 7,228,133 and U.S. Patent No. 7,356,013, each hereby incorporated by reference in its entirety, modify the mobile IP registration process to add parameter extensions to specify the access technology (e.g., WiFi, Ethernet, Bluetooth or cellular) that is used to connect to the network. Similarly, the disclosure in U.S. Patent No. 7,290,064, hereby incorporated by reference in its entirety, extends the mobile IP sublayer to provide special support to applications such as RTP/RTCP and to dynamically adapt to changes in link characteristics.

[0065] The disclosure in U.S. Patent No. 7,471 ,661 , hereby incorporated by reference in its entirety, does not use the mobile IP sublayer in the client device. The node waits for the client to send at least one IP packet to determine the IP address of the mobile client and its home IP address. However, this prior approach still depends on layer 3 communication and does not derive the required IP addresses of the client and home node from the client's MAC address. [0066] Advantages of this approach include the following: (1 ) no software agent or mobile IP sublayer is required on the client devices, and (2) the speed of hand-off is faster as layer 3 communication between the node and the client is reduced or eliminated. [0067] Before we describe our method in more detail, we briefly describe the hand-off in Mobile IP. To perform the hand-off process, the foreign node, home node and the client use software agents that implement complex protocols to exchange the required information. This hand-off process consists of the following tasks:

1. The software agents on the home and foreign nodes periodically send out IRDP (Internet Router Discovery Protocol) messages to advertise their services.

2. The software agent in the mobile client listens to the IRDP messages to detect the presence of a foreign node and registers with the foreign node. 3. The foreign node requests the home node to set up an IP over IP tunnel by providing its own IP address. The foreign node also requests the home node to install forwarding route entry for the client in its routing tables. The home node receives packets for the mobile client via normal routing and then forwards the packets to the client over the tunnel. 4. The foreign node sets up an IP over IP tunnel to the home node and installs appropriate routing entries for the client in its local IP routing tables.

MOBILITY FRAMEWORK

[0068] Figure 4 shows a client mobility framework in a mesh network according to one embodiment. [0069] A significant aspect of our new mobility architecture is that a client has a designated virtual home node (herein called the PoBOX) instead of a home node as in Mobile IP. The home node in Mobile IP is the first node the client connects to, whereas the IP address of the PoBOX is statically determined from the MAC address of the client itself. All communication with the mobile station is through the PoBOX node.

[0070] In the PoBOX approach, the nodes and mobile clients do not use IRDP messages as in Mobile IP to exchange IP information, as the IP addresses of the mobile client and its home node are directly computed from the MAC address of the mobile client itself. This speeds up the hand-off process.

[0071] Each of the n nodes in the mesh has a unique PoBOX IP address. In one embodiment, IP addresses for clients are chosen from a predetermined address range AR which is usually specified in address/prefix form (for example, 10.0.0.0/8). Let the size of this address range be N. This address range is divided into n disjoint subnets AR 1, AR 2,... AR_n, and each subnet is associated with one of the n nodes in the mesh. Alternatively, the subnets ARJ may be chosen for each node and address range AR constructed from it by merging.

[0072] Thus, given any client IP address in the address range AR, the subnet ARJ to which the client IP belongs to and the corresponding PoBOX IP can be determined. A client's IP address is chosen from the AR as follows. Let STA_MAC be the MAC address of the client and STAJP its IP address. offset = as {STA_MAC)

STAJP = base_address( Af?) + offset.

[0073] The hash function generates an integer between 0 and N, the size of the address range AR (for example, the size of 10.0.0.0/8 is about 16M). The hash function can be any function, such as MD5, that can sufficiently randomize the input string. The function base_address gives the lowest address of the address range. The same hashing function is used at all the nodes, and as a result the station's MAC address is sufficient for a node to uniquely determine the client's IP address. From the client's IP address the corresponding subnet ARJ and the PoBOX's IP can be determined. [0074] The PoBOX node is not necessarily the first node the station connects to in the mesh as in Mobile IP, and in fact it is possible that the client may never actually connect to the PoBOX node. [0075] To automatically determine the client IP address and the PoBOX IP address requires routing protocol support in the network infrastructure. The routing protocols in the mesh network propagate the address range ARJ associated with every node to every other node. On the other hand, the ARJ of the nodes can be explicitly configured. Or, each node can download this information from a central server or a gateway node at the time of node initialization. Thus, given an IP address, the subset ARJ containing it can be determined, and from it, the corresponding PoBOX node's IP address is obtained.

[0076] Instead of using a standard DHCP server that issues an IP address to the requesting client from a preconfigured address range, in this embodiment we use a modified DHCP server that issues the IP address generated from the MAC address of the client. As a result, the client's IP address does not change as it moves from one node to another, and each node can quickly determine all the required IP addresses (client IP, node IP, PoBOX IP) without requiring mobile IP support in the client device.

[0077] Now, the hand-off process using the new PoBOX approach is described in more detail for this embodiment as follows. When a client connects to a node , the following tasks are performed:

1. The node /^' obtains the MAC address of the client during the 802.11 association process.

2. From the MAC address, the node derives the client's IP address and the IP address of its PoBOX using the hash function described above.

3. The required data (i.e., node IP, client IP and the PoBOX IP) is known to the node to install IP-IP tunnels. The node installs an IP-IP tunnel from the node to the PoBOX node of the client. This tunnel is called the reverse tunnel.

4. The node requests the PoBOX node to set up an IP-IP tunnel from the PoBOX node to this node. This is called the forward tunnel. 5. The node requests the PoBOX node to set up a destination IP address- based route in its routing table. The packets addressed to the client that are received at the PoBOX node will be forwarded to the node / over the forward tunnel. 6. A source IP address-based route is added to the routing table of the node /^'. Packets received from the client by node / are routed over the reverse tunnel from node / to the PoBOX node.

[0078] In this embodiment, the foreign node determines the client IP address when the client's presence is detected by the 802.11 protocol. No layer 3 packets need to be exchanged. As a further optional enhancement, the foreign node can set up tunnels immediately after a client is detected at layer 2 (i.e., even before the client sends an association request to the node to establish a connection) and defer activation of the tunnel via route set-up only after the client actually connects. This results in faster hand-off. MESH NETWORK HARDWARE AND SOFTWARE IMPLEMENTATION EXAMPLES

[0079] The hierarchical mesh network architecture, hardware and methods as described in United States Patent Application Publication No. 2008/0192713 (published August 14, 2008, and titled "Hierarchical Mesh Network for Wireless Access" by Mighani et al.), hereby incorporated by reference in its entirety, provide hardware and software components that may be used when implementing the Client Load Balancing, Power Management, and Client Mobility approaches and various embodiments described above. For example, in some embodiments, the mesh network device 200 of U.S. Patent Application Publication No. 2008/0192713 is suitable for and may be used as a node in the improved mesh network implementations described above.

[0080] In addition, the hardware, software, and methods described in U.S. Patent No. 7,177,649, U.S. Patent No. 7,554,962, U.S. Patent No. 7,406,319, and U.S. Patent Application Publication No. 2003/0139197, each incorporated herein by reference in its entirety, in some cases provide hardware and/or software components that may be adapted for use in one or more of the Client Load Balancing, Power Management, and Client Mobility mesh network implementations described above.

[0081 ] Further, the hardware, software, and methods described in U.S. Patent No. 7,228,133, U.S. Patent No. 7,356,013, U.S. Patent No. 7,290,064, and U.S. Patent No. 7,471 ,661 , each incorporated herein by reference in its entirety, also in some cases provide hardware and/or software components that may be adapted for use in one or more of the Client Load Balancing, Power Management, and Client Mobility mesh network implementations described above.

[0082] Finally, the hardware, software, and methods described in U.S. Patent No. 7,366,202, U.S. Patent Application Publication No. 2008/0274745, U.S. Patent No. 7,058,146, and U.S. Patent Application Publication No. 2009/0197590, each incorporated herein by reference in its entirety, in some cases provide hardware and/or software components that may be adapted for use in one or more of the Client Load Balancing, Power Management, and Client Mobility mesh network implementations described above.

CLOSING

[0083] In this description, various functions and operations may be described as being performed by or caused by software code to simplify description. However, those skilled in the art will recognize what is meant by such expressions is that the functions result from execution of the code by a processor, such as a microprocessor. Alternatively, or in combination, the functions and operations can be implemented using special purpose circuitry, with or without software instructions, such as using an Application-Specific Integrated Circuit (ASIC) or a Field-Programmable Gate Array (FPGA). Embodiments can be implemented using hardwired circuitry without software instructions, or in combination with software instructions. Thus, the techniques are limited neither to any specific combination of hardware circuitry and software, nor to any particular source for the instructions executed by the data processing system.

[0084] While some embodiments can be implemented in fully functioning computers and computer systems, various embodiments are capable of being distributed as a computing product in a variety of forms and are capable of being applied regardless of the particular type of machine or computer-readable media used to actually effect the distribution.

[0085] At least some aspects disclosed can be embodied, at least in part, in software. That is, the techniques may be carried out in a computer system or other data processing system in response to its processor, such as a microprocessor, executing sequences of instructions contained in a memory, such as ROM, volatile RAM, non-volatile memory, cache or a remote storage device.

[0086] Routines executed to implement the embodiments may be implemented as part of an operating system, middleware, service delivery platform, SDK (Software Development Kit) component, web services, or other specific application, component, program, object, module or sequence of instructions referred to as "computer programs." Invocation interfaces to these routines can be exposed to a software development community as an API (Application Programming Interface). The computer programs typically comprise one or more instructions set at various times in various memory and storage devices in a computer, and that, when read and executed by one or more processors in a computer, cause the computer to perform operations necessary to execute elements involving the various aspects. [0087] A machine readable medium can be used to store software and data which when executed by a data processing system causes the system to perform various methods. The executable software and data may be stored in various places including for example ROM, volatile RAM, non-volatile memory and/or cache. Portions of this software and/or data may be stored in any one of these storage devices. Further, the data and instructions can be obtained from centralized servers or peer to peer networks. Different portions of the data and instructions can be obtained from different centralized servers and/or peer to peer networks at different times and in different communication sessions or in a same communication session. The data and instructions can be obtained in entirety prior to the execution of the applications. Alternatively, portions of the data and instructions can be obtained dynamically, just in time, when needed for execution. Thus, it is not required that the data and instructions be on a machine readable medium in entirety at a particular instance of time. [0088] Examples of computer-readable media include but are not limited to recordable and non-recordable type media such as volatile and non-volatile memory devices, read only memory (ROM), random access memory (RAM), flash memory devices, floppy and other removable disks, magnetic disk storage media, optical storage media (e.g., Compact Disk Read-Only Memory (CD ROMS), Digital Versatile Disks (DVDs), etc.), among others. The instructions may be embodied in digital and analog communication Jinks for electrical, optical, acoustical or other forms of propagated signals, such as carrier waves, infrared signals, digital signals, etc.

[0089] In general, a machine readable medium includes any mechanism that provides (i.e., stores and/or transmits) information in a form accessible by a machine (e.g., a computer, network device, personal digital assistant, manufacturing tool, any device with a set of one or more processors, etc.).

[0090] In various embodiments, hardwired circuitry may be used in combination with software instructions to implement the techniques. Thus, the techniques are neither limited to any specific combination of hardware circuitry and software nor to any particular source for the instructions executed by the data processing system. EXEMPLARY HIERARCHICAL MESH NETWORKS

[0091] Some exemplary, non-limiting, hierarchical mesh networks are described below. These networks may be used to implement the Client Load Balancing, Power Management, and Client Mobility approaches and various embodiments described above. These examples are not meant to limit the generality of the foregoing disclosure.

[0092] The exemplary networks below relate generally to communication systems and, more particularly, to wireless broadband communication networks and methods for data-voice transmission and reception. [0093] One form of commonly-used communication system is a so-called "mesh network." A mesh network is a netlike communications network in which there are typically at least two pathways to each node. A fully-meshed network means that every node has a direct connection to every other node, which is often an elaborate and expensive architecture. Most mesh networks are partially meshed and require the traversing of one or more nodes to go from any given node to every other node in the network.

[0094] Mesh networks may be implemented as wired and/or wireless networks. When implemented in a wireless form, mesh networks can be used to provide voice, data, and video services to geographic areas that do not yet have a wired telecommunications infrastructure. Although existing cellular phone systems may provide limited wireless services in such areas, cellular systems do not adequately deliver broadband services.

[0095] Wireless mesh networks may be particularly valuable in developing countries that do not have a developed traditional telecommunications infrastructure. However, existing wireless mesh solutions are generally designed for developed markets such that they are too expensive for developing countries. Also, existing wireless mesh products typically focus on indoor applications, and are not as suitable for outdoor and roaming applications. [0096] Mesh networks may also be used to provide broadband services to densely populated regions. Examples of such regions may include urban and suburban locations such as housing and apartment developments, university campuses, and research and development and industrial parks. Mesh networks are often used to provide connections for wireless devices in, for example, consumer homes or apartments through the network to a gateway connected to a larger (typically wired) network such as, for example, the Internet. As the number of nodes in such a mesh network increases, and especially in a densely-populated mesh, transmissions from any given node may reach receivers on many other nodes such that the management of interference and routing between such nodes becomes increasingly complex.

[0097] One problem sometimes associated with poor routing management in a wireless mesh network is the introduction of communication delays that undesirably creates so-called "jitter" in voice communications. Another problem sometimes related to routing management is the maintaining of efficient bandwidth utilization, which is much more critical to performance in a wireless, as contrasted to a hard-wired, network. Further increasing the complexity of managing interference and routing is that the operation of each node may not be fully under the control of a central operations center and that all traffic in the network preferably should be secure.

[0098] As used in this exemplary mesh network, an "access point node" means a node in a mesh network that may provide wireless access to at least one end- user device as well as wireless communication with at least one other access point node in the mesh network. As used in this exemplary mesh network, a "gateway node" means a node in a mesh network that provides a communication path between at least two levels in a mesh network.

[0099] This exemplary wireless mesh network is useful in communications such as, for example, wireless broadband communications, and also in methods for configuring and operating the network for data and voice transmissions. The mesh network is typically configured as a multiple-level hierarchical network, with each level of the hierarchy having a plurality of nodes. For example, several clusters of nodes each acting as access points may be organized in numerous first-level meshes, with each of the first-level meshes being coupled to a second-level mesh having several nodes acting as gateways to the first-level meshes. The number of nodes in each of the first-level and second-level meshes may be limited to a predetermined number of nodes in order to improve the performance of the mesh network.

[00100] Figure 5 illustrates a two-level mesh network 100 comprising first-level mesh networks 102, 104, 106, and 108. A mesh network at a particular hierarchical level is sometimes referred to herein as simply a "mesh" for ease of discussion. Each of the first-level meshes comprises a number of nodes configured to communicate with one another. Each first-level mesh may be, for example, a fully-meshed network as in mesh 104 or a partially-meshed network as in mesh 02. Also, some nodes may be connected to only one other node in the first-level mesh.

[00101] The nodes in a first-level mesh typically act as an access point to provide wireless communications from an end-user device 1 10 to mesh network 100. One possible communications use for network 100 is to connect end-user device 1 10 to an external network 1 12, which may be, for example, the Internet or another wired or wireless network. For example, external network 1 12 may be an extension of a business's local area network (LAN) and share security and other attributes of that network facility. Typically, external network 1 12 will have a much higher bandwidth than that available within network 100, and so it is desired that the communications route from end-user device 1 10 to external network 1 12 be as direct as possible.

[00102] Mesh 104 may include a number of access point nodes such as nodes AP1 and AP2. Each access point node may be implemented, for example, using an access point device as described below. An access point node may optionally provide multiple virtual secured access points using, for example, multiple SSID's and supporting, for example, wireless security such as through the use of Wired Equivalent Privacy (WEP) or a Virtual Private Network (VPN).

[00103] Configuration of and routing of data and voice communications within mesh network 100 may be, for example, implemented using the Ad hoc On-Demand Distance Vector (AODV) routing protocol as described in Experimental Request for Comments (RFC) 3561 of the Network Working Group of the Internet Engineering Task Force (IETF) published July 2003, which RFC is incorporated herein by reference. Each first-level and second-level mesh may be configured, for example, as a single flat mesh using AODV. Each first-level and second-level mesh may also be referred to herein as a submesh or cluster. Each submesh/cluster is preferably configured as a subnet. In one approach, this helps ensure that AODV discovery packets do not propagate throughout the so-called "super-mesh." This helps reduce the overall routing overhead, and may be an advantage for some embodiments that results from the use of a hierarchical mesh topology. Certain modifications to AODV that may be useful with mesh network 100 are described further below.

[00104] A second-level mesh includes nodes G1 , G2, G3, G4, and G5, each of which may be configured as a gateway device as described below to act as a gateway node to a first-level mesh. Generally, a gateway node in a second-level mesh will communicate with at least one access point node in a first-level mesh. For example, node G1 is in regular communication with node AP1. A gateway node may be coupled to external network 112. Additional gateway nodes may be coupled to other external networks outside of mesh network 100. It should be noted that, for example, node G1 is a member of both first-level mesh 104 and of the second-level mesh that includes nodes G2, G3, G4 and G5, in that node G1 has at least two radios- one radio communicates with first-level mesh 104 and the other radio communicates with the second-level mesh (these radios are discussed further below).

[00105] Each first-level mesh generally is configured so that one of its nodes is designated as a lead node to couple to the second-level mesh. A back-up node in the first-level mesh is also generally configured to provide an alternate route to the second-level mesh from the first-level mesh in the event that the lead node ceases to communicate with the second-level mesh. For example, in mesh 104, node AP1 may be a lead node, and node AP2 may be a back-up node. Other nodes in mesh 104 may also act as a lead or back-up node.

[00106] End-user device 1 10 may be, for example, a personal computer, a telephone, a personal digital assistant (PDA) or other device that is enabled for wireless communications, and may enable, for example, data or voice communications over mesh network 100. Examples of such communications include data and Voice over Internet Protocol (VoIP) communications. Wireless standards that may be used for implementing wireless communications within mesh network 100 include, for example, the various 802.1 1 standards published by the IEEE such as, for example, the 802.1 b standard, and all such standards are incorporated herein by reference. Typically, an access point node will be deployed for each end-user desiring to use mesh network 100. For example, each access point node may be deployed in an end-user's residence or business location. Specific examples may include residences located in apartment buildings or housing developments or university campuses, and businesses located in research and development, office, or industrial business parks.

[00107] As discussed in more detail below, the radio signal power output of each node in a first-level mesh is controlled so that the node is able to communicate with other nodes in close proximity (e.g., at least one or two other access point nodes in the same first-level mesh), but will not effectively communicate with nodes further away (e.g., in a first-level mesh further away). The irregular circular regions illustrated for each of the first-level meshes 102, 104, 106, and 108 in Figure 5 are intended to graphically illustrate the approximate effective physical reach of the radio signal output from access point nodes in each such mesh. [00108] Mesh network 100 may be used to provide wireless, multi-service (e.g., voice, data, and/or video) access over small or large regions, and for a small or large number of end-users that may access network 100 from time-to-time using one or more end-user devices. The size of the region serviced varies with each installation. Some regions, for example, cover several square kilometers and include thousands of nodes, while other installations may service a single office building. Although network 100 is illustrated in Figure 5 as a two-level network, the basic concept may be extended to three, four or more hierarchical levels. For a larger region, typically a three or four-level network will be used.

[00109] Figure 6 illustrates a mesh network device 200 suitable for use in the mesh network of Figure 5. Device 200 is generally a radio device operating under software/firmware control. Device may be configured for use, for example, as either an access point node or a gateway node. This configuration may be software-selectable in device 200. In a typical installation, each device 200 performs meshing and routing in mesh network 100 or other mesh networks under the control of software executing on device 200. The use of device 200 may permit the incremental building of a large mesh network by adding a few such devices at a time. Each added device may automatically configure itself to join the network as described in more detail below.

[00110] Device 200 typically connects one or more end-user devices within radio range to the mesh network and may communicate with other mesh network radio devices located at other nodes of the mesh network. In some embodiments, all nodes in the mesh network, or in one or more individual meshes of the network, or for all access point nodes used in the network, or for all gateway nodes in the network, may use the same or substantially the same hardware of device 200. The different operational roles of device 200 such as access point node or gateway node is typically software controlled and programmable. Further, the software may be updated as wireless communication standards change and evolve. [00111] Typically, device 200 is programmed to communicate with other nodes that are in close proximity to device 200 such as, for example, where such close proximity is based on closeness in physical distance between nodes or based on the radio signal strength from other nodes as received by device 200. Close proximity may also be based on the hierarchical relationship of a given node relative to other nodes.

[00112] More specifically, device 200 includes radios 204. Generally, device 200 includes at least two radios Radio 1 and Radio 2. Additional radios may included in other embodiments. Radios 204 may be coupled to processor 206, which may be coupled to volatile memory 208 and non-volatile memory 210. Radios 204 may operate simultaneously to support connections to two or more wireless meshes and/or end-user devices. Processor 206 may be, for example, an ARM- based network processor or an XSCALE processor from Intel Corporation. The software running on processor 206 may be, for example, a Linux operating system and include additional software layers, which may vary for different implementations, all the way up to the application level.

[00113] Non-volatile memory 210 (e.g., 32 MB flash memory) may be used to store software/firmware that is used to operate processor 206. Volatile memory 208 (e.g., 128 MB SDRAM) may be used to store, for example, a routing table used by device 200 during communications and interaction with mesh network 100. Radios 204 typically are controlled by processor 206. However, in some embodiments, software may be stored on one or more of radios 204 and/or radios 204 be configured to perform some operations independently of processor 206. [00114] It should be noted that radios 204 generally may be able to operate in several different frequency spectrums. For example, Radio 1 in an access point device may communicate with end-user devices on one of the 2.4GHz (WiFi "b/g band") channels. Radio 1 of each access point node in the same first-level mesh may include, for example, an auto-select feature to select different channels for communications with end-user devices. This auto-select feature may be provided, for example, by the medium access control (MAC) layer in the IEEE 802.1 1 standard. Other algorithms may be used to select the channel for communication with end-user devices. For example, in one embodiment an algorithm may use the distance of the node from the nearest gateway device that has a wired connection to the Internet as a selection parameter, with each access point device using a different channel based on the nodes ordered distance from the Internet (e.g., the closest node uses a first channel, the next- closest node uses a second channel.

[00115] Radio 2 in an access point device may communicate with other nodes in a first-level mesh on one of the 5.8 GHz (WiFi "a band") channels. Radio 2 of each access point node in the same mesh typically communicates with each other on the same channel. The use of two radios as described above typically assists in reducing radio interference and allows simultaneous communication of a node with an end user device and with another node in the mesh network, typically reducing delay.

[00116] With respect to gateway nodes, the channel selected by each gateway device for communication with a first-level mesh may be selected automatically by an algorithm that uses the distance of the gateway node from the nearest other gateway node, in the same-level mesh, that has a wired connection to the Internet as a selection parameter, with each gateway device using a different channel based on the gateway node's ordered distance from the Internet (e.g., the closest gateway node uses a first channel, the next-closest gateway node uses a second channel, . . . ).

[001 17] Each of radios 204 may operate as two or more virtual radios so that the gateway node and access node may effectively be incorporated into the same node hardware. In addition, one or more of radios 204 could be incorporated as part of or on the same chip as processor 206.

[00118] Processor 206 may be coupled to a network interface 212, which is, for example, an Ethernet network interface. Device 200 may obtain power for some or all operations through network interface 212. Other power sources such as a battery may also be used. Network interface 212 is typically connected to an Ethernet or other interface at the location of an end-user to, for example, provide connection to a wired network such as a LAN in the end-user's residence or business premises. For example, the Power over Ethernet (PoE) standard may be used to provide power over interface 212.

[00119] Device 200 may be, for example, encased in a weather-resistant shell and placed outdoors on the side of a building such as a residential house. Alternatively, certain of devices 200 in a network may be designed for interior use.

Three and Higher-Level Mesh Networks

[00120] Figure 7 illustrates a three-level mesh network 300. The structure of network 300 may be, for example, implemented similarly as described for Figure 5 above except that an additional third hierarchical level is included in network 300. More specifically, meshes 302, 304, 306, and 308 are first-level meshes, and meshes 310 and 312 are second-level meshes. Gateway nodes G10, G11 , G12, G13, G14 and G15 are configured into a third-level mesh 318, which is coupled to external networks 314 and 316 using gateway nodes G12 and G13 respectively. External networks 314 and 316 may be similar to external network 1 12 as discussed above.

[00121] Similarly as for each first-level mesh in network 100 described above, each second-level mesh 310 and 312 generally has one node designated as a lead node for communication with the next higher level and another node designated as a back-up node. For example, for mesh 310, node G15 may be a lead node of second-level mesh 310, and node G10 may be a back-up node, each capable of communicating with node G12. Similarly as discussed above for node G1 above, it should be noted that, for example, node G15 is a member of and lead node for second-level mesh 310 and also a member of third-level mesh 318. [00122] As discussed above, communication within each first-level mesh occurs between nodes that are in close proximity. In addition, in a preferred mode of operation, all nodes within a first-level mesh operate on the same channel or frequency. For example, within first-level mesh 306 all nodes communicate on a radio channel "A", with the nodes taking turns in using channel A as is implemented, for example, in the 802.11 standard's MAC functionality to arbitrate channel access.

[00123] In addition, network 300 may be configured so that all nodes in adjacent first-level meshes operate on a different channel or frequency. For example, nodes in mesh 304 may communicate on a channel "B", which is expected to reduce interference with communications in mesh 306. Adjacent meshes are typically those meshes that are in closest proximity to a given mesh (and often such adjacent meshes are child meshes at the same hierarchical level in the network under a common parent mesh). [00124] When a mesh is further removed from a first mesh (i.e., is not an adjacent mesh), such a further removed mesh may operate on the same channel as the first mesh. For example, mesh 302 is further removed from mesh 306, and may operate on the same radio channel. Interference is much less likely due to the greater distance between meshes 302 and 306, in contrast to the closer proximity between meshes 304 and 306. In addition, the radio signal power output of each node in meshes 302 and 306 may be controlled as discussed further below so that the effective radio signal range of each of meshes 302 and 306 will not overlap. Typically, meshes 302 and 306 will be sufficiently physically distant such that, with the output transmit power limited within each submesh/cluster as described herein, radio signals from each mesh should not interfere with one another.

[00125] Even higher-level mesh networks, such as four-level networks, may be created by applying the principles described above. A large, multiple-level hierarchical mesh network built as described herein may have a size of, for example, about 40,000 nodes in an area of about twenty square kilometers. In one embodiment, it is preferred that such a large number of nodes be divided into groups of about two to four thousand nodes with a three-level mesh network servicing each such group, and with a fourth-level mesh implemented using wired communications (e.g., the Internet) between the Internet gateways for each of the third-level meshes.

[00126] The nodes at each level of a hierarchical mesh network are typically configured so that they will interact similarly with one another regardless of the particular mesh level in which they are located. For example, second-level meshes 310 and 312, if they are adjacent as discussed above with respect to first-level meshes, may operate on different channels (indicated in Figure 7 as "B" and "C"). Often, there is a greater distance between second and higher-level meshes so that the channel of operation used for operation is less likely to create interference with another mesh of the same level.

[00127] Also, it should be noted that even though the mesh network 300 has been described with reference to access point and gateway nodes, and the exemplary use of device 200 for each of these nodes, in other embodiments, it is not required that every node in a first or other level be an access point device or that every node in a second or higher level be a gateway device. It is possible, for example, that in some embodiments both access point nodes and gateway nodes may be present on a second or third-level mesh. Also, some or all of the gateway nodes might additionally act as an access point node.

Control of Radio Channel and Signal Power

[00128] As mentioned above, the radio signal power output of each node in a first or other level mesh may be controlled by device 200 so that the nodes within a given mesh are able to effectively communicate, but so that interference is not created with communications in other meshes, and in particular other meshes in close proximity. Software running on device 200 may be used to control the power output, and the software may be stored, for example, in nonvolatile memory 210. [00129] For example, software executed by device 200, when operating as a gateway device, may include instructions to control the radio signal power output of Radio 1 by its first scanning for other wireless devices operating in the neighborhood of the Radio 1 , and then based on information obtained from this scan, to adjust the radio signal power output of Radio 1 to a level sufficient for device 200 to communicate (e.g., with sufficient reliability and/or bandwidth) with at least one node on a higher-level mesh and/or another node in the same-level mesh. As also mentioned above, device 200 may control the selection of radio channel to use.

[00130] In one approach, each node is capable of scanning its neighborhood (i.e., other nearby nodes in the mesh network) to determine/locate the neighboring nodes while using various communications speeds. First, based on the initial gathered information, a cluster is formed and a lead node is selected. The gathered information may also be communicated to a central node (e.g., a mesh network management system, which is located external to the mesh network) for enhancement of the configuration. The central node may refine/reconfigure each of the clusters based on a few selected criteria (e.g., the received signal strength indicator (RSSI) for various communication speeds, the number of nodes, the number of hops within the cluster, and/or the accessibility of the lead and back-up nodes (primary and secondary) to the next higher level of hierarchy in the mesh network).

[00131 ] From the foregoing neighborhood scanning, there are typically two groups of nodes: a first group of nodes that have a high RSSI at even the highest communication speeds, and a second group of nodes with a rather low/moderate RSSI at even the lowest communication speeds. The output power of the first group of nodes may be adjusted such that they are well-heard in their respective clusters. The second group of nodes may each be configured to run at the lowest communication speed, and preferably at an output power sufficiently high so that each node in the second group may be heard by two or more nodes in a nearby cluster. Radio and Wireless Standards

[00132] In order to provide wireless broadband access for both voice and data, a preferred RF standard, which is economic and fairly mature technology, is the 802.1 1 standard. The protocols, security, QoS, mesh routing protocols, and other aspects of the 802.11 standard are substantially well-defined. One shortcoming of the 802.1 1 standard is its reach and noise (e.g., due to the usage of an unlicensed RF band).

[00133] For certain applications (e.g., dense residential), the reach is not a significant issue. However, nodes that aggregate traffic (e.g., back-bone nodes) may require larger coverage, in which case, for example, directional antennas and/or PA's (power amplifiers) with higher gain may be used. Also, other proprietary techniques may be used. Alternatively, to combat coverage and noise shortcomings, other standards (e.g., WiMax) may be used to carry the back-bone traffic. It is preferred that, if used, that the selected hardware implementing the WiMax standard be mature and affordable. Also, the mesh network described herein is preferably RF agnostic so that, with reduced effort, the mesh network can be implemented with yet other RF standards and solutions.

Self-Configuring Nodes

[00 34] The software used to control device 200 may provide a self-configuring capability so that the device senses other devices in an existing mesh network and configures its routing, security, and other characteristics so that it may connect to and begin operation within the mesh network without the need for initial external interaction between device 200 and a person or computer outside of the existing mesh network. A node typically only needs one link to successfully join the mesh network, but two or more links are usually preferred. Also, nodes may in general be added on the edge or in the middle of an existing mesh network. [00135] More specifically, the access point devices already in, or to be added to, a mesh network may use a mesh-joining algorithm to form first-level meshes. Each access point device at its start-up may probe each channel used to communicate with other access point devices and record, for example, in a table in volatile memory 208, the signal strength reception for each channel and neighboring node identified. The algorithm may then direct the access point device to join or cluster with a number of the closest nodes identified into a first- level mesh (but typically as limited to the maximum total number of nodes permitted within a single first-level mesh, as discussed further below). Also, it is preferred that each device 200, when it will be used in mesh network 300, be configured (e.g., prior to its joining the network) as to whether the device will operate as an access point node or as a gateway node.

[00136] Each cluster in the mesh network will preferably have a lead node and a back-up node. The lead node is active, and the back-up node is generally in a standby mode. Both the lead and the back-up nodes (also referred to herein as primary and secondary nodes) are considered as members of the next higher level of hierarchy (e.g., the lead and back-up nodes in a first-level mesh are also considered to be and configured for actual communication, or potential communication in the case of a back-up node, as nodes within the second-level mesh to which they couple the first-level mesh). In typical operation, the back-up node will be visible by the higher-level mesh, but will not be actively communicating with the higher-level mesh. The back-up node should normally itself store the same set routing and other active connections information stored in the lead node so that, in case of failure of the lead node, the back-up node may assume the role of the failed lead node. It should be noted that the failed lead node, when and if it is restored to being again functional, may then assume the role of a back-up node and remain in a standby mode. Clustering of Nodes in a Hierarchical Mesh Network

[00137] When building a mesh network as described herein, in a preferred embodiment, the maximum number of nodes permitted in any given first or other level mesh is limited to a predetermined total number of nodes. This predetermined number may be established, for example, by performance simulation done for a particular mesh network installation.

[00138] It has been found in general that this predetermined total number is preferably no more than about 50 nodes, and even more preferably no more than 32 nodes for each level. Also, it has been found that for higher levels in the mesh, the maximum total number of nodes may be preferably less than for the first-level meshes. For example, each node in a mesh network may be programmed (e.g., by software running on each node) so that a mesh network configures itself (as mentioned above) with each first-level mesh having 32 or fewer nodes, each second-level mesh having sixteen or fewer nodes, and each third-level mesh having eight or fewer nodes.

[00139] Nodes may self-configure into a hierarchical mesh network by in general scanning for other nodes that are nearest or closest to a given node, which will store information in its routing table to record an identifier of the closest nodes along with the relative signal strengths received. This set of nodes is then bisected incrementally until the total number of nodes is less than the predetermined total number of nodes (e.g., 32 total nodes) permitted in a cluster for a given level.

Mesh Network Routing Protocols

[00140] Routing protocols that govern the routing of communications (e.g., data packets) from each node in the mesh network may be implemented, for example, in software stored in non-volatile memory 210 and executed by processor 206 of mesh network device 200. For example, the AODV routing protocol mentioned above may be used. Other embodiments may use other routing protocols. It should be noted that the routing protocols used may be independent of the radio standards used for wireless communications between nodes and with end-user devices.

[00141] Each node in a mesh network may typically access a routing table to determine the routes for communications from the node to gateways to external networks (e.g., Internet gateways). This routing table may be stored, for example, in volatile memory 208 and/or non-volatile memory 210. The table entries include the path (nodes) that its packets must traverse to get to that destination.

[00142] Device 200 may select routes for a node based on one or more criteria such as, for example, the route that will incur the least number of hops from the node to external network 1 12 (e.g., the Internet), the quality of the route connection to external network 1 12, the bandwidth availability along the route, and/or the priority of the end-user device accessing the node.

[00143] In one approach, each source node initially sends discovery packets into its immediate submesh to the immediate nodes (i.e., neighbors) connected to the source node. The immediate nodes forward the discovery packets to their respective immediate nodes. Each node maintains information about the discovery packets. If a node receives the discovery packet again, the packet will not be forwarded. Finally, typically multiple copies of the discovery packets will reach a given destination node in the mesh network via different routes.

[00144] Each copy of the discovery packet carries information such as, for example, the number of hops from the source node to the destination node, and the load and RSSI on each link. Through the use of the "flooding" method above, the destination node typically learns information about multiple routes to the destination node from the source node. The destination node uses this information to select two of the routes (a primary and a secondary route) and communicates the selected routes back to the source node. The selected routes are stored in the routing tables of the source node. Preferably, each node in the mesh network will know the location of a default gateway to an external network. The routing protocol used above may also include a fairness algorithm so that users of end-user devices will have balanced access to the available bandwidth.

[00145] An access point node may implement mesh routing on one of its radios, and the other radio may be the interface to the end-user device in which the radio may examine packet information to implement services all the way up to the application layer (e.g., examination of packet information for security applications). The gateway node may implement mesh routing on both interfaces. Typically, the gateway node will only examine a packet to the extent necessary to determine routing information. [00146] It should also be noted that in some embodiments the mesh network described herein may implement a multicasting technique. In more detail, traffic in the mesh network may be branched off at a node closest to the end nodes that are part of the same multicast group.

Mesh Network Routing: Flow-Based Caching

[00147] In the operation of a mesh network as described herein, nodes in the network may be configured to direct, for example, voice traffic leaving the node by using the same route. This is typically desirable when directing voice traffic so that delay and delay jitter and any other problems with voice quality and connection for a given user session are minimized. [00148] Each communications session typically involves a session identifier or ID. Each node may be configured so that it stores these session identifiers and corresponding routes in a route table, and packets associated with the session identifier are directed from the node using a common route. The route table may be stored in, for example, volatile memory 208. [00149] Also, each node may be optionally configured so that an aging mechanism is applied to the routes in the route table. For example, software running on device 200 may invalidate a route if it has been inactive for a predetermined period of time. This aging mechanism is usually configured so that it will not invalidate a route while a session is active.

Mesh Network Routing: Detection of Node Failure

[00150] In some embodiments, one or more nodes in a mesh network may be configured to sense the presence of a failed node in another portion of the network (e.g., within the same first or other level mesh or in a different mesh). In a first approach, it may be done actively, such as by sending keep alive packets or signals (known as so-called "keep alives") to the other nodes and awaiting a response. The keep alives may be send, for example, to the immediate neighbors of the node, or to the default or otherwise defined external network gateway for the node.

[00151] In another approach, the sending of keep alives may be done passively, such as when a node sends a data packet to another node and awaits an acknowledgement. If no acknowledgement is received within a configurable time period, the packet is re-transmitted. This may be tried a number of times. If no acknowledgement packet is received after a certain number of re-tries, then the route is then considered to be down. Using either method above, each node may be programmed to periodically check for other nodes that have failed.

[00152] Upon detection of a failed node, the detecting node may communicate information relating to the failure to a source node located elsewhere in the mesh network. The source node(s) that include the failed node in at least one of the routes stored in a routing table used by the source node(s) will typically receive this information.

[00153] Upon receiving notification of a failed node, each source node may update its routing table to select an alternate route. The alternate route may already be stored in the route table, or the source node may explore the mesh network for a new route (e.g., using AODV). This approach may also be used to handle changes in node devices as part of reconfiguring the network as changes in network topology are implemented, or to respond to other link breakages. Also, as mentioned above, in the event of the failure of a lead node for a given mesh, in a preferred approach, any other node in that mesh could assume the role of lead node.

Load Balancing

[00154] As traffic on a mesh network increases, it is typically desirable to add additional connection points to external networks such as external networks 314 and 316 in Figure 7. Traffic through the mesh network may be load balanced across these external gateways. Further, the nodes in the mesh network may be configured (e.g., using software included on device 200) so that there is an automatic fail-over mechanism in case an external gateway (e.g., an Internet gateway) fails due to loss of power, gateway failure, or otherwise. Also, as new external gateways are connected, the nodes in the mesh network may adjust their configuration so that traffic load is redistributed and the number of hops from each node to the external gateway is minimized. [00155] Also, static load balancing may be implemented. For example, each node may store a table of gateways to external networks (e.g., gateways to wired networks). Through the use of network management/configuration, the gateway addresses may be communicated to each first-level node and to other nodes acting as access points (collectively referred to as "end nodes"). The table entries may have an order associated to them. For example, the end node may always use a designated primary gateway number to communicate with the outside world. It also may use the keep alive approach described above to sense if this primary gateway is down. If the primary gateway fails, then the node may switch to a secondary gateway stored in the table. The order assignment of the table entries may be done by, for example, an off-line algorithm.

Traffic Management

[00156] Several traffic management features may be implemented in the mesh network described above. For example, this may include a policing feature in which each end user has a defined service level (e.g., 256 Kbps). The traffic for the user may be monitored to ensure that the limit is not violated. This policing may also ensure that the user always obtains any guaranteed minimum level of services. [00157] Traffic management may also include priority routing in which traffic is classified, for example, into four priority classes (e.g., voice, guaranteed data, best efforts data, and other). Each node may have, for example, four priority queues at each of its interfaces with each traffic class queued into its their respective output queue. The queues may be serviced (i.e., packets transmitted) based on a configurable scheduling mechanism (e.g., round-robin, weighted round-robin, preemptive, non-preemptive).

[00158] Another aspect of traffic management may include congestion avoidance in which, by a node's sensing the load in portions of the mesh network, a connection is created by a node only when there is sufficient bandwidth, for example, to ensure a quality connection. Load may be sensed, for example, by examining the length of the queues at a node. Also, each node may be configured to sense the network load and throttle back the best-effort traffic so that higher priority traffic may pass through.

[00159] Traffic management may additionally include, for voice traffic within each cluster, a manner of operation in which nodes will transmit voice in an orderly fashion (e.g., the sending of voice traffic in preference to data traffic). One benefit of this may be the reduction of jitter. Also, to further reduce jitter, buffering of voice traffic at one or more nodes may be used to smooth out the voice delivery to the end user. Closed Loop Optimization

[00160] When planning is started for a new network or for new nodes/elements (e.g., router, voice gateways, etc.) that may be introduced into the network, it is may be desirable to simulate and predict the effect of the changes. When dealing with a lot of variables in a mesh network (e.g., ranging from the radio frequency issues in the environment to human behavioral issues), closed loop optimization and planning may provide significant advantages. The closed loop optimization approach provides the network operator/manager the ability to make intelligent decisions regarding the mesh network. The visibility into the network through such analysis may assist the network operator to save time and cost in overseeing and managing the network.

[00161] Closed loop optimization may be implemented in several basic steps (e.g., track, analyze, model, decide, and act). To track is to collect the vitals of the existing network. Through off-line simulation the vitals are fed to the network model to see the shortcoming of the existing network architecture/topology. Once the short-comings are identified, then a new model is generated that suggests where improvements can be made. Then, a decision whether to implement the improvements is made and acted upon.

[00162] Decisions whether to implement an improvement may be made, for example, by an Internet service provider (ISP) manager. The decision is often typically made based on the severity of the problem in terms of performance and the cost of alternative solutions to improve performance and robustness. By reconfiguring the mesh network using the above approach, there may be no need for additional new equipment, which would reduce cost. [00163] Since a large mesh network is typically a dynamic network environment (e.g., in which users are added and removed from the network, the users each have different behaviors, and the behaviors typically change over time), it is preferred that there be a dynamic/automatic way to tune the network. Typically, the network is being constantly monitored and analyzed by dynamic activity. The use of the analysis tool described above assists the network operator in making intelligent operational decisions.

Roaming

[00164] Roaming may be supported by the mesh network (e.g., low-speed roaming). For example, a user may move around with a wireless device (e.g., a WiFi phone, or a PDA) and remain connected to the network. As they are moving away from a first access point (as determined, e.g., by checking the applicable RSSI), a new access point may be detected and the connection (e.g., destination IP address) and security (e.g., assignment of a new key), and other content may be either forwarded to or negotiated with a next access point.

[00165] In the foregoing specification, the disclosure has been described with reference to specific exemplary embodiments thereof. It will be evident that various modifications may be made thereto without departing from the broader spirit and scope as set forth in the following claims. The specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense.

Claims

CLAIMS What is claimed is:

1. A method for client load balancing, implemented in a wireless hierarchical mesh network, the method comprising: grouping a plurality of nodes in the network; broadcasting a load information message (LIM) from each of the nodes to provide a plurality of LIMs, each LIM comprising a node-to-stations mapping for stations detected by each respective node; computing a station-to-nodes mapping for each respective node using data

collected from the plurality of LIMs; and determining a station-to-node assignment using rank and signal strength

information obtained from processing the collected data.

2. A method for power management, implemented in a wireless hierarchical mesh network, the method comprising: generating a signal vector for each node of the mesh network by using radio

calibration, each respective signal vector being generated for each of a plurality of power levels of its respective node; using a metric to measure radio interference in the mesh in order to determine a plurality of interference costs; selecting a power vector based on the plurality of interference costs in order to reduce interference; and sending the power vector to at least one node in the network.

3. A method for client mobility, implemented in a wireless hierarchical mesh network, to connect a client to a first node, the method comprising: obtaining, by the first node, a MAC address of the client during association; deriving, from the MAC address, the client's IP address and the IP address of a home node for the client using a hash function; and installing a tunnel from the first node to the home node.

4. The method of claim 3, further comprising: requesting, by the first node, that the home node set up a destination IP address- based route in its routing table so that packets addressed to the client and received at the home node will be forwarded to the first node over the tunnel; and adding a source IP address-based route to the routing table of the first node so that packets received from the client by the first node will be routed from the first node to the home node.