CA2159269A1 - Method and Apparatus for Achieving Uniform Data Distribution in a Parallel Database System - Google Patents

Method and Apparatus for Achieving Uniform Data Distribution in a Parallel Database System

Info

Publication number
CA2159269A1
CA2159269A1 CA2159269A CA2159269A CA2159269A1 CA 2159269 A1 CA2159269 A1 CA 2159269A1 CA 2159269 A CA2159269 A CA 2159269A CA 2159269 A CA2159269 A CA 2159269A CA 2159269 A1 CA2159269 A1 CA 2159269A1
Authority
CA
Canada
Prior art keywords
nodes
data
database system
resource loading
parallel database
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CA2159269A
Other languages
French (fr)
Other versions
CA2159269C (en
Inventor
Chaitanya K. Baru
Fred Koo
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
IBM Canada Ltd
Original Assignee
IBM Canada Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by IBM Canada Ltd filed Critical IBM Canada Ltd
Priority to CA002159269A priority Critical patent/CA2159269C/en
Priority to US08/665,031 priority patent/US5970495A/en
Publication of CA2159269A1 publication Critical patent/CA2159269A1/en
Application granted granted Critical
Publication of CA2159269C publication Critical patent/CA2159269C/en
Anticipated expiration legal-status Critical
Expired - Fee Related legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5083—Techniques for rebalancing the load in a distributed system
    • G06F9/5088—Techniques for rebalancing the load in a distributed system involving task migration
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00—Data processing: database and file management or data structures
    • Y10S707/99941—Database schema or data structure
    • Y10S707/99943—Generating database or data structure, e.g. via user interface

Abstract

The invention provides a method and apparatus for distributing data of a table substantially uniformly across a parallel database system having a plurality of interlinked database nodes.
Data of the table is distributed across a group of nodes (nodegroup) in accordance with a partitioning arrangement.
Resource loading, for example, the workload or storage volume of the nodes is monitored. Data is moved from one or more nodes having higher resource loading to nodes having lower resource loading to achieve a substantially uniform distribution of the resource loading across the group of nodes concerned. In the course of moving data the selection of groups of data to be moved is performed in a manner to reduce the amount of data movement.
CA002159269A 1995-09-27 1995-09-27 Method and apparatus for achieving uniform data distribution in a parallel database system Expired - Fee Related CA2159269C (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CA002159269A CA2159269C (en) 1995-09-27 1995-09-27 Method and apparatus for achieving uniform data distribution in a parallel database system
US08/665,031 US5970495A (en) 1995-09-27 1996-06-10 Method and apparatus for achieving uniform data distribution in a parallel database system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CA002159269A CA2159269C (en) 1995-09-27 1995-09-27 Method and apparatus for achieving uniform data distribution in a parallel database system

Publications (2)

Publication Number Publication Date
CA2159269A1 true CA2159269A1 (en) 1997-03-28
CA2159269C CA2159269C (en) 2000-11-21

Family

ID=4156665

Family Applications (1)

Application Number Title Priority Date Filing Date
CA002159269A Expired - Fee Related CA2159269C (en) 1995-09-27 1995-09-27 Method and apparatus for achieving uniform data distribution in a parallel database system

Country Status (2)

Country Link
US (1) US5970495A (en)
CA (1) CA2159269C (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5970495A (en) * 1995-09-27 1999-10-19 International Business Machines Corporation Method and apparatus for achieving uniform data distribution in a parallel database system
US6125370A (en) * 1998-04-01 2000-09-26 International Business Machines Corporation Repartitioning data
US6269375B1 (en) 1998-04-01 2001-07-31 International Business Machines Corporation Rebalancing partitioned data

Families Citing this family (95)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3510042B2 (en) * 1996-04-26 2004-03-22 株式会社日立製作所 Database management method and system
US6496823B2 (en) * 1997-11-07 2002-12-17 International Business Machines Corporation Apportioning a work unit to execute in parallel in a heterogeneous environment
US6363396B1 (en) * 1998-12-21 2002-03-26 Oracle Corporation Object hashing with incremental changes
US6691166B1 (en) * 1999-01-07 2004-02-10 Sun Microsystems, Inc. System and method for transferring partitioned data sets over multiple threads
US6542854B2 (en) 1999-04-30 2003-04-01 Oracle Corporation Method and mechanism for profiling a system
US6549931B1 (en) 1999-09-27 2003-04-15 Oracle Corporation Distributing workload between resources used to access data
US6609131B1 (en) 1999-09-27 2003-08-19 Oracle International Corporation Parallel partition-wise joins
US6665684B2 (en) * 1999-09-27 2003-12-16 Oracle International Corporation Partition pruning with composite partitioning
US6470331B1 (en) * 1999-12-04 2002-10-22 Ncr Corporation Very large table reduction in parallel processing database systems
US7246120B2 (en) 2000-01-28 2007-07-17 Oracle International Corporation Techniques for achieving higher availability of resources during reconfiguration of a cluster
US6751616B1 (en) 2000-01-28 2004-06-15 Oracle International Corp. Techniques for DLM optimization with re-mapping responsibility for lock management
US6920454B1 (en) 2000-01-28 2005-07-19 Oracle International Corporation Techniques for DLM optimization with transferring lock information
US6529906B1 (en) 2000-01-28 2003-03-04 Oracle Corporation Techniques for DLM optimization with re-mastering events
US6523036B1 (en) * 2000-08-01 2003-02-18 Dantz Development Corporation Internet database system
CA2319918A1 (en) * 2000-09-18 2002-03-18 Linmor Technologies Inc. High performance relational database management system
US6944607B1 (en) * 2000-10-04 2005-09-13 Hewlett-Packard Development Compnay, L.P. Aggregated clustering method and system
IL141599A0 (en) * 2001-02-22 2002-03-10 Infocyclone Inc Information retrieval system
JP4232357B2 (en) * 2001-06-14 2009-03-04 株式会社日立製作所 Computer system
US7028054B2 (en) * 2001-07-02 2006-04-11 International Business Machines Corporation Random sampling as a built-in function for database administration and replication
US7024401B2 (en) * 2001-07-02 2006-04-04 International Business Machines Corporation Partition boundary determination using random sampling on very large databases
US6801903B2 (en) * 2001-10-12 2004-10-05 Ncr Corporation Collecting statistics in a database system
US20030158842A1 (en) * 2002-02-21 2003-08-21 Eliezer Levy Adaptive acceleration of retrieval queries
US7346690B1 (en) 2002-05-07 2008-03-18 Oracle International Corporation Deferred piggybacked messaging mechanism for session reuse
US20040003022A1 (en) * 2002-06-27 2004-01-01 International Business Machines Corporation Method and system for using modulo arithmetic to distribute processing over multiple processors
US20040006622A1 (en) * 2002-07-03 2004-01-08 Burkes Don L. Optimized process for balancing load for data mirroring
US7020661B1 (en) * 2002-07-10 2006-03-28 Oracle International Corporation Techniques for pruning a data object during operations that join multiple data objects
US7778996B2 (en) * 2002-09-25 2010-08-17 Teradata Us, Inc. Sampling statistics in a database system
US7797450B2 (en) * 2002-10-04 2010-09-14 Oracle International Corporation Techniques for managing interaction of web services and applications
US7293024B2 (en) * 2002-11-14 2007-11-06 Seisint, Inc. Method for sorting and distributing data among a plurality of nodes
US7657540B1 (en) 2003-02-04 2010-02-02 Seisint, Inc. Method and system for linking and delinking data records
US7447786B2 (en) 2003-05-09 2008-11-04 Oracle International Corporation Efficient locking of shared data that is accessed for reads in a cluster database
US20040260745A1 (en) * 2003-06-18 2004-12-23 Gage Christopher A. S. Load balancer performance using affinity modification
JP4330941B2 (en) 2003-06-30 2009-09-16 株式会社日立製作所 Database divided storage management apparatus, method and program
US7379952B2 (en) * 2004-01-30 2008-05-27 Oracle International Corporation Techniques for multiple window resource remastering among nodes of a cluster
EP1626339B1 (en) * 2004-08-13 2016-02-24 Sap Se Data processing system and method for assigning objects to processing units
US20060200469A1 (en) * 2005-03-02 2006-09-07 Lakshminarayanan Chidambaran Global session identifiers in a multi-node system
US7209990B2 (en) * 2005-04-05 2007-04-24 Oracle International Corporation Maintain fairness of resource allocation in a multi-node environment
US7366716B2 (en) * 2005-05-06 2008-04-29 Microsoft Corporation Integrating vertical partitioning into physical database design
US7493400B2 (en) 2005-05-18 2009-02-17 Oracle International Corporation Creating and dissolving affinity relationships in a cluster
US8037169B2 (en) * 2005-05-18 2011-10-11 Oracle International Corporation Determining affinity in a cluster
US7539661B2 (en) * 2005-06-02 2009-05-26 Delphi Technologies, Inc. Table look-up method with adaptive hashing
US8326990B1 (en) * 2005-07-15 2012-12-04 Symantec Operating Corporation Automated optimal workload balancing during failover in share-nothing database systems
US7814065B2 (en) * 2005-08-16 2010-10-12 Oracle International Corporation Affinity-based recovery/failover in a cluster environment
US8027684B2 (en) * 2005-08-22 2011-09-27 Infosys Technologies, Ltd. System for performing a task in a communication network and methods thereof
US20070162506A1 (en) * 2006-01-12 2007-07-12 International Business Machines Corporation Method and system for performing a redistribute transparently in a multi-node system
US8005836B2 (en) * 2006-01-13 2011-08-23 Teradata Us, Inc. Method and system for performing logical partial declustering
JP2007249468A (en) * 2006-03-15 2007-09-27 Hitachi Ltd Cpu allocation method, cpu allocation program, cpu allocation device and database management system
US7921416B2 (en) * 2006-10-20 2011-04-05 Yahoo! Inc. Formal language and translator for parallel processing of data
US7698529B2 (en) * 2007-01-10 2010-04-13 International Business Machines Corporation Method for trading resources between partitions of a data processing system
US20080168077A1 (en) * 2007-01-10 2008-07-10 Eric Lawrence Barsness Pre-loading of an in memory database
US7769732B2 (en) * 2007-08-27 2010-08-03 International Business Machines Corporation Apparatus and method for streamlining index updates in a shared-nothing architecture
US8892558B2 (en) 2007-09-26 2014-11-18 International Business Machines Corporation Inserting data into an in-memory distributed nodal database
US8027996B2 (en) * 2007-11-29 2011-09-27 International Business Machines Corporation Commitment control for less than an entire record in an in-memory database in a parallel computer system
US8209334B1 (en) * 2007-12-28 2012-06-26 Don Doerner Method to direct data to a specific one of several repositories
US8266168B2 (en) 2008-04-24 2012-09-11 Lexisnexis Risk & Information Analytics Group Inc. Database systems and methods for linking records and entity representations with sufficiently high confidence
US20090307329A1 (en) * 2008-06-06 2009-12-10 Chris Olston Adaptive file placement in a distributed file system
US8661026B2 (en) * 2008-07-02 2014-02-25 Lexisnexis Risk Solutions Fl Inc. Entity representation identification using entity representation level information
US7774311B2 (en) * 2008-07-17 2010-08-10 International Business Machines Corporation Method and apparatus of distributing data in partioned databases operating on a shared-nothing architecture
US9996572B2 (en) 2008-10-24 2018-06-12 Microsoft Technology Licensing, Llc Partition management in a partitioned, scalable, and available structured storage
US9244793B1 (en) 2008-11-04 2016-01-26 Teradata Us, Inc. Using target database system statistics in emulation
EP2372568A4 (en) * 2008-12-24 2016-03-30 Fujitsu Ltd Configuration management system, proxy system, and configuration management method
US8078825B2 (en) * 2009-03-11 2011-12-13 Oracle America, Inc. Composite hash and list partitioning of database tables
US9251212B2 (en) * 2009-03-27 2016-02-02 Business Objects Software Ltd. Profiling in a massive parallel processing environment
US9411859B2 (en) 2009-12-14 2016-08-09 Lexisnexis Risk Solutions Fl Inc External linking based on hierarchical level weightings
US9171044B2 (en) * 2010-02-16 2015-10-27 Oracle International Corporation Method and system for parallelizing database requests
US8849749B2 (en) * 2010-05-14 2014-09-30 Oracle International Corporation Load balancing in parallel database systems using multi-reordering
US8768973B2 (en) * 2010-05-26 2014-07-01 Pivotal Software, Inc. Apparatus and method for expanding a shared-nothing system
US9189505B2 (en) 2010-08-09 2015-11-17 Lexisnexis Risk Data Management, Inc. System of and method for entity representation splitting without the need for human interaction
US8583687B1 (en) * 2012-05-15 2013-11-12 Algebraix Data Corporation Systems and methods for indirect algebraic partitioning
US8903876B2 (en) 2012-08-15 2014-12-02 Facebook, Inc. File storage system based on coordinated exhaustible and non-exhaustible storage
US9229657B1 (en) 2012-11-01 2016-01-05 Quantcast Corporation Redistributing data in a distributed storage system based on attributes of the data
US9811529B1 (en) * 2013-02-06 2017-11-07 Quantcast Corporation Automatically redistributing data of multiple file systems in a distributed storage system
US9792295B1 (en) 2013-02-06 2017-10-17 Quantcast Corporation Distributing data of multiple logically independent file systems in distributed storage systems including physically partitioned disks
AU2013381504B2 (en) * 2013-03-12 2016-06-23 Kabushiki Kaisha Toshiba Database system, program, and data processing method
CN103336792B (en) * 2013-06-07 2016-11-23 华为技术有限公司 Data partition method and device
JP6382819B2 (en) * 2013-08-21 2018-08-29 株式会社東芝 Database system, node, management apparatus, program, and data processing method
WO2015029139A1 (en) * 2013-08-27 2015-03-05 株式会社東芝 Database system, program, and data processing method
US9934323B2 (en) * 2013-10-01 2018-04-03 Facebook, Inc. Systems and methods for dynamic mapping for locality and balance
US9372907B2 (en) * 2013-11-26 2016-06-21 Sap Se Table placement in distributed databases
US9830346B2 (en) 2013-11-26 2017-11-28 Sap Se Table redistribution in distributed databases
JP2015170101A (en) * 2014-03-06 2015-09-28 富士通株式会社 biometric authentication device, method and program
CN105205052B (en) * 2014-05-30 2019-01-25 华为技术有限公司 A kind of data digging method and device
US9860316B2 (en) 2014-09-19 2018-01-02 Facebook, Inc. Routing network traffic based on social information
US10558637B2 (en) * 2015-12-17 2020-02-11 Sap Se Modularized data distribution plan generation
CN105959419A (en) * 2016-07-15 2016-09-21 浪潮(北京)电子信息产业有限公司 Establishment method and system for distributed storage structure based on consensus tree
KR101951999B1 (en) 2016-08-31 2019-05-10 재단법인대구경북과학기술원 Storage system and storing method of relational database for high query performance with low data redundancy and processing method of query based on storing method of relational database
US10534765B2 (en) * 2017-04-07 2020-01-14 Micro Focus Llc Assigning segments of a shared database storage to nodes
US20180322397A1 (en) * 2017-05-08 2018-11-08 International Business Machines Corporation Detecting case families with anomalous frequencies in rule-based decision policies
US10459810B2 (en) 2017-07-06 2019-10-29 Oracle International Corporation Technique for higher availability in a multi-node system using replicated lock information to determine a set of data blocks for recovery
US11163764B2 (en) 2018-06-01 2021-11-02 International Business Machines Corporation Predictive data distribution for parallel databases to optimize storage and query performance
US11157496B2 (en) 2018-06-01 2021-10-26 International Business Machines Corporation Predictive data distribution for parallel databases to optimize storage and query performance
US10949254B2 (en) * 2018-09-28 2021-03-16 Atlassian Pty Ltd. Systems and methods for scheduling tasks
CN110196882B (en) * 2019-05-27 2021-06-08 上海达梦数据库有限公司 Method and device for determining data redistribution mode, server and storage medium
US11526500B2 (en) * 2019-12-12 2022-12-13 Sap Se System and method for initiating bulk inserts in a distributed database
CN111274028B (en) * 2020-01-15 2023-09-05 新方正控股发展有限责任公司 Partitioning method, partitioning device and readable storage medium based on database middleware

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4403286A (en) * 1981-03-06 1983-09-06 International Business Machines Corporation Balancing data-processing work loads
US4630264A (en) * 1984-09-21 1986-12-16 Wah Benjamin W Efficient contention-resolution protocol for local multiaccess networks
US5117350A (en) * 1988-12-15 1992-05-26 Flashpoint Computer Corporation Memory address mechanism in a distributed memory architecture
US5625836A (en) * 1990-11-13 1997-04-29 International Business Machines Corporation SIMD/MIMD processing memory element (PME)
US5325525A (en) * 1991-04-04 1994-06-28 Hewlett-Packard Company Method of automatically controlling the allocation of resources of a parallel processor computer system by calculating a minimum execution time of a task and scheduling subtasks against resources to execute the task in the minimum time
US5555404A (en) * 1992-03-17 1996-09-10 Telenor As Continuously available database server having multiple groups of nodes with minimum intersecting sets of database fragment replicas
US5390283A (en) * 1992-10-23 1995-02-14 North American Philips Corporation Method for optimizing the configuration of a pick and place machine
US5625811A (en) * 1994-10-31 1997-04-29 International Business Machines Corporation Method and system for database load balancing
US5687372A (en) * 1995-06-07 1997-11-11 Tandem Computers, Inc. Customer information control system and method in a loosely coupled parallel processing environment
CA2159269C (en) * 1995-09-27 2000-11-21 Chaitanya K. Baru Method and apparatus for achieving uniform data distribution in a parallel database system
US5758345A (en) * 1995-11-08 1998-05-26 International Business Machines Corporation Program and method for establishing a physical database layout on a distributed processor system
US5764905A (en) * 1996-09-09 1998-06-09 Ncr Corporation Method, system and computer program product for synchronizing the flushing of parallel nodes database segments through shared disk tokens
US5799312A (en) * 1996-11-26 1998-08-25 International Business Machines Corporation Three-dimensional affine-invariant hashing defined over any three-dimensional convex domain and producing uniformly-distributed hash keys

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5970495A (en) * 1995-09-27 1999-10-19 International Business Machines Corporation Method and apparatus for achieving uniform data distribution in a parallel database system
US6125370A (en) * 1998-04-01 2000-09-26 International Business Machines Corporation Repartitioning data
US6269375B1 (en) 1998-04-01 2001-07-31 International Business Machines Corporation Rebalancing partitioned data

Also Published As

Publication number Publication date
CA2159269C (en) 2000-11-21
US5970495A (en) 1999-10-19

Similar Documents

Publication Publication Date Title
CA2159269A1 (en) Method and Apparatus for Achieving Uniform Data Distribution in a Parallel Database System
CA2150745A1 (en) Method and apparatus for implementing partial declustering in a parallel database system
WO1994025913A3 (en) Method and apparatus for enterprise desktop management
Olson et al. Decision support models and expert systems
WO1998027650A3 (en) Method and apparatus for n choose m device selection
CA2204431A1 (en) Method for making supported catalyst systems, and catalyst systems therefrom
CA2297001A1 (en) Systems and methods for implementing a dynamic cache in a supervisory control system
DE69534248D1 (en) Multi-user system for data storage, access and distribution, with distributed storage of a continuous data stream
CA2196483A1 (en) Method and apparatus for providing enhanced pay per view in a video server
CA2192581A1 (en) Method and System for Management of Frequency Spectrum Among Multiple Applications on a Shared Medium
EP0326671A3 (en) A computer system and a method of monitoring transient data structures of a computer system
DE68927624D1 (en) MULTIPLE DISK STORAGE SYSTEM WITH HIGH CAPACITY AND PROCESS
CA2078045A1 (en) Global Management of Telephone Directory
FI940949A0 (en) A method for assigning a telephone number associated with a telephone subscription, the method being implemented by telephone equipment and a mobile telephone
EP0686915A3 (en) Hierarchical resource management method
AU5598996A (en) Extensible model network representation system for process planning
WO1997042571A3 (en) Adaptive process modelling and control of rule-based agents
EP0321723A3 (en) Apparatus for a data processing system having a peer relationship among a plurality of central processing units
WO1998035519A3 (en) Adaptive frequency reuse plan
WO1995026095A3 (en) Video system
BR8401456A (en) FILLING DISTRIBUTION EQUIPMENT, EQUIPMENT FOR MEASURING AND DISTRIBUTING PREDETERMINATED QUANTITIES OF LIQUID ADDITIVE AND DISTRIBUTOR FILLING PROCESS
CA2094210A1 (en) Multiple Queue Resource Management
WO1997042575A3 (en) System and method for handling technical information
DK0663951T3 (en) Chitosan matrices for encapsulated cells
DE68925438D1 (en) High performance network with illustrated memory arrangement for real time process control system

Legal Events

Date Code Title Description
EEER Examination request
MKLA Lapsed