WO2012083336A1 - Processing engine - Google Patents
Processing engine Download PDFInfo
- Publication number
- WO2012083336A1 WO2012083336A1 PCT/AU2010/001735 AU2010001735W WO2012083336A1 WO 2012083336 A1 WO2012083336 A1 WO 2012083336A1 AU 2010001735 W AU2010001735 W AU 2010001735W WO 2012083336 A1 WO2012083336 A1 WO 2012083336A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- unstructured
- descriptor
- processing engine
- mappable
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2291—User-Defined Types; Storage management thereof
Definitions
- the present invention relates to the processing and egorisation of large volumes of data and, more particularly but not exclusively to financial reporting data.
- the technical problem to be addressed is that of how to take a databit stream coded according to a first protocol and representing unstructured data and apply logical processing to that databit stream so as to produce tocol and representing structured data.
- the structured a can then have further processing performed on it so as achieve outcomes that could not be achieved either at all or in a reasonable time frame but for the preceding logical processing step.
- the processing engine of a preferred embodiment of the present invention when applied to a financial database automatically "tags” (also commonly referred to as “maps” or “allocates") information from an ined function codes, groups or fields; thus identifying populating .new and usable ways of filtering and regating the financial database information.
- tags also commonly referred to as “maps” or “allocates”
- a processing engine for processing unstructured data .located in an unstructured data table; said data comprising quantity data associated with an unstructured descriptor element; said engine comparing each said unstructured descriptor element against a master list of mappable data descriptors to create a single data description per entry; in the event of a match between an unstructured descriptor element and a mappable data descriptor in the mappable data descriptors list then quantity data associated with said unstructured data element is associated with the matching mappable data descriptor and copied into a partly structured data table.
- a processing engine which receives a databit stream coded according to a first protocol and representing unstructured data; said processing engine applying logical ;abit stream coded according to a second protocol and >resenting structured data.
- a processing engine which automatically "tags” (also commonly referred to as “maps” or “allocates”) information ' from an already existing financial database with, newly created and defined function codes, groups or fields; thus identifying and populating new and usable ways of filtering and aggregating the financial database information.
- Preferably said partly structured data table is reviewed manually.
- those unstructured descriptor elements for which no match is available are manually reviewed and a manual :ch made so as to produce a fully structured data table >m said partially structured data table.
- those unstructured descriptor elements for which no match is available comprise less than 25 per cent of said unstructured data.
- said unstructured data comprises accounting data.
- the structured data table can be filtered and aggregated by all data fields, including newly created data descriptions, thereby to provide new ways of viewing, summarising and disseminating data.
- a processing engine for processing unstructured data located in an unstructured data ' table in a computer memory; said data comprising quantity data associated with an unstructured descriptor element; said engine including a ment against a master list of mappable data descriptors create a single data description 1 per entry; in the event a match signal from said comparator indicating a match between an unstructured descriptor element and a mappable data descriptor in the mappable data descriptors list then quantity data associated with said unstructured data element is associated with, the matching mappable data descriptor and copied into a partly structured data table in said computer memory.
- said database and said comparator are implemented via a microprocessor in communication with a computer memory.
- Figure 1 is a block diagram of dataflow through a processing engine in accordance with a first preferred embodiment of the present invention
- FIG. 2 is a logic flow diagram of processing steps applied to data in accordance with the first embodiment
- Figure 3 illustrates data mapping for a particular instance of data
- Figure 4 is an example of a fully structured data de
- Fig 5 illustrates a further example of the tagging process where translation is to a common descriptor base - in this case an XBRL common descriptor set,
- Fig 6 is a simplified explanation in table form of an embodiment of the technology process showing how the Master Chart Of Accounts tag may be determined.
- the processing engine 10 comprises a translator 11 which receives input from comparator 12.
- the comparator 12 receives and compares data in unstructured data table 13 and mappable data descriptor table 14.
- the output of translator 11 is input to partly structured table 15.
- a manual processing operation is then applied to table 15, the output of which results in a fully structured data table 16.
- FIG. 2 a logic flow diagram is provided for processing steps applied to data passing tance, the data relates to trial balance data.
- the processing engine is termed the itoallocator" in Figure 2.
- Figure 3 illustrates data mapping in the instance where data is trial balance data and comprises a listing of unstructured descriptor elements 17 and a corresponding listing of mappable data descriptors 18. So, for example, unstructured descriptor element 17A comprising the •descriptor "till floats" maps to 'a matched mappable data descriptor 18a being descriptor "cash on hand.”
- the processing engine automatically "tags” (also commonly referred to as “maps” or “allocates”) information ated and defined function codes, groups or fields; thus ntifying and populating new and usable ways of filtering aggregating the financial database information.
- tags also commonly referred to as “maps” or “allocates”
- the "tagging” (also commonly referred to as “mapping” or “allocating”) process uses elements of existing financial database fields (such as the general ledger code, account name / description or balance) to predict a new field (such as a Master Chart of Accounts category or XBRL Value) using, amongst other describable processes, a matrix and array linear pattern.
- the technology also contains an inbuilt validation process, largely centred around the GL Code (with ranges input to match the current GL, ' which varies from system to system) and the GL Balance.
- the tagging process is instantaneous - once you import the financial database (e.g. the trial balance), the new function codes (e.g. Master ' Chart of Accounts) appears immediately.
- 5 illustrates a further example of the tagging process re translation is to a common descriptor base - in this e an XBRL common descriptor set.
- Fig 6 is a simplified explanation of the technology process as to how it determines the Master Chart Of Accounts tag.
- a fully structured data table 16 is illustrated with reference to a particular example.
- the unstructured descriptor "Westpac account” is associated with quantity .ql . It is mapped by way of the table of Figure 3 to descriptor “cash at bank” associated with quantity ql.
- the descriptor "trade debtors,” is initially associated with quantity q2 and is mapped to be associated with descriptor "trade receivables.”
- mappable data descriptor table the smaller the amount of manual allocation will be as a proportion of total amount of data processed thereby resulting in the potential for relatively rapid processing of large volumes of data presenting as unstructured data from disparate sources.
Abstract
A processing engine which automatically "tags" (also commonly referred to as "maps" or "allocates") information from an already existing financial database with newly created and defined function codes, groups or fields; thus identifying and populating new and usable ways of filtering and aggregating the financial database information. The biggest advantages of this are elimination of time for manual creation; and also reliable and consistent treatment of these new populations thus significantly reducing errors.
Description
PROCESSING ENGINE
The present invention relates to the processing and egorisation of large volumes of data and, more particularly but not exclusively to financial reporting data.
BACKGROUND
Computers are known for their ability to process large volumes of data. However, in the case where the data, or at least parts of it, is unstructured there can be difficulties in the computer recognising the information contained in the data and knowing how to handle it.
There are many instances in industry where unstructured data presents a processing problem to data processing devices. For example accounting information in the form of trial balances from general ledger systems will present to a data processing system in an unstructured way and typically a different way depending on the general ledger system from which the data is derived.
The technical problem to be addressed is that of how to take a databit stream coded according to a first protocol and representing unstructured data and apply logical processing to that databit stream so as to produce
tocol and representing structured data. The structured a can then have further processing performed on it so as achieve outcomes that could not be achieved either at all or in a reasonable time frame but for the preceding logical processing step.
It is an object of the present invention to address or at least ameliorate some of the above disadvantages.
Notes
1. The term "comprising" (and grammatical variations thereof) is used in this specification in the inclusive sense of "having" or "including", and not in the exclusive sense of "consisting only of".
2. The above discussion of the prior art in the .
Background of the invention, is not an admission that any information discussed therein is citable prior art or part of the common general knowledge of persons skilled in the art in any country. BRIEF DESCRIPTION OF INVENTION
In a broad form the processing engine of a preferred embodiment of the present invention when applied to a financial database automatically "tags" (also commonly referred to as "maps" or "allocates") information from an
ined function codes, groups or fields; thus identifying populating .new and usable ways of filtering and regating the financial database information. The biggest advantages of this are elimination of time for manual creation; and also reliable .and consistent treatment of these new populations thus significantly reducing errors.
' Accordingly, in a further broad form of the present invention there is provided a processing engine for processing unstructured data .located in an unstructured data table; said data comprising quantity data associated with an unstructured descriptor element; said engine comparing each said unstructured descriptor element against a master list of mappable data descriptors to create a single data description per entry; in the event of a match between an unstructured descriptor element and a mappable data descriptor in the mappable data descriptors list then quantity data associated with said unstructured data element is associated with the matching mappable data descriptor and copied into a partly structured data table.
In a further broad form of the present invention there is provided a processing engine which receives a databit stream coded according to a first protocol and representing unstructured data; said processing engine applying logical
;abit stream coded according to a second protocol and >resenting structured data.
In a further broad form of the present invention there is provided a method for converting a databit stream coded according- to a first protocol and representing unstructured ■data via a step of logical processing to that databit stream so as to produce a second databit stream coded according to a second protocol and representing structured data.
In yet a further broad form of the present invention there is provided a processing engine which automatically "tags" (also commonly referred to as "maps" or "allocates") information' from an already existing financial database with, newly created and defined function codes, groups or fields; thus identifying and populating new and usable ways of filtering and aggregating the financial database information.
Preferably said partly structured data table is reviewed manually.
Preferably those unstructured descriptor elements for which no match is available are manually reviewed and a manual
:ch made so as to produce a fully structured data table >m said partially structured data table.
Preferably those unstructured descriptor elements for which no match is available comprise less than 25 per cent of said unstructured data.
Preferably said unstructured data comprises accounting data. -
Preferably the structured data table can be filtered and aggregated by all data fields, including newly created data descriptions, thereby to provide new ways of viewing, summarising and disseminating data.
Preferably advantages of employing the processing engine are elimination of time for manual creation; and also reliable and consistent treatment of these new data descriptions thus significantly reducing errors.
In yet a further broad form of the invention there is provided a processing engine for processing unstructured data located in an unstructured data ' table in a computer memory; said data comprising quantity data associated with an unstructured descriptor element; said engine including a
ment against a master list of mappable data descriptors create a single data description1 per entry; in the event a match signal from said comparator indicating a match between an unstructured descriptor element and a mappable data descriptor in the mappable data descriptors list then quantity data associated with said unstructured data element is associated with, the matching mappable data descriptor and copied into a partly structured data table in said computer memory.
'
Preferably said database and said comparator are implemented via a microprocessor in communication with a computer memory. BRIEF DESCRIPTION OF DRAWINGS
Embodiments of the present invention will now be described with reference to the accompanying drawings wherein:
Figure 1, . is a block diagram of dataflow through a processing engine in accordance with a first preferred embodiment of the present invention,
Figure 2, is a logic flow diagram of processing steps applied to data in accordance with the first embodiment,
Figure 3, illustrates data mapping for a particular instance of data,
Figure 4 is an example of a fully structured data de,
Fig 5 illustrates a further example of the tagging process where translation is to a common descriptor base - in this case an XBRL common descriptor set,
Fig 6 is a simplified explanation in table form of an embodiment of the technology process showing how the Master Chart Of Accounts tag may be determined.
DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS
With reference to Figure 1 there is illustrated a processing engine 10 in accordance with a first preferred embodiment of the present invention. In this instance the processing engine 10 comprises a translator 11 which receives input from comparator 12. The comparator 12 receives and compares data in unstructured data table 13 and mappable data descriptor table 14. The output of translator 11 is input to partly structured table 15. A manual processing operation is then applied to table 15, the output of which results in a fully structured data table 16.
With reference to Figure 2 a logic flow diagram is provided for processing steps applied to data passing
tance, the data relates to trial balance data. In this tance the processing engine is termed the itoallocator" in Figure 2.
FRS AutoAllocator - outline
Using the FRS AutoAllocator (part of the FRS Process), we can define the process as:
' Importing Trial Balances from any General Ledger system and the FRS System (part of the FRS Process) automatically knows what Master Chart Of Account Allocation (based on data input from the International Financial Reporting Standards) to give it.
For example it would recognise:
GL Account Name Master Chart
Till floats as Cash on hand
estpac account as Cash at bank
Figure 3 illustrates data mapping in the instance where data is trial balance data and comprises a listing of unstructured descriptor elements 17 and a corresponding listing of mappable data descriptors 18. So, for example, unstructured descriptor element 17A comprising the •descriptor "till floats" maps to 'a matched mappable data descriptor 18a being descriptor "cash on hand."
In summary the processing engine automatically "tags" (also commonly referred to as "maps" or "allocates") information
ated and defined function codes, groups or fields; thus ntifying and populating new and usable ways of filtering aggregating the financial database information. The biggest advantages of this are elimination of time for manual creation; and also reliable and consistent treatment of these new populations thus significan ly reducing errors.
Explanation of tagging process
The "tagging" (also commonly referred to as "mapping" or "allocating") process uses elements of existing financial database fields (such as the general ledger code, account name / description or balance) to predict a new field (such as a Master Chart of Accounts category or XBRL Value) using, amongst other describable processes, a matrix and array linear pattern. The technology also contains an inbuilt validation process, largely centred around the GL Code (with ranges input to match the current GL,' which varies from system to system) and the GL Balance.
The tagging process is instantaneous - once you import the financial database (e.g. the trial balance), the new function codes (e.g. Master' Chart of Accounts) appears immediately.
5 illustrates a further example of the tagging process re translation is to a common descriptor base - in this e an XBRL common descriptor set. Fig 6 is a simplified explanation of the technology process as to how it determines the Master Chart Of Accounts tag.
In Use
With reference to Figure 4 a fully structured data table 16 is illustrated with reference to a particular example. In this instance the unstructured descriptor "Westpac account" is associated with quantity .ql . It is mapped by way of the table of Figure 3 to descriptor "cash at bank" associated with quantity ql. Similarly, the descriptor "trade debtors," is initially associated with quantity q2 and is mapped to be associated with descriptor "trade receivables."
In the case of the descriptor "special advance" there is no mapping available from the table of Figure 3 so, initially, no mapping takes place in the creation of the partially structured data table. The manual data entry step is then performed whereby the descriptor "special advance" is mapped to "other receivables" and remains associated with quantity qn.
Having produced the fully structured data table the
it in an automated fashion, it will be appreciated that more comprehensive the mappable data descriptor table the smaller the amount of manual allocation will be as a proportion of total amount of data processed thereby resulting in the potential for relatively rapid processing of large volumes of data presenting as unstructured data from disparate sources.
The above describes only some embodiments of the present invention and modifications, obvious to those skilled in the art, can be made thereto without departing from the scope of the present invention.
Claims
A processing engine for processing unstructured data located in an unstructured data table; said data comprising quantity data associated with an unstructured descriptor element; said engine comparing each said unstructured descriptor element against a master list of mappable data descriptors to create a single data description per entry; in the event of a match between an unstructured descriptor element and a mappable data descriptor in the mappable data descriptors list then quantity data associated with said unstructured data element is associated with the matching mappable data descriptor and copied into a partly structpred data table.
The processing engine of claim 1 wherein said partly structured data table is reviewed manually
The processing engine of claim 2 wherein those unstructured descriptor elements for which no match is available are manually reviewed and a manual match made so as to produce a fully structured data table from said partially structured data table. The processing engine of claim 3 wherein those unstructured descriptor elements for which no match is available comprise less than 25 per cent of said
5. The processing engine ' of claim 3 wherein said unstructured data comprises accounting data.
6. The processing engine of any previous claim wherein the structured data table can be filtered and aggregated by all data fields,, including newly created data descriptions, thereby to provide new ways of viewing, summarising and disseminating data.
7. The processing engine of any previous claim wherein advantages of employing the processing engine are elimination of time for manual creation; and also reliable and consistent treatment of these new data descriptions thus significantly reducing errors.
8. A processing engine for processing unstructured data located in an unstructured data table in a computer memory; said data comprising quantity data associated with an unstructured descriptor element; said engine including a comparator which compares each said unstructured descriptor element against a master list of mappable data descriptors to create a single data description per entry; in the event of a match signal from said comparator indicating a match between an unstructured descriptor element and a mappable data descriptor in the mappable data descriptors list then quantity data associated with said unstructured data element is associated with
into a partly structured data table in said computer memory.
9. The processing engine of any previous claim wherein said database and said comparator are implemented via a microprocessor in communication with a computer memory.
10. A machine readable medium including executable code executable on a processor which implements the processing engine of any one of claims 1 to 9.
11. A processing engine which receives a databit stream coded according to a first protocol and representing unstructured data? said processing engine applying logical processing to that databit stream so as to produce a second databit stream coded according to a second protocol and representing structured data.
12. The processing engine of claim 11 wherein said databit stream comprises quantity data associated with an unstructured descriptor element; said engine including a comparator which compares each said unstructured descriptor element against a master list of mappable data descriptors to create a single data description per entry; in the event of a match
between an unstructured descriptor element and a mappable data descriptor in the mappable data descriptors list then quantity data associated with said unstructured data element is associated with the matching mappable data descriptor and copied via said second databit stream into a partly structured data table in said computer memory.
The processing engine of claim 12 wherein said database and said comparator are implemented via a microprocessor in communication with a ■ computer memory.
A method for converting a databit stream coded according to a first protocol and representing unstructured data via a step of logical processing to that databit stream so as to . produce a second databit stream coded according to a second protocol and representing structured data.
The method of claim 14 wherein said step of logical processing is performed by a processing engine for processing unstructured data located in an unstructured data table in a computer memory; said unstructured data comprising quantity data associated with an unstructured descriptor element; said engine including a comparator which compares
master list of mappable data descriptors to create a single data description per entry; in the event of a match signal from said comparator indicating a match between an unstructured descriptor element and a mappable data descriptor in the mappable data descriptors list then quantity data associated with said unstructured data element is associated with the matching mappable data descriptor and copied into a partly structured data table in said computer memory via said second databit stream coded according to a second protocol and representing structured data.
16. The method of claim 14 or 15 wherein said database and said comparator are implemented via a microprocessor in communication with a computer memory.
.. A processing engine which automatically "tags" (also commonly referred to as "maps" or "allocates") information from an already., existing financial database with newly created and defined function codes, groups or fields; thus identifying and populating new and usable ways of filtering and aggregating the financial database information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/AU2010/001735 WO2012083336A1 (en) | 2010-12-23 | 2010-12-23 | Processing engine |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/AU2010/001735 WO2012083336A1 (en) | 2010-12-23 | 2010-12-23 | Processing engine |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2012083336A1 true WO2012083336A1 (en) | 2012-06-28 |
Family
ID=46312874
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/AU2010/001735 WO2012083336A1 (en) | 2010-12-23 | 2010-12-23 | Processing engine |
Country Status (1)
| Country | Link |
|---|---|
| WO (1) | WO2012083336A1 (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2003058504A1 (en) * | 2001-12-27 | 2003-07-17 | Fair Isaac Corporation | Augmenting data in a database for predictive modeling |
| US20050108256A1 (en) * | 2002-12-06 | 2005-05-19 | Attensity Corporation | Visualization of integrated structured and unstructured data |
| US7668849B1 (en) * | 2005-12-09 | 2010-02-23 | BMMSoft, Inc. | Method and system for processing structured data and unstructured data |
-
2010
- 2010-12-23 WO PCT/AU2010/001735 patent/WO2012083336A1/en active Application Filing
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2003058504A1 (en) * | 2001-12-27 | 2003-07-17 | Fair Isaac Corporation | Augmenting data in a database for predictive modeling |
| US20050108256A1 (en) * | 2002-12-06 | 2005-05-19 | Attensity Corporation | Visualization of integrated structured and unstructured data |
| US7668849B1 (en) * | 2005-12-09 | 2010-02-23 | BMMSoft, Inc. | Method and system for processing structured data and unstructured data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108268593B (en) | Method, device, server and storage medium for processing credit card insurance information | |
| CN111475513B (en) | Form generation method and device, electronic equipment and medium | |
| CN116205724A (en) | Large scale heterogeneous data ingestion and user resolution | |
| US7624113B2 (en) | Data element naming system and method | |
| CN110083623B (en) | Business rule generation method and device | |
| CN103428080B (en) | A kind of data unpack and organize bag method | |
| CN109086433A (en) | A kind of file management method and server based on big data analysis | |
| CN112115145A (en) | Data acquisition method and device, electronic equipment and storage medium | |
| CN111986039A (en) | Claims wind control identification method and device, computer equipment and readable storage medium | |
| CN111897856A (en) | Supervision message generation method and device, electronic equipment and readable storage medium | |
| CN113434674A (en) | Data analysis method and device, electronic equipment and readable storage medium | |
| CN105389346A (en) | Bidding clearing method and apparatus in electronic bidding evaluation system | |
| CN110097444A (en) | A kind of flowing water code generating method and device | |
| CN113157853A (en) | Problem mining method and device, electronic equipment and storage medium | |
| CN111985194A (en) | Data storage method and device, electronic equipment and storage medium | |
| CN112541688A (en) | Service data checking method and device, electronic equipment and computer storage medium | |
| CN112101889A (en) | Metadata-based business flow control method and device | |
| CN111159183A (en) | Report generation method, electronic device and computer readable storage medium | |
| US10042902B2 (en) | Business rules influenced quasi-cubes with higher diligence of data optimization | |
| AU2009233605B2 (en) | Processing Engine | |
| WO2012083336A1 (en) | Processing engine | |
| CN110648219B (en) | Method and device for standardizing input area of bank transaction system | |
| CN115034674A (en) | Charging management method, system and electronic equipment | |
| CN101425167A (en) | Method for generating and parsing trading information | |
| CN106570004B (en) | Data management method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 10861183 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 10861183 Country of ref document: EP Kind code of ref document: A1 |