An annual report from the National Archives and Records Administration finds little overall improvement from the last year in agency records management, although some individual agencies have excelled particularly in the realm of social media.

Read more: Federal Radio News, May 3, 2012

I’ve always considered aggregation a cornerstone of records management. Conceptually aggregations make sense to me – records or “the record” of a particular transaction or event being a group of “documents” which provide the complete picture.

In my mind I link aggregation with the principle Records should be complete  a principle which appears in many earlier Australian recordkeeping standards and guidelines.

Records should be complete: a record should contain not only the content, but also the structural and contextual information necessary to document a transaction. It should be possible to understand a record in the context of the organizational processes that produced it and of other, linked records. CAARA Principle 9 Policy 07 – Principles on Full and Accurate Records

As a concept aggregation appears to have been well understood WRT physical records where physical documents are aggregated into files with the File Title reflecting the context of a particular transaction or series of transactions.  

Metadata for physical records and rules for processing physical records (access, security, disposal) were also based around the aggregation. Disposition schedules  - classes and rules are strongly based on aggregations.  For example in NAA ADFA 2010 Class 1010 specifies:

Records documenting routine arrangements supporting ceremonies to mark special occasions. Includes catering, venue bookings and entertainment. Destroy 2 years after action completed.

and the inference is that the “action” refers to the completion of transactions for the ceremony, not for an action on a particular document within the aggregation.

Managing aggregations of digital records is not so simple. Here are just some of the challenges:

Functional requirements

• How to establish terminology and definitions which can be applied to to digital systems. What does last action really mean? Current functional specifications have done nothing to address this issue.
• How to educate system developers to address aggregation within digital systems. ie metadata connected to aggregations; linking through metadata or linking functionality; 
• Many systems such as SharePoint lack the mechanisms to dispose of aggregations, and can only dispose of individual documents.
• Many systems lack the mechanisms to retain evidence (metadata) of disposed records once the document have been destroyed or transferred.

Metadata standards

• Recordkeeping metadata standards specify the capture of functions and activities for context. For these we mostly have controlled vocabularies. We don’t yet have standards controlling other descriptive metadata. The absence of vocabulary control for subject  and transaction metadata will always result in incomplete records.

Disposition schedules

• Many (dare I say most) disposition schedules are still being written for legacy records management systems. We need to develop disposition schedules which can function more easily within business systems, because if we have learned anything over the past few years it is that business users are resistant to using enterprise records management systems.

These are complex challenges but ones that must be addressed becasue the principle of aggregation is (I strongly believe) essential to best practice recordkeeping.

Click here for a glossary of information governance definitions.

Compared to the other markets surveyed, Australians spend more time receiving and managing information (54%) than actually using that information (46%).

58% of Australian workers have disagreements among colleagues about the right way to organise information at least once in an average week, and 48% have delivered incomplete documents, email or other communications because they could not find the necessary information in time.

Only one in five (22%) Australian professionals responded that their company has offered training in information management in the past two years, despite 60% wanting their companies to do so.

Data, data everywhere

A Different Game
Information is transforming traditional businesses

Needle in a haystack
The uses of information about information

All too much
Monstrous amounts of data

The Open Society
Governments are letting in the light

New rules for big data
Regulators are having to rethink their brief

Handling the cornucopia
The best way to deal with all that information is to use machines. But they need watching

http://www.cioupdate.com/insights/article.php/3917496/ITs-Time-for-Information-Governance.htm

On being asked to identy their biggest problems with managing digital information management ICT personnel nominated:

• Overwhelming volume of storage and problems with storage management
• Too many systems/formats and inadequate technology or poor integration
• Inadequate training/communication or failure of staff to meet their responsibilities
• Issues with authenticity and security
• Difficulties managing and preserving digital information over time
• Little support from senior management
• Inadequate resources
• Lack of guidance and standards
• Poor email management
• Lack of appraisal and retention/disposal requirements
• Issues with ownership
• Poor compliance and quality control
• Approaches poorly aligned to business processes

A number of respondents identified that structuring of information within systems was either absent, too complex, poorly done, not appropriate for digital records,
not implemented effectively or not accepted/used by staff. “We experience problems with] classification, categorisation etc to enable information retrieval.”
Some were concerned with naming conventions for files, while others expressed the need for naming conventions for documents and network folders, and better
version control (17 respondents mentioned this specifically). Eight respondents indicated that metadata in general needs to be assigned or improved.
Retrieval was seen by 57 of the 140 respondents (26.3%) as the ultimate goal of the classification/structuring of information. Searching across multiple systems,
the inability to access archival data or data at remote sites and the inability to share data across organisations were raised as particular issues.
Duplication was also listed as an issue by 14 respondents (6.5%), and inadequate classification (combined with a lack of guidance regarding what to capture) may
exacerbate this problem.

The full report can be found at: http://www.records.nsw.gov.au/recordkeeping/topics/records-management/documents/recordkeeping-surveys/ICT%20attitudes%20-%20survey%20report.pdf

The DIRKS methodology is based on traditional system design methodologies which have been adapted for the records management environment. The step-by-step analysis described in DIRKS examines many organizational facets – structure, environment, stakeholders, processes, inputs and outputs.

In many ways the DIRKS outputs can be likened to the DNA of an organization. DNA is the molecule that encodes genetic information in the nucleus of cells and determines the structure, function, and behavior of cells. Similarly DIRKS describes how an organization is structured from the broadest functions to the minutest detail of business activity.

Steps B and C of the DIRKS methodology deliver a framework for identifying and linking essential recordkeeping elements within connected metadata structure.

• Step B is the analysis of business activity resulting in the identification of business functions, activities, tasks and record or document types.
• Step C is the identification of recordkeeping requirements for each class of records based on the analysis of sources (i.e. legislation) and stakeholder requirements.

From this analysis, many of the facets needed to create the tools required for RIM systems can be brought together along with other taxonomy/metadata facets to build file plans, thesauri, retention schedules, access and security models, and workflows definitions.

The DIRKS methodology has been around now for nearly a decade. The tragedy is that the RIM industry has been slow to realize the benefits of the connected DIRKS framework in that it provides real opportunities for automated retention management. However we believe that the tools are now available to prove the model and it is possible to build an information architecture that utilizes DIRKS to its full potential. More about that later.

Metadata is a hard sell. It is expensive to create and difficult to maintain. Executives have a tough time understanding how the problem of having too much information to manage can be solved by adding on yet more information. Metadata is a bit of a “hair of the dog” solution. We add a little information to make a lot of information more usable. As to the expense, the answer is, or course, pay now or pay more later; sometimes a lot more. A few minutes tagging a document can save hours hunting for it later.

In Australia government agencies have been subject to a range of metadata standards i.e. the Australian Commonwealth Standard for Recordkeeping.  Metadata standards been incorporated into eCM systems to deliver “recordkeeping compliance”.   In nearly all cases, business users have been expected to input recordkeeping metadata.  Much effort involved. Yet so little effort has gone into providing metadata for business use – i.e. metadata that will help business users find, filter, group or sort information.  eCM developers have not seen fit to deliver functionality for adding, connecting  and leveraging metadata.

It’s hardly surprising that business users opt out of records management systems and create their own information management systems.

I believe that SharePoint’s enormous popularity has been because it has given control of metadata back to the business users, hence the proliferation of team sites and databases. However an absence of controls wrt metadata leave SharePoint users with the same old problems in other systems – what may be a good set of metadata for one set of users is not necessarily going to benefit the organisation as a whole.

As Steward says, metadata provides best value for money when it is controlled – i.e. when it is created and used consistently, at a user, enterprise or global level, ensuring that everybody is speaking the same language.

Taxonomies are at the same time deceptively simple and fiendishly complicated.  They are simple because they are absolutely basic to human consciousness, so everybody manipulates and creates them with great ease.

Taxonomies are complicated because we use them for the most part unreflectingly – they are simply part of our mental and social background- and we use categorise in a hyge variety of ways, often in competing and inconsistent ways.

Different people with different tasks will order their knowledge assets differently to suit the tasks they have to do…

Our clients can and do categorise continually. They are confident about knowing how to sort things out . But they are not skilled at conscious, strategic organisation of their knowledge assets to suit collective tasks.

When it comes to records and information management (RIM) systems Lambe hits the nail on the head. Classification schemes are more than just search and retrieval tool. We need them to achieve a whole range of purposes including:

• Aggregation of content (grouping records together for business or evidential purposes)
• Browsing and navigation
• Access & security management
• Disposition management
• Filtering  and segmentation
• Managing provenance
• Reporting

Web research confirms a renewed interest in classification (taxonomies and metadata). There are a whole raft of new taxonomy and metadata products available, including those inbuilt into eCM systems like SharePoint 2010. However the marketing blurb suggests that the purpose of classification schemes is in the delivery of improved deliver search and retrieval.  Little is said about the theory and discipline of building classification schemes.

Both Lambe and Stewart in their books address the need to understand how different criteria are needed for different purposes. Lambe (Organising Knowledge, 2007) lists the following criteria (facets) used to classify content:

• Business activity i.e. functions, activities, processes, transactions
• Subject matter i.e. products, services, disciplines, domains, assets
• Names of people and organisations
• Aggregation i.e. cases
• Record or document type
• Geographic  location
• Time or sequence

The issue taxonomy developers face is that there is no single criteria which is able to realise all of the collective classification needs of the organisation.

The following matrix illustrates how different classification criteria (or facets) meet different needs. Some meet the aggregation and retrieval needs of business users, others meet compliance needs of the organisation. The art of building effective classification schemes is to identify all of the organisation’s classification needs and to weave them together into an IDEAL information architecture.

Synercon Classification Matrix

One of our pet projects is the development of an IDEAL set of functional requirements for RIM systems. IDEAL requirements address five usability concerns: Integration, Description, Effort, Arrangement and Linking, not necessarily in this order. In this blog we are attempting to define a set of IDEAL requirements to benchmark against. We welcome your feedback.