09 May Priority data stores
Challenge
Australia possesses many significant data stores within government agencies and research organisations (eg GA’s DEA, the National Computational Infrastructure, jurisdictional agency systems, and NCRIS facilities to name a few) which have been created fit for a specific purpose. These have been or are in the process of being migrated to cloud environments, mostly owned and operated by multi-national private sector providers, some of which are located off-shore.The use of cloud infrastructures has increased exposure in terms of sovereignty, security, fragmentation, non-optimised critical mass, duplication, barriers to access, data currency, and the role of the private sector in public-private partnership. Optimising data storage and prioritising the custodianship and physical location offers the opportunity for improvement. It is timely to examine the risks to national spatial data stores, infrastructure, systems and analytics, including the physical location of the systems on-shore and off-shore.
Another key issue is to inventory data stores related to Critical infrastructure, ie. determine which of Australia’s data stores need to be afforded special protection status, with increased governance and security, and overseen by formal data policy (including access protocols etc).
There is also an opportunity, once defined, to look at icon examples to build and use other nationally enabling infrastructure. A good example is DEA, which has increasing government and private use within Australia, and is also well regarded globally as a leading example of open data cubes. There are other examples of coordinated data collections and data stores, such as the square kilometre array, which is a next-generation radio telescope and will yield data volumes of approximately 300 PB per telescope per year during full science operations, and the data will be produced at a rate of approximately 0.5–1TB per second.
And what of the potential to create and manage datastores on-board in space? Is this considered a priority?
OPPORTUNITY FOR GROWTH
Perhaps one recent and highly relevant example which illustrates a number of the issues outlined above is in the use of information in relation to preparing and responding to bushfires. The National Natural Disaster Arrangements Royal Commission (Bushfires Royal Commission) made the following recommendations in their report, primarily within the chapter on supporting better decisions:
- Rec 4.1: Australian, state and territory governments should prioritise the implementation of harmonised data governance and national data standards.
- Rec 4.4: The National Disaster Risk Information Services Capability should include tools and systems to support operational and strategic decision making, including integrated climate and disaster risk scenarios tailored to the various needs of relevant industry sectors and end users.
- Whilst not a recommendation, the reports states that “Australian, state and territory governments should explore the feasibility and practicalities of developing and maintaining nationally consistent:
- Assessments of frequency, intensity and spatial distribution of natural hazards in Australia, and
- Projections of the frequency, intensity and spatial distribution of natural hazards in Australia.”
- The report also makes reference to GA’s Australian Exposure Information Platform (AEIP) as a tool to support decision makers understand state based exposure to natural hazards but notes that this tools does not currently provide information through a geospatial mapping layer. This tools and the underlying database, the National Exposure Information System (NEXIS), “.. should be maintained and improved”.
- Rec 4.7: Australian, state and territory governments should continue to develop a greater capacity to collect and share standardised and comprehensive natural disaster impact data.
ACTIONS
1. An audit and prioritisation of critical data priority stores that underpin national decision making. One example is Bushfire response.
2. Development of secure data stores with appropriate collaborative infrastructures that facilitate data access, with appropriate privacy and ethics requirements. This should also encompass appropriate governance including legal mechanisms, regulation frameworks, and ethics guidelines which facilitate collaborations between government, universities and industry in relation to data ethics, transparency, autonomy, and replicability.
3. The importance of standards and common frameworks for collection, storage and delivery of spatial data and that appropriate metadata needs to be emphasised to underpin management and discovery of the data and for users to determine how the data can be used and reused.
4. Consideration should also be given to required private data stores as well as shared infrastructures driven by citizen contributions.
5. Coincident computational information processing capability will need to be developed in tandem with appropriate sharing frameworks that allow researchers to collaborate and innovate.