The increasingly digital nature of our world has seen a huge rise in data size and quantity. Now, new technologies such as the Square Kilometre Array are about to create an even fiercer deluge of data. Baz Macdonald investigates.
In 2011 NZ became one of the founding members of the SKA organisation, a not-for-profit company dedicated to creating the world’s largest radio telescope.
When completed, the Square Kilometre Array (SKA) will be the most sensitive telescope ever created, and allow us to see 10 times deeper into space than we ever have before. However, a telescope of this size and sensitivity also brings with it some significant challenges, the foremost of which is the amount of data it will create – with some estimates showing that in one day the array will create more data than the whole internet has created in the past 30 years.
The telescope will require thousands of dishes and aperture array telescopes, which will be installed in the deserts of Australia and South Africa.
While NZ may not be hosting the telescope infrastructure, we are playing an important role by helping to advance technologies that will support it. NZ researchers and research and development firms are working to find ways of dealing with the big data deluge the array will create.
This research is underway, and although SKA is not completed, we are already starting to reap the benefits – with our country considered one of the world leaders in tackling this data challenge, as we continue to invest in fields of Artificial Intelligence, data learning and cloud computation.
The big data deluge
Nicolás Erdödy the founder and CEO of Open Parallel, an R&D firm that is working alongside our universities and companies in developing the technology necessary to make SKA a success, said they are continually discovering practical applications for industries.
“To process [SKA’s data] you need supercomputers, as well as new tools, operating systems, storage and applications. By working on that, we start to learn how we can use these new tools to optimise primary industries”
Open Parallel demonstrated this by using an algorithm developed for SKA to show how satellite data could be used to read water levels on an area of pasture, making water and irrigation management easier for farmers.