Is Data Archiving Obsolete?

With so much data available to organizations these days, is there any benefit associated with archiving data?

Data management is one of an IT department’s key costs. One estimate is that chief information offices spend roughly 70% of their budget maintaining existing systems, which includes data-inundated warehouses and those legacy applications associated with them. Some organizations can see its data volume increase three to five times or more in a two-year period.

Information technology professionals have long advocated a back-up-everything policy; however, Jim McGann, vice president, Index Engines, writing in Enterprise Systems, says some data is important for adhering with legal and compliance issues, but retaining other data could prove a corporate liability.

“Archiving all corporate data forever is not smart. It is not a policy that any legal or compliance team would endorse,” says McGann. He estimates that roughly 90% of an organization’s data does not need to be archived, nor should it be.

Big data and technologies such as cloud computing are pushing organizations to adopt integrated data structures, which may require consideration of new, more complex methods for managing data. These systems will need to use existing information such as structured-transactional data plus new forms of information gathered from social media or sensor networks, for example.

A large global financial institution with more than 200TB of data and as many as five million unique user queries daily was having difficulty managing its data. It was ultimately able to archive and off-load data from its data warehouse, saving more than US$22 million, according to Venkat Lakshminarasimha, global big data integration specialist with Informatica, in a workshop presentation at FutureGov Singapore Forum 2013. With savings continuing to increase, the organization was able to implement data management software that saved 100TB of storage space in two-and-a-half years and, among the other benefits, resulted in a prompt return on investment.

Although hardware-based storage may seem obsolete, it is still effective and efficient for storing. Deni Connor, founding analyst, Storage Strategies NOW/Systems Strategies NOW, told Network Computing that tape is still used, but disk storage has become increasingly important for those users wanting to protect their data. Some small- to medium-sized businesses are moving to cloud-based storage solutions in lieu of tape media archival.

Even with resources like the cloud available, Connor says organizations still need to have some system in place for governance, risk management, and compliance. This means an organization must have systems on-site that can allow it to respond to data queries very quickly.

McGann adds that when organizations are smart about the data they archive, they reap a wide range of benefits, including reduced storage costs as well as managing corporate liability. What are your thoughts? Let us know, we’d love to get your opinion on data archiving.