NDSA:Discussions on Preservation Storage Topics - Revision history

Dlfadm: 24 revisions imported: Migrate NDSA content from Library of Congress

2016-02-11T19:19:19Z

24 revisions imported: Migrate NDSA content from Library of Congress

← Older revision		Revision as of 15:19, 11 February 2016
(No difference)

Jbailey: /* Other Ideas */

2012-03-28T15:09:51Z

Other Ideas

← Older revision		Revision as of 11:09, 28 March 2012
Line 11:		Line 11:

	==Other Ideas==		==Other Ideas==
			*Frequency of fixity checking
			*Number of preservation copies maintained
			*Storage hierarchy
			*Continued discussions around use of the cloud

Jbailey at 15:08, 28 March 2012

2012-03-28T15:08:04Z

Show changes

Jbailey at 15:05, 28 March 2012

2012-03-28T15:05:19Z

@@ Line 5: / Line 5: @@
 ===Questions===
-Data compression can decrease the cost of long-term preservation by reducing the amount of storage required.  There are at least three types of compression to consider:
+The discussion on compression can be found at [[NDSA:Preservation Storage Topic 2: Compression]]
 ==Topic 1: Encryption==

Jbailey: /* Topic 2: Compression */

2012-03-08T14:55:26Z

Topic 2: Compression

← Older revision		Revision as of 10:55, 8 March 2012
Line 4:		Line 4:
	==Topic 2: Compression==		==Topic 2: Compression==

			===Questions===
	Data compression can decrease the cost of long-term preservation by reducing the amount of storage required. There are at least three types of compression to consider:		Data compression can decrease the cost of long-term preservation by reducing the amount of storage required. There are at least three types of compression to consider:

Jbailey: /* Topic 1: Encryption */

2012-03-08T14:54:54Z

Topic 1: Encryption

← Older revision		Revision as of 10:54, 8 March 2012
Line 48:		Line 48:

	==Topic 1: Encryption==		==Topic 1: Encryption==

			===Questions===
	'''Do you have any opinions on it? What are your reasons for your opinions (gut feelings are OK)?'''		'''Do you have any opinions on it? What are your reasons for your opinions (gut feelings are OK)?'''

Jbailey: /* Topic 2: Compression */

2012-03-07T21:06:58Z

Topic 2: Compression

← Older revision		Revision as of 17:06, 7 March 2012
Line 3:		Line 3:

	==Topic 2: Compression==		==Topic 2: Compression==

			Data compression can decrease the cost of long-term preservation by reducing the amount of storage required. There are at least three types of compression to consider:

			*file compression, using a file compression algorithm suited to the file type
			*hardware compression, which usually means compression done by a tape drive as the data is written to tape
			*disk compression, which is performed by many new storage appliances and uses a combination of compression and de-duplication

			If there are other kinds of compression, please add that into this discussion.

			For each of these types of compression:

			*1. Are you currently using this type of compression in your own archival storage (in the OAIS sense of long-term preservation storage)?
			*2. How do you feel about using this type of compression for archival storage? Is this legitimate or something that Best Practice would discourage?
			*3. What are the particular risks of this type of compression, if any, for preservation?
			*4. Are there any advantages to using this type of compression beyond reducing storage costs?
			*5. How do you trade off cost vs. risk?

			Compressed formats are in general much more sensitive to data corruption than uncompressed formats. Due to the 'amplification' effect that compression has on data corruption, the percentage saving in storage space is often much less than the percentage increase in the amount of information that is affected by data corruption.

			===A Selected Bibliography===

			(Priscilla: I looked for best practices or other documents that addressed compression in the preservation context. Below are some snippets of what I found, addressing mainly file and hardware compression.)

			*Case Western University Archives: Compression adds complexity to long-term preservation. Some compression techniques shed "redundant" information. As an example, JPEG removes information to reduce file size. The image might look fine on your current monitor, but as monitors improve, the lower quality of the image will be more obvious.

			*Wright, Miller and Addis in http://www.prestoprime.org/docs/training/Cost_of_risk_RW.pdf

			Not encoding, in particular not using compression, typically results in files that have minimal sensitivity to corruption. In this way, the choice not to use compression is a way to mitigate against loss.

			*TNA on Image Compression in http://www.nationalarchives.gov.uk/documents/image_compression.pdf

			It is recommended that algorithms should only be used in the circumstances for which they are most efficient. It is also strongly recommended that archival master versions of images should only be created and stored using lossless algorithms. The Intellectual Property Rights status of a compression algorithm is primarily an issue for developers of format specifications, and software encoders/decoders. However, the use of open, non-proprietary compression techniques is recommended for the purposes of sustainability.

			*Howard Besser, quoted in http://digitalpreservationstrategies.blogspot.com/

			Data is often compressed or "scrambled" to assist in its storage and or protect it's intellectual content. These compression and encryption algorythms are often developed by private organisations who will one day cease to support them. If this happens you're stuck between a rock and a hard place. If you don't want to get into legal trouble you are no longer able to read your data; and if you go ahead and "do the unwrapping yourself" it's quite possible you're breaking copyright law.

			*NINCH Guide to Good Practice: http://www.nyu.edu/its/humanities/ninchguide/XIV/

			A similar obsolescence problem will have to be addressed with the file formats and compression techniques you choose. Do not rely on
			proprietary file formats and compression techniques, which may not be supported in the future as the companies which produce them merge, go out of business or move on to new products. In the cultural heritage community, thede factostandard formats are uncompressed TIFF for images and PDF, ACSII (SGML/XML markup) and RTF for text. Migration to future versions of these formats is likely to be well-supported, since they are so widely used. Digital objects for preservation should not be stored in compressed or encrypted formats.

			*PRESTO Centre, Threats to Data Integrity from Large-Scale Management Environments: http://www.prestocentre.org/library/resources/threats-data-integrity-use-large-scale-management-environments

	==Topic 1: Encryption==		==Topic 1: Encryption==

Jbailey: /* Topic 1: Encryption */

2012-03-07T20:59:40Z

Topic 1: Encryption

← Older revision		Revision as of 16:59, 7 March 2012
Line 56:		Line 56:
	*No (all respondents)... I think we've identified a real gap in the literature.		*No (all respondents)... I think we've identified a real gap in the literature.

			===Attempt at Decision Analysis===
	~~----~~


	~~'''~~Attempt at Decision Analysis~~'''~~

	1) Theoretically, maintenance of keys is straightforward although complex: e.g.		1) Theoretically, maintenance of keys is straightforward although complex: e.g.

Jbailey at 20:58, 7 March 2012

2012-03-07T20:58:52Z

← Older revision		Revision as of 16:58, 7 March 2012
Line 1:		Line 1:

	==Statement of Purpose==		==Statement of Purpose==
	The Infrastructure Working Group, in February 2012, initiated a series of open conversations on detailed aspects of preservation storage. These conversations are conducted over the listserv and each topic is discussed over the course of 2-3 weeks. A list of future, potential discussion topics is maintained at bottom and can be augmented by group members. This page serves to capture the content of those conversations for further elaboration by group members.		The Infrastructure Working Group, in February 2012, initiated a series of open conversations on detailed aspects of preservation storage. These conversations are conducted over the listserv and each topic is discussed over the course of 2-3 weeks. A list of future, potential discussion topics is maintained at bottom and can be augmented by group members. This page serves to capture the content of those conversations for further elaboration by group members.

			==Topic 2: Compression==

	==Topic 1: Encryption==		==Topic 1: Encryption==
Line 90:		Line 91:
	Addendum: In addition to loss of access to content because of loss of proprietary encryption technology/knowledge, would it also not be a possible threat that the entire PKI infrastructure could change over a long enough period of time? That, for example, current mechanisms for obtaining keys and revocation lists etc. could become unused and unavailable?		Addendum: In addition to loss of access to content because of loss of proprietary encryption technology/knowledge, would it also not be a possible threat that the entire PKI infrastructure could change over a long enough period of time? That, for example, current mechanisms for obtaining keys and revocation lists etc. could become unused and unavailable?

	A selected bibliography ~~from~~ Micah Altman:		===A selected bibliography===

			(From Micah Altman)

	While PKI isn't my area of research here are some relevant articles from a shallow dive:		While PKI isn't my area of research here are some relevant articles from a shallow dive:
Line 107:		Line 110:

	(And 150+ citing articles: http://scholar.google.com/scholar?hl=en&lr=&cites=49628270739110236&um=1&ie=UTF-8&ei=zjRNT_K8BYuD0QGlzon3Ag&sa=X&oi=science_links&ct=sl-citedby&resnum=3&ved=0CD4QzgIwAg)		(And 150+ citing articles: http://scholar.google.com/scholar?hl=en&lr=&cites=49628270739110236&um=1&ie=UTF-8&ei=zjRNT_K8BYuD0QGlzon3Ag&sa=X&oi=science_links&ct=sl-citedby&resnum=3&ved=0CD4QzgIwAg)

			Although weighted towards "key escrow" solutions they seem to have substantial relevance to long term access.

	==Potential Future Discussion Topics==		==Potential Future Discussion Topics==

Jbailey: /* Topic 1: Encryption */

2012-03-07T20:53:32Z

Topic 1: Encryption

← Older revision		Revision as of 16:53, 7 March 2012
Line 92:		Line 92:
	A selected bibliography from Micah Altman:		A selected bibliography from Micah Altman:

	While PKI isn't my area of research here are some relevant articles from a shallow dive:		While PKI isn't my area of research here are some relevant articles from a shallow dive:

	"Strategies for Ensuring Data Accessibility When Cryptographic Keys Are Lost"		"Strategies for Ensuring Data Accessibility When Cryptographic Keys Are Lost"