DLF Wiki - User contributions [en]

NDSA:March 29, 2013 Call

2013-03-29T22:25:39Z

Chris Dietrich:

Back to [[NDSA:Standards_and_Best_Practices_Working_Group | Standards Working Group Main Page]]

Back to [[NDSA:PDF_Exploration | PDF Exploration Page]]

=Agenda=
Discussion with Stephen Levenson, an IT Specialist for Policy and Planning at the Office of the US Courts and the chair of the PDF/A working group.

=Participants=

Don Chalfant, Kate Murray, Sheila Morrissey, Kevin DeVorsey, Chris Dietrich, Carl Fleischhauer, Stephen Levenson, Butch Lazorchak

=Meeting Notes=
*A rough transcription of our conversation*

Stephen Levenson discussing how the PDF/A family of standards are being implemented by the ISO community.

Steve: PDF/A-3 doesn't necessarily replace A-1 or A-2. Should be able to use a PDF/A-1 file 30 years from now. The methodology used should not change for rendering these files in the future.

PDF/A movement highly influenced by manufacturers, now PDF/A center, dominated by the Germans. Had many use cases for instances where creators wanted to include the original files wihtin a PDF/A document.

Brazilian government wanted to preserve their material as XML but XML wasn't trusted by users because of complexity. Wanted to make a more presentation-ready format but didn't want to throw away the XML.

U.S. Courts, bankruptcy court, claims, when an individual goes into court, and the claims are laid out, in order for someone to assert the claim, we print a document for them. That's what they bring back to court to assert their claim. Ginny Mae started this and Mastercard is also movin gon this. We get the PDF but then have to reenter the data from this doucment in their case management systems.

We're putting an XML output of what the claim represents inside the PDF document. Ginny Mae's automated processes work on this XML.

Down with PDF/A-3 we have downstream functions that leverage the inner materials. Adobe's server product does not currently output A-3 files.

Chris: How will hidden content be protected from certain readers of the document.

Steve: For A-3 you can still include the information as "private data" that would make it hidden. Conforming reader would recognize it as A-3 and set up an additional dialog.

Caroline: we're in the position of having to preserve files that somebody else created. We need tools to characterize files. Some PDF/A-3 files may not have any embedded content so they'd actually behave like a PDF/A-2.

Steve: We'd have to talk to the developers about that. There is a vendor that is looking to create an independent service to validate the writers to ensure that they are actually complying with the standards. The software would have to get certified that it works. Then we'd actually have validators at ingestion. We have to get somebody interested in creating this software as a business and we think we finally have somebody who will do this.

DOD standard 5015.2 that says if you're going to be a document management system you have to do certain things. Has to be sent to Fort Huachuca in AZ to a testing center to ensure that it conforms to DOD 5015.2. And this validator would do the same kind of thing.

right now we have no validator that says a Word document is actually a word document. There are a lot of bad writers out there.

Kevin: we're working on policy and guidance side. We ask people to keep temporary, permanent and non-record material separate from each other. Does PDF/A-3 run counter to that? Might encourage people to mix record and non-record material together in the same file?

Steve: If there's a relationship, don't you need that for provenance information?

Kevin: we need to educate our folks.

Steve: We've been dealing with current technologies on these things and who knows what technology will afford us in the future. We'll be able to hedge our bets.

Sheila: Question to me is what is the relationship between the PDF/A-3 container and the embedded XML, that is, in the Brazilian example, which one has the force of law? And how do you ensure that they say the same thing? In Germany they're planing on using this for commerce and the validation of invoices, but processing thing in the mass pragmatically means that you're going to look at one or the other. What warrant is there that they're going to stay the same way. The Germans said that the "embedded content" has no standing. Only the archival version has standing."

Steve: I can talk about legal. We would assume that the entire document was the legal evidence. In the case of dual content, according to the "best evidence" rule. If the company put a courtesy copy inside and that it's the main document that is their record. For example, if you requested an invoice and you received what you might see in a PDF versus XML data, then that's the evidence.

Folks coming in to a reading room. We may have to set up rules at ingestion, they could either strip it out and put the file on a diet and store the other content. We, in the committee, didn't want to dictate to preservationists how to do their job.

If NARA said they didn't want a part of A-3 then the agencies shouldn't store in A-3. In our Pacer system you can pull down a PDF document but stored inside is an MP3 files that allows you to understand the provenance a little more. The PDF is the metadata around the MP3 file. these are temporary records, so it's not the same issues.

Chris: So the PDF is acting as a manifest for the MP3 file. No validators, if someone if processing a bunch of files into PDF and embedding an XML version. Something could go wrong and you embed the wrong versions of the XML. There's not way to validate the right coordinated content.

Steve: archivists are going to have to get more involved in advising creators on the types of files they create.

PDF/E and A are essentially the same, they just couldn't find an open codec for rendering 3D documents.

PDF/A is pretty much asleep now, making sure we keep up with the latest version of the PDF specification.

Caroline:

----

Chris Dietrich's Notes:
PDF/A-3 Use Cases 20130329

NDSA PDF/A-3 work group discuss (positive) use cases with Stephen Levenson of the U.S. Courts

PDF/A has 3 versions: No versions (A-1, A-2, or-A3) will be replaced by subsequent versions reducing the need for migration

• PDF/A-1 = ???? [didn’t capture what makes a PDF/A-1 different from a PDF]

• PDF/A-2 = PDF/A-1 + ability to embed a second PDF/A-2 copy of the same document(?) (recursive?)

• PDF/A-3 = PDF/A-2 + any file embedded

• Genesis for A-3 was to include other types like XML along with the PDF (not just PDF/A-2)

• Private Data section in A-2:

• Use Case: Brazil uses to embed other data like XML version of the file (legal docs)

• Is/can be protected from view

• Was the genesis for creating A-3 to store other data not protected from view (?)

• Allows for human- and machine-readable versions of a document in the same container

• Viewing software (i.e. Adobe Reader et al.) will not render additional content but will notify consumer that additional content exists

All three versions are PDFs, no need to distinguish between the three versions except by the presentation interface which should be designed to detect and display embedded additional content

Currently no validators to ensure that any particular file (PDF, .doc, etc.) is actually what the extension indicates or is high-quality (e.g. some PDF generators are better than others)

• There is a vendor looking to create an ISO-compliant PDF validator which could be used by repositories to validate incoming PDF files (still years away)

Ability to embed XML allows both human- and machine-readable versions of a document in the same container

• Useful for automated ingest and presentation by repositories

Question arises which of the two versions (PDF or XML) is the authoritative version

• Use-case: PACER system for courtroom audio stored as MP3 and packaged in PDF/A-3

• PDF/A-3 acts as the metadata/manifest for the embedded recording

• Recordings are access copies, not high-quality originals for long-term preservation/archiving

• These are temp records, disposed of after 5 years

NDSA:March 29, 2013 Call

2013-03-29T22:25:18Z

Chris Dietrich:

NDSA:March 29, 2013 Call

2013-03-29T22:24:38Z

Chris Dietrich:

Back to [[NDSA:Standards_and_Best_Practices_Working_Group | Standards Working Group Main Page]]

Back to [[NDSA:PDF_Exploration | PDF Exploration Page]]

=Agenda=
Discussion with Stephen Levenson, an IT Specialist for Policy and Planning at the Office of the US Courts and the chair of the PDF/A working group.

=Participants=

Don Chalfant, Kate Murray, Sheila Morrissey, Kevin DeVorsey, Chris Dietrich, Carl Fleischhauer, Stephen Levenson, Butch Lazorchak

=Meeting Notes=
*A rough transcription of our conversation*

Stephen Levenson discussing how the PDF/A family of standards are being implemented by the ISO community.

Steve: PDF/A-3 doesn't necessarily replace A-1 or A-2. Should be able to use a PDF/A-1 file 30 years from now. The methodology used should not change for rendering these files in the future.

PDF/A movement highly influenced by manufacturers, now PDF/A center, dominated by the Germans. Had many use cases for instances where creators wanted to include the original files wihtin a PDF/A document.

Brazilian government wanted to preserve their material as XML but XML wasn't trusted by users because of complexity. Wanted to make a more presentation-ready format but didn't want to throw away the XML.

U.S. Courts, bankruptcy court, claims, when an individual goes into court, and the claims are laid out, in order for someone to assert the claim, we print a document for them. That's what they bring back to court to assert their claim. Ginny Mae started this and Mastercard is also movin gon this. We get the PDF but then have to reenter the data from this doucment in their case management systems.

We're putting an XML output of what the claim represents inside the PDF document. Ginny Mae's automated processes work on this XML.

Down with PDF/A-3 we have downstream functions that leverage the inner materials. Adobe's server product does not currently output A-3 files.

Chris: How will hidden content be protected from certain readers of the document.

Steve: For A-3 you can still include the information as "private data" that would make it hidden. Conforming reader would recognize it as A-3 and set up an additional dialog.

Caroline: we're in the position of having to preserve files that somebody else created. We need tools to characterize files. Some PDF/A-3 files may not have any embedded content so they'd actually behave like a PDF/A-2.

Steve: We'd have to talk to the developers about that. There is a vendor that is looking to create an independent service to validate the writers to ensure that they are actually complying with the standards. The software would have to get certified that it works. Then we'd actually have validators at ingestion. We have to get somebody interested in creating this software as a business and we think we finally have somebody who will do this.

DOD standard 5015.2 that says if you're going to be a document management system you have to do certain things. Has to be sent to Fort Huachuca in AZ to a testing center to ensure that it conforms to DOD 5015.2. And this validator would do the same kind of thing.

right now we have no validator that says a Word document is actually a word document. There are a lot of bad writers out there.

Kevin: we're working on policy and guidance side. We ask people to keep temporary, permanent and non-record material separate from each other. Does PDF/A-3 run counter to that? Might encourage people to mix record and non-record material together in the same file?

Steve: If there's a relationship, don't you need that for provenance information?

Kevin: we need to educate our folks.

Steve: We've been dealing with current technologies on these things and who knows what technology will afford us in the future. We'll be able to hedge our bets.

Sheila: Question to me is what is the relationship between the PDF/A-3 container and the embedded XML, that is, in the Brazilian example, which one has the force of law? And how do you ensure that they say the same thing? In Germany they're planing on using this for commerce and the validation of invoices, but processing thing in the mass pragmatically means that you're going to look at one or the other. What warrant is there that they're going to stay the same way. The Germans said that the "embedded content" has no standing. Only the archival version has standing."

Steve: I can talk about legal. We would assume that the entire document was the legal evidence. In the case of dual content, according to the "best evidence" rule. If the company put a courtesy copy inside and that it's the main document that is their record. For example, if you requested an invoice and you received what you might see in a PDF versus XML data, then that's the evidence.

Folks coming in to a reading room. We may have to set up rules at ingestion, they could either strip it out and put the file on a diet and store the other content. We, in the committee, didn't want to dictate to preservationists how to do their job.

If NARA said they didn't want a part of A-3 then the agencies shouldn't store in A-3. In our Pacer system you can pull down a PDF document but stored inside is an MP3 files that allows you to understand the provenance a little more. The PDF is the metadata around the MP3 file. these are temporary records, so it's not the same issues.

Chris: So the PDF is acting as a manifest for the MP3 file. No validators, if someone if processing a bunch of files into PDF and embedding an XML version. Something could go wrong and you embed the wrong versions of the XML. There's not way to validate the right coordinated content.

Steve: archivists are going to have to get more involved in advising creators on the types of files they create.

PDF/E and A are essentially the same, they just couldn't find an open codec for rendering 3D documents.

PDF/A is pretty much asleep now, making sure we keep up with the latest version of the PDF specification.

Caroline:

----

Chris Dietrich's Notes:
PDF/A-3 Use Cases 20130329

NDSA PDF/A-3 work group discuss (positive) use cases with Stephen Levenson of the U.S. Courts

PDF/A has 3 versions: No versions (A-1, A-2, or-A3) will be replaced by subsequent versions reducing the need for migration

• PDF/A-1 = ???? [didn’t capture what makes a PDF/A-1 different from a PDF]

• PDF/A-2 = PDF/A-1 + ability to embed a second PDF/A-2 copy of the same document(?) (recursive?)

• PDF/A-3 = PDF/A-2 + any file embedded

• Genesis for A-3 was to include other types like XML along with the PDF (not just PDF/A-2)

• Private Data section in A-2:

• Use Case: Brazil uses to embed other data like XML version of the file (legal docs)

• Is/can be protected from view

• Was the genesis for creating A-3 to store other data not protected from view (?)

• Allows for human- and machine-readable versions of a document in the same container

• Viewing software (i.e. Adobe Reader et al.) will not render additional content but will notify consumer that additional content exists

All three versions are PDFs, no need to distinguish between the three versions except by the presentation interface which should be designed to detect and display embedded additional content

Currently no validators to ensure that any particular file (PDF, .doc, etc.) is actually what the extension indicates or is high-quality (e.g. some PDF generators are better than others)

There is a vendor looking to create an ISO-compliant PDF validator which could be used by repositories to validate incoming PDF files (still years away)

Ability to embed XML allows both human- and machine-readable versions of a document in the same container

• Useful for automated ingest and presentation by repositories

Question arises which of the two versions (PDF or XML) is the authoritative version

• Use-case: PACER system for courtroom audio stored as MP3 and packaged in PDF/A-3

• PDF/A-3 acts as the metadata/manifest for the embedded recording

• Recordings are access copies, not high-quality originals for long-term preservation/archiving

• These are temp records, disposed of after 5 years

I think I missed another use case here…

NDSA:March 29, 2013 Call

2013-03-29T22:23:54Z

Chris Dietrich:

Back to [[NDSA:Standards_and_Best_Practices_Working_Group | Standards Working Group Main Page]]

Back to [[NDSA:PDF_Exploration | PDF Exploration Page]]

=Agenda=
Discussion with Stephen Levenson, an IT Specialist for Policy and Planning at the Office of the US Courts and the chair of the PDF/A working group.

=Participants=

Don Chalfant, Kate Murray, Sheila Morrissey, Kevin DeVorsey, Chris Dietrich, Carl Fleischhauer, Stephen Levenson, Butch Lazorchak

=Meeting Notes=
*A rough transcription of our conversation*

Stephen Levenson discussing how the PDF/A family of standards are being implemented by the ISO community.

Steve: PDF/A-3 doesn't necessarily replace A-1 or A-2. Should be able to use a PDF/A-1 file 30 years from now. The methodology used should not change for rendering these files in the future.

PDF/A movement highly influenced by manufacturers, now PDF/A center, dominated by the Germans. Had many use cases for instances where creators wanted to include the original files wihtin a PDF/A document.

Brazilian government wanted to preserve their material as XML but XML wasn't trusted by users because of complexity. Wanted to make a more presentation-ready format but didn't want to throw away the XML.

U.S. Courts, bankruptcy court, claims, when an individual goes into court, and the claims are laid out, in order for someone to assert the claim, we print a document for them. That's what they bring back to court to assert their claim. Ginny Mae started this and Mastercard is also movin gon this. We get the PDF but then have to reenter the data from this doucment in their case management systems.

We're putting an XML output of what the claim represents inside the PDF document. Ginny Mae's automated processes work on this XML.

Down with PDF/A-3 we have downstream functions that leverage the inner materials. Adobe's server product does not currently output A-3 files.

Chris: How will hidden content be protected from certain readers of the document.

Steve: For A-3 you can still include the information as "private data" that would make it hidden. Conforming reader would recognize it as A-3 and set up an additional dialog.

Caroline: we're in the position of having to preserve files that somebody else created. We need tools to characterize files. Some PDF/A-3 files may not have any embedded content so they'd actually behave like a PDF/A-2.

Steve: We'd have to talk to the developers about that. There is a vendor that is looking to create an independent service to validate the writers to ensure that they are actually complying with the standards. The software would have to get certified that it works. Then we'd actually have validators at ingestion. We have to get somebody interested in creating this software as a business and we think we finally have somebody who will do this.

DOD standard 5015.2 that says if you're going to be a document management system you have to do certain things. Has to be sent to Fort Huachuca in AZ to a testing center to ensure that it conforms to DOD 5015.2. And this validator would do the same kind of thing.

right now we have no validator that says a Word document is actually a word document. There are a lot of bad writers out there.

Kevin: we're working on policy and guidance side. We ask people to keep temporary, permanent and non-record material separate from each other. Does PDF/A-3 run counter to that? Might encourage people to mix record and non-record material together in the same file?

Steve: If there's a relationship, don't you need that for provenance information?

Kevin: we need to educate our folks.

Steve: We've been dealing with current technologies on these things and who knows what technology will afford us in the future. We'll be able to hedge our bets.

Sheila: Question to me is what is the relationship between the PDF/A-3 container and the embedded XML, that is, in the Brazilian example, which one has the force of law? And how do you ensure that they say the same thing? In Germany they're planing on using this for commerce and the validation of invoices, but processing thing in the mass pragmatically means that you're going to look at one or the other. What warrant is there that they're going to stay the same way. The Germans said that the "embedded content" has no standing. Only the archival version has standing."

Steve: I can talk about legal. We would assume that the entire document was the legal evidence. In the case of dual content, according to the "best evidence" rule. If the company put a courtesy copy inside and that it's the main document that is their record. For example, if you requested an invoice and you received what you might see in a PDF versus XML data, then that's the evidence.

Folks coming in to a reading room. We may have to set up rules at ingestion, they could either strip it out and put the file on a diet and store the other content. We, in the committee, didn't want to dictate to preservationists how to do their job.

If NARA said they didn't want a part of A-3 then the agencies shouldn't store in A-3. In our Pacer system you can pull down a PDF document but stored inside is an MP3 files that allows you to understand the provenance a little more. The PDF is the metadata around the MP3 file. these are temporary records, so it's not the same issues.

Chris: So the PDF is acting as a manifest for the MP3 file. No validators, if someone if processing a bunch of files into PDF and embedding an XML version. Something could go wrong and you embed the wrong versions of the XML. There's not way to validate the right coordinated content.

Steve: archivists are going to have to get more involved in advising creators on the types of files they create.

PDF/E and A are essentially the same, they just couldn't find an open codec for rendering 3D documents.

PDF/A is pretty much asleep now, making sure we keep up with the latest version of the PDF specification.

Caroline:

----

Chris Dietrich's Notes:
PDF/A-3 Use Cases 20130329

NDSA PDF/A-3 work group discuss (positive) use cases with Stephen Levenson of the U.S. Courts

PDF/A has 3 versions: No versions (A-1, A-2, or-A3) will be replaced by subsequent versions reducing the need for migration

• PDF/A-1 = ???? [didn’t capture what makes a PDF/A-1 different from a PDF]

• PDF/A-2 = PDF/A-1 + ability to embed a second PDF/A-2 copy of the same document(?) (recursive?)

• PDF/A-3 = PDF/A-2 + any file embedded

• Genesis for A-3 was to include other types like XML along with the PDF (not just PDF/A-2)

• Private Data section in A-2:

• Use Case: Brazil uses to embed other data like XML version of the file (legal docs)

• Is/can be protected from view

• Was the genesis for creating A-3 to store other data not protected from view (?)

• Allows for human- and machine-readable versions of a document in the same container

• Viewing software (i.e. Adobe Reader et al.) will not render additional content but will notify consumer that additional content exists

All three versions are PDFs, no need to distinguish between the three versions except by the presentation interface which should be designed to detect and display embedded additional content

Currently no validators to ensure that any particular file (PDF, .doc, etc.) is actually what the extension indicates or is high-quality (e.g. some PDF generators are better than others)

There is a vendor looking to create an ISO-compliant PDF validator which could be used by repositories to validate incoming PDF files (still years away)

Ability to embed XML allows both human- and machine-readable versions of a document in the same container
• Useful for automated ingest and presentation by repositories

Question arises which of the two versions (PDF or XML) is the authoritative version

• Use-case: PACER system for courtroom audio stored as MP3 and packaged in PDF/A-3

• PDF/A-3 acts as the metadata/manifest for the embedded recording

• Recordings are access copies, not high-quality originals for long-term preservation/archiving

• These are temp records, disposed of after 5 years

I think I missed another use case here…

NDSA:March 29, 2013 Call

2013-03-29T22:23:12Z

Chris Dietrich:

Back to [[NDSA:Standards_and_Best_Practices_Working_Group | Standards Working Group Main Page]]

Back to [[NDSA:PDF_Exploration | PDF Exploration Page]]

=Agenda=
Discussion with Stephen Levenson, an IT Specialist for Policy and Planning at the Office of the US Courts and the chair of the PDF/A working group.

=Participants=

Don Chalfant, Kate Murray, Sheila Morrissey, Kevin DeVorsey, Chris Dietrich, Carl Fleischhauer, Stephen Levenson, Butch Lazorchak

=Meeting Notes=
*A rough transcription of our conversation*

Stephen Levenson discussing how the PDF/A family of standards are being implemented by the ISO community.

Steve: PDF/A-3 doesn't necessarily replace A-1 or A-2. Should be able to use a PDF/A-1 file 30 years from now. The methodology used should not change for rendering these files in the future.

PDF/A movement highly influenced by manufacturers, now PDF/A center, dominated by the Germans. Had many use cases for instances where creators wanted to include the original files wihtin a PDF/A document.

Brazilian government wanted to preserve their material as XML but XML wasn't trusted by users because of complexity. Wanted to make a more presentation-ready format but didn't want to throw away the XML.

U.S. Courts, bankruptcy court, claims, when an individual goes into court, and the claims are laid out, in order for someone to assert the claim, we print a document for them. That's what they bring back to court to assert their claim. Ginny Mae started this and Mastercard is also movin gon this. We get the PDF but then have to reenter the data from this doucment in their case management systems.

We're putting an XML output of what the claim represents inside the PDF document. Ginny Mae's automated processes work on this XML.

Down with PDF/A-3 we have downstream functions that leverage the inner materials. Adobe's server product does not currently output A-3 files.

Chris: How will hidden content be protected from certain readers of the document.

Steve: For A-3 you can still include the information as "private data" that would make it hidden. Conforming reader would recognize it as A-3 and set up an additional dialog.

Caroline: we're in the position of having to preserve files that somebody else created. We need tools to characterize files. Some PDF/A-3 files may not have any embedded content so they'd actually behave like a PDF/A-2.

Steve: We'd have to talk to the developers about that. There is a vendor that is looking to create an independent service to validate the writers to ensure that they are actually complying with the standards. The software would have to get certified that it works. Then we'd actually have validators at ingestion. We have to get somebody interested in creating this software as a business and we think we finally have somebody who will do this.

DOD standard 5015.2 that says if you're going to be a document management system you have to do certain things. Has to be sent to Fort Huachuca in AZ to a testing center to ensure that it conforms to DOD 5015.2. And this validator would do the same kind of thing.

right now we have no validator that says a Word document is actually a word document. There are a lot of bad writers out there.

Kevin: we're working on policy and guidance side. We ask people to keep temporary, permanent and non-record material separate from each other. Does PDF/A-3 run counter to that? Might encourage people to mix record and non-record material together in the same file?

Steve: If there's a relationship, don't you need that for provenance information?

Kevin: we need to educate our folks.

Steve: We've been dealing with current technologies on these things and who knows what technology will afford us in the future. We'll be able to hedge our bets.

Sheila: Question to me is what is the relationship between the PDF/A-3 container and the embedded XML, that is, in the Brazilian example, which one has the force of law? And how do you ensure that they say the same thing? In Germany they're planing on using this for commerce and the validation of invoices, but processing thing in the mass pragmatically means that you're going to look at one or the other. What warrant is there that they're going to stay the same way. The Germans said that the "embedded content" has no standing. Only the archival version has standing."

Steve: I can talk about legal. We would assume that the entire document was the legal evidence. In the case of dual content, according to the "best evidence" rule. If the company put a courtesy copy inside and that it's the main document that is their record. For example, if you requested an invoice and you received what you might see in a PDF versus XML data, then that's the evidence.

Folks coming in to a reading room. We may have to set up rules at ingestion, they could either strip it out and put the file on a diet and store the other content. We, in the committee, didn't want to dictate to preservationists how to do their job.

If NARA said they didn't want a part of A-3 then the agencies shouldn't store in A-3. In our Pacer system you can pull down a PDF document but stored inside is an MP3 files that allows you to understand the provenance a little more. The PDF is the metadata around the MP3 file. these are temporary records, so it's not the same issues.

Chris: So the PDF is acting as a manifest for the MP3 file. No validators, if someone if processing a bunch of files into PDF and embedding an XML version. Something could go wrong and you embed the wrong versions of the XML. There's not way to validate the right coordinated content.

Steve: archivists are going to have to get more involved in advising creators on the types of files they create.

PDF/E and A are essentially the same, they just couldn't find an open codec for rendering 3D documents.

PDF/A is pretty much asleep now, making sure we keep up with the latest version of the PDF specification.

Caroline:

----

Chris Dietrich's Notes:
PDF/A-3 Use Cases 20130329

NDSA PDF/A-3 work group discuss (positive) use cases with Stephen Levenson of the U.S. Courts

PDF/A has 3 versions: No versions (A-1, A-2, or-A3) will be replaced by subsequent versions reducing the need for migration

• PDF/A-1 = ???? [didn’t capture what makes a PDF/A-1 different from a PDF]

• PDF/A-2 = PDF/A-1 + ability to embed a second PDF/A-2 copy of the same document(?) (recursive?)

• PDF/A-3 = PDF/A-2 + any file embedded

• Genesis for A-3 was to include other types like XML along with the PDF (not just PDF/A-2)

• Private Data section in A-2:

• Use Case: Brazil uses to embed other data like XML version of the file (legal docs)

• Is/can be protected from view

• Was the genesis for creating A-3 to store other data not protected from view (?)

• Allows for human- and machine-readable versions of a document in the same container

• Viewing software (i.e. Adobe Reader et al.) will not render additional content but will notify consumer that additional content exists

All three versions are PDFs, no need to distinguish between the three versions except by the presentation interface which should be designed to detect and display embedded additional content

Currently no validators to ensure that any particular file (PDF, .doc, etc.) is actually what the extension indicates or is high-quality (e.g. some PDF generators are better than others)

There is a vendor looking to create an ISO-compliant PDF validator which could be used by repositories to validate incoming PDF files (still years away)

Ability to embed XML allows both human- and machine-readable versions of a document in the same container
Useful for automated ingest and presentation by repositories

Question arises which of the two versions (PDF or XML) is the authoritative version
-Use-case: PACER system for courtroom audio stored as MP3 and packaged in PDF/A-3
--PDF/A-3 acts as the metadata/manifest for the embedded recording
--Recordings are access copies, not high-quality originals for long-term preservation/archiving
--These are temp records, disposed of after 5 years

I think I missed another use case here…

NDSA:March 29, 2013 Call

2013-03-29T22:22:41Z

Chris Dietrich:

Back to [[NDSA:Standards_and_Best_Practices_Working_Group | Standards Working Group Main Page]]

Back to [[NDSA:PDF_Exploration | PDF Exploration Page]]

=Agenda=
Discussion with Stephen Levenson, an IT Specialist for Policy and Planning at the Office of the US Courts and the chair of the PDF/A working group.

=Participants=

Don Chalfant, Kate Murray, Sheila Morrissey, Kevin DeVorsey, Chris Dietrich, Carl Fleischhauer, Stephen Levenson, Butch Lazorchak

=Meeting Notes=
*A rough transcription of our conversation*

Stephen Levenson discussing how the PDF/A family of standards are being implemented by the ISO community.

Steve: PDF/A-3 doesn't necessarily replace A-1 or A-2. Should be able to use a PDF/A-1 file 30 years from now. The methodology used should not change for rendering these files in the future.

PDF/A movement highly influenced by manufacturers, now PDF/A center, dominated by the Germans. Had many use cases for instances where creators wanted to include the original files wihtin a PDF/A document.

Brazilian government wanted to preserve their material as XML but XML wasn't trusted by users because of complexity. Wanted to make a more presentation-ready format but didn't want to throw away the XML.

U.S. Courts, bankruptcy court, claims, when an individual goes into court, and the claims are laid out, in order for someone to assert the claim, we print a document for them. That's what they bring back to court to assert their claim. Ginny Mae started this and Mastercard is also movin gon this. We get the PDF but then have to reenter the data from this doucment in their case management systems.

We're putting an XML output of what the claim represents inside the PDF document. Ginny Mae's automated processes work on this XML.

Down with PDF/A-3 we have downstream functions that leverage the inner materials. Adobe's server product does not currently output A-3 files.

Chris: How will hidden content be protected from certain readers of the document.

Steve: For A-3 you can still include the information as "private data" that would make it hidden. Conforming reader would recognize it as A-3 and set up an additional dialog.

Caroline: we're in the position of having to preserve files that somebody else created. We need tools to characterize files. Some PDF/A-3 files may not have any embedded content so they'd actually behave like a PDF/A-2.

Steve: We'd have to talk to the developers about that. There is a vendor that is looking to create an independent service to validate the writers to ensure that they are actually complying with the standards. The software would have to get certified that it works. Then we'd actually have validators at ingestion. We have to get somebody interested in creating this software as a business and we think we finally have somebody who will do this.

DOD standard 5015.2 that says if you're going to be a document management system you have to do certain things. Has to be sent to Fort Huachuca in AZ to a testing center to ensure that it conforms to DOD 5015.2. And this validator would do the same kind of thing.

right now we have no validator that says a Word document is actually a word document. There are a lot of bad writers out there.

Kevin: we're working on policy and guidance side. We ask people to keep temporary, permanent and non-record material separate from each other. Does PDF/A-3 run counter to that? Might encourage people to mix record and non-record material together in the same file?

Steve: If there's a relationship, don't you need that for provenance information?

Kevin: we need to educate our folks.

Steve: We've been dealing with current technologies on these things and who knows what technology will afford us in the future. We'll be able to hedge our bets.

Sheila: Question to me is what is the relationship between the PDF/A-3 container and the embedded XML, that is, in the Brazilian example, which one has the force of law? And how do you ensure that they say the same thing? In Germany they're planing on using this for commerce and the validation of invoices, but processing thing in the mass pragmatically means that you're going to look at one or the other. What warrant is there that they're going to stay the same way. The Germans said that the "embedded content" has no standing. Only the archival version has standing."

Steve: I can talk about legal. We would assume that the entire document was the legal evidence. In the case of dual content, according to the "best evidence" rule. If the company put a courtesy copy inside and that it's the main document that is their record. For example, if you requested an invoice and you received what you might see in a PDF versus XML data, then that's the evidence.

Folks coming in to a reading room. We may have to set up rules at ingestion, they could either strip it out and put the file on a diet and store the other content. We, in the committee, didn't want to dictate to preservationists how to do their job.

If NARA said they didn't want a part of A-3 then the agencies shouldn't store in A-3. In our Pacer system you can pull down a PDF document but stored inside is an MP3 files that allows you to understand the provenance a little more. The PDF is the metadata around the MP3 file. these are temporary records, so it's not the same issues.

Chris: So the PDF is acting as a manifest for the MP3 file. No validators, if someone if processing a bunch of files into PDF and embedding an XML version. Something could go wrong and you embed the wrong versions of the XML. There's not way to validate the right coordinated content.

Steve: archivists are going to have to get more involved in advising creators on the types of files they create.

PDF/E and A are essentially the same, they just couldn't find an open codec for rendering 3D documents.

PDF/A is pretty much asleep now, making sure we keep up with the latest version of the PDF specification.

Caroline:

----

Chris Dietrich's Notes:
PDF/A-3 Use Cases 20130329

NDSA PDF/A-3 work group discuss (positive) use cases with Stephen Levenson of the U.S. Courts

PDF/A has 3 versions: No versions (A-1, A-2, or-A3) will be replaced by subsequent versions reducing the need for migration
• PDF/A-1 = ???? [didn’t capture what makes a PDF/A-1 different from a PDF]
• PDF/A-2 = PDF/A-1 + ability to embed a second PDF/A-2 copy of the same document(?) (recursive?)
• PDF/A-3 = PDF/A-2 + any file embedded
• Genesis for A-3 was to include other types like XML along with the PDF (not just PDF/A-2)
• Private Data section in A-2:
• Use Case: Brazil uses to embed other data like XML version of the file (legal docs)
• Is/can be protected from view
• Was the genesis for creating A-3 to store other data not protected from view (?)
• Allows for human- and machine-readable versions of a document in the same container
• Viewing software (i.e. Adobe Reader et al.) will not render additional content but will notify consumer that additional content exists

All three versions are PDFs, no need to distinguish between the three versions except by the presentation interface which should be designed to detect and display embedded additional content

Currently no validators to ensure that any particular file (PDF, .doc, etc.) is actually what the extension indicates or is high-quality (e.g. some PDF generators are better than others)

There is a vendor looking to create an ISO-compliant PDF validator which could be used by repositories to validate incoming PDF files (still years away)

Ability to embed XML allows both human- and machine-readable versions of a document in the same container
Useful for automated ingest and presentation by repositories

Question arises which of the two versions (PDF or XML) is the authoritative version
-Use-case: PACER system for courtroom audio stored as MP3 and packaged in PDF/A-3
--PDF/A-3 acts as the metadata/manifest for the embedded recording
--Recordings are access copies, not high-quality originals for long-term preservation/archiving
--These are temp records, disposed of after 5 years

I think I missed another use case here…

NDSA:March 29, 2013 Call

2013-03-29T22:19:12Z

Chris Dietrich:

Back to [[NDSA:Standards_and_Best_Practices_Working_Group | Standards Working Group Main Page]]

Back to [[NDSA:PDF_Exploration | PDF Exploration Page]]

=Agenda=
Discussion with Stephen Levenson, an IT Specialist for Policy and Planning at the Office of the US Courts and the chair of the PDF/A working group.

=Participants=

Don Chalfant, Kate Murray, Sheila Morrissey, Kevin DeVorsey, Chris Dietrich, Carl Fleischhauer, Stephen Levenson, Butch Lazorchak

=Meeting Notes=
*A rough transcription of our conversation*

Stephen Levenson discussing how the PDF/A family of standards are being implemented by the ISO community.

Steve: PDF/A-3 doesn't necessarily replace A-1 or A-2. Should be able to use a PDF/A-1 file 30 years from now. The methodology used should not change for rendering these files in the future.

PDF/A movement highly influenced by manufacturers, now PDF/A center, dominated by the Germans. Had many use cases for instances where creators wanted to include the original files wihtin a PDF/A document.

Brazilian government wanted to preserve their material as XML but XML wasn't trusted by users because of complexity. Wanted to make a more presentation-ready format but didn't want to throw away the XML.

U.S. Courts, bankruptcy court, claims, when an individual goes into court, and the claims are laid out, in order for someone to assert the claim, we print a document for them. That's what they bring back to court to assert their claim. Ginny Mae started this and Mastercard is also movin gon this. We get the PDF but then have to reenter the data from this doucment in their case management systems.

We're putting an XML output of what the claim represents inside the PDF document. Ginny Mae's automated processes work on this XML.

Down with PDF/A-3 we have downstream functions that leverage the inner materials. Adobe's server product does not currently output A-3 files.

Chris: How will hidden content be protected from certain readers of the document.

Steve: For A-3 you can still include the information as "private data" that would make it hidden. Conforming reader would recognize it as A-3 and set up an additional dialog.

Caroline: we're in the position of having to preserve files that somebody else created. We need tools to characterize files. Some PDF/A-3 files may not have any embedded content so they'd actually behave like a PDF/A-2.

Steve: We'd have to talk to the developers about that. There is a vendor that is looking to create an independent service to validate the writers to ensure that they are actually complying with the standards. The software would have to get certified that it works. Then we'd actually have validators at ingestion. We have to get somebody interested in creating this software as a business and we think we finally have somebody who will do this.

DOD standard 5015.2 that says if you're going to be a document management system you have to do certain things. Has to be sent to Fort Huachuca in AZ to a testing center to ensure that it conforms to DOD 5015.2. And this validator would do the same kind of thing.

right now we have no validator that says a Word document is actually a word document. There are a lot of bad writers out there.

Kevin: we're working on policy and guidance side. We ask people to keep temporary, permanent and non-record material separate from each other. Does PDF/A-3 run counter to that? Might encourage people to mix record and non-record material together in the same file?

Steve: If there's a relationship, don't you need that for provenance information?

Kevin: we need to educate our folks.

Steve: We've been dealing with current technologies on these things and who knows what technology will afford us in the future. We'll be able to hedge our bets.

Sheila: Question to me is what is the relationship between the PDF/A-3 container and the embedded XML, that is, in the Brazilian example, which one has the force of law? And how do you ensure that they say the same thing? In Germany they're planing on using this for commerce and the validation of invoices, but processing thing in the mass pragmatically means that you're going to look at one or the other. What warrant is there that they're going to stay the same way. The Germans said that the "embedded content" has no standing. Only the archival version has standing."

Steve: I can talk about legal. We would assume that the entire document was the legal evidence. In the case of dual content, according to the "best evidence" rule. If the company put a courtesy copy inside and that it's the main document that is their record. For example, if you requested an invoice and you received what you might see in a PDF versus XML data, then that's the evidence.

Folks coming in to a reading room. We may have to set up rules at ingestion, they could either strip it out and put the file on a diet and store the other content. We, in the committee, didn't want to dictate to preservationists how to do their job.

If NARA said they didn't want a part of A-3 then the agencies shouldn't store in A-3. In our Pacer system you can pull down a PDF document but stored inside is an MP3 files that allows you to understand the provenance a little more. The PDF is the metadata around the MP3 file. these are temporary records, so it's not the same issues.

Chris: So the PDF is acting as a manifest for the MP3 file. No validators, if someone if processing a bunch of files into PDF and embedding an XML version. Something could go wrong and you embed the wrong versions of the XML. There's not way to validate the right coordinated content.

Steve: archivists are going to have to get more involved in advising creators on the types of files they create.

PDF/E and A are essentially the same, they just couldn't find an open codec for rendering 3D documents.

PDF/A is pretty much asleep now, making sure we keep up with the latest version of the PDF specification.

Caroline:

----

Chris Dietrich's Notes:
PDF/A-3 Use Cases 20130329

NDSA PDF/A-3 work group discuss (positive) use cases with Stephen Levenson of the U.S. Courts

PDF/A has 3 versions: No versions (A-1, A-2, or-A3) will be replaced by subsequent versions reducing the need for migration
-PDF/A-1 = ???? [didn’t capture what makes a PDF/A-1 different from a PDF]
-PDF/A-2 = PDF/A-1 + ability to embed a second PDF/A-2 copy of the same document(?) (recursive?)
-PDF/A-3 = PDF/A-2 + any file embedded
--Genesis for A-3 was to include other types like XML along with the PDF (not just PDF/A-2)
--Private Data section in A-2:
--Use Case: Brazil uses to embed other data like XML version of the file (legal docs)
--Is/can be protected from view
--Was the genesis for creating A-3 to store other data not protected from view (?)
--Allows for human- and machine-readable versions of a document in the same container
--Viewing software (i.e. Adobe Reader et al.) will not render additional content but will notify consumer that additional content exists

All three versions are PDFs, no need to distinguish between the three versions except by the presentation interface which should be designed to detect and display embedded additional content

Currently no validators to ensure that any particular file (PDF, .doc, etc.) is actually what the extension indicates or is high-quality (e.g. some PDF generators are better than others)

There is a vendor looking to create an ISO-compliant PDF validator which could be used by repositories to validate incoming PDF files (still years away)

Ability to embed XML allows both human- and machine-readable versions of a document in the same container
Useful for automated ingest and presentation by repositories

Question arises which of the two versions (PDF or XML) is the authoritative version
-Use-case: PACER system for courtroom audio stored as MP3 and packaged in PDF/A-3
--PDF/A-3 acts as the metadata/manifest for the embedded recording
--Recordings are access copies, not high-quality originals for long-term preservation/archiving
--These are temp records, disposed of after 5 years

I think I missed another use case here…

NDSA:PDF Exploration

2013-01-07T21:44:01Z

Chris Dietrich: /* PDF/A-3 Use Case Scenarios */

Back to [[NDSA:Standards_and_Best_Practices_Working_Group | Standards Working Group Main Page]]

==Title of Activity or Project==
NDSA PDF/A-3 Scoping Project

==One Sentence Description:==
NDSA PDF/A-3 Scoping Project working group members will research the pros and cons of using the PDF/A-3 standard as an all-purpose wrapper for various digital asset/media types including: textual, audio, video, photo, and GIS data.

==Statement of the Problem and Goals for Addressing the Problem:==
It is unclear whether PDF/A-3, which was designed to accommodate supplementary media files for text documents, is appropriate as a de facto normalization wrapper format for all media types. The goal is to develop guidelines for the appropriate use of PDF/A-3 with respect to different media types that includes both detailed technical information and a practical quick reference guide for end-users.

==Strategic Value of Activity:==
* Improve understanding of best practices for using PDF/A-3 in digital preservation activities
* Enhance consistency and improve long-term viability of digitally preserved content
* Provide guidance to those considering PDF/A-3 as a long-term archiving format

==Required Resources:==
* Time of working group members
* Publishing venue(s)
* Communication channels

==Roadmap:==
# Hold regular working group conference calls (monthly, between NDSA Standards WG calls)
# Draft document and review
# Invite broader NDSA member feedback
# Publish document (digitalpreservation.gov, others?)

==Dissemination of Knowledge:==
* Publish report on digitalpreservation.gov
* Write a blog post
* Announce on NDSA member organization communication channels
* Present at conferences that members (and non-members?) are attending

==Signifiers of Success and Outcomes:==
* Completed guidelines document published on digitalpreservation.gov
* Guidelines document referenced on related Wikipedia pages
* Guidelines in use or recommended by NDSA participating organizations or others
* Publication at other conferences/other journals

==Questions to Ask and Answer==
*Talk about background (what is pdf/a-3 and how is it different from earlier versions of PDF/A)
*Iterate categories of materials/use cases/concrete examples where it makes sense to use A-3 and other categories where it doesn't make sense. Example: if you're sending a video file don't put it in a PDF! If you had a certain kind of a journal article that had a static version of the spreadsheet in the doc but a malleable version embedded perhaps that argues for it.
*Risks to the format (scenarios in why this might be bad and why)
*Possibilities of the format (scenarios in why this might be good and why)
*Have list of defined terms in our document. How do these relate to the terms in the ISO spec. Leverage NDSA Levels of Preservation glossary. Link to glossary.

==PDF/A-3 Use Case Scenarios==
Add them here! We can create a separate page as necessary.
----
Example: Federal agency with a document management system puts an MPEG video file (and nothing else) into a PDF/A-3 file to store and then, later, to submit as an SIP (Submission Information Package) to NARA for long-term management.

Example: Publisher has a text-only article and puts it into a PDF/A-3 file, even though, in the past, the publisher used PDF/A-2. The article is then sent to library where it will be preserved for the long term.

Example: Publisher has an article that includes a complicated table, "frozen" in place, and puts it into a PDF/A-3 file, along with the Excel file from which the table was generated, in order to make it easier for a future researcher to have a malleable version of the table for use when writing another article on the same subject.

Example: Data creator has a digital map, a report, a database, digital photos, and detailed metadata that comprise a whole and wants to archive these together for the long-term.

==Members==
*Caroline Arms, Library of Congress (caar@loc.gov)
*Don Chalfant, NARA (Donald.Chalfant@nara.gov)
*Kevin DeVorsey, NARA (Kevin.DeVorsey@nara.gov)
*Chris Dietrich, National Park Service (chris_dietrich@nps.gov)
*Carl Fleischauer, Library of Congress (cfle@loc.gov)
*Butch Lazorchak, Library of Congress (wlaz@loc.gov)
*Sheila Morrissey, Ithaka (Sheila.Morrissey@ithaka.org)
*Kate Murrary, NARA (Kate.Murray1@nara.gov)

==Calls and Notes==

Call information:

*Call-in toll-free number (US/Canada): 866-469-3239
*Participant access code: 21408589

Next call: Tuesday Jan. 22, 2013, 2:00 P.M.

==Background Materials==

*[http://www.digitalpreservation.gov:8081/formats/fdd/fdd000360.shtml Library of Congress Sustainability of Digital Formats DRAFT PDF/A-3 site]
*[http://blogs.loc.gov/digitalpreservation/2012/11/all-in-embedded-files-in-pdfa/ Blog Post on PDF/A-3 on the Signal]
*[[NDSA:Media: TheNetworkIsTheFormat.pdf | Sheila M. Morrissey, The Network is the Format: PDF and the Long-term Use of Digital Content, Archiving 2012, pg. 200-203 (2012)]]
*[[NDSA:Media: CommentsOnISO19005-3_smorrissey.pdf | Ithaka comments on ISO 19005-3 draft]]
*Caroline's document
*In future set up calls with Steve Levinson (U.S. Courts) and Leonard Rosenthal (Adobe)

NDSA:PDF Exploration

2013-01-07T21:39:10Z

Chris Dietrich: /* PDF/A-3 Use Case Scenarios */

Back to [[NDSA:Standards_and_Best_Practices_Working_Group | Standards Working Group Main Page]]

==Title of Activity or Project==
NDSA PDF/A-3 Scoping Project

==One Sentence Description:==
NDSA PDF/A-3 Scoping Project working group members will research the pros and cons of using the PDF/A-3 standard as an all-purpose wrapper for various digital asset/media types including: textual, audio, video, photo, and GIS data.

==Statement of the Problem and Goals for Addressing the Problem:==
It is unclear whether PDF/A-3, which was designed to accommodate supplementary media files for text documents, is appropriate as a de facto normalization wrapper format for all media types. The goal is to develop guidelines for the appropriate use of PDF/A-3 with respect to different media types that includes both detailed technical information and a practical quick reference guide for end-users.

==Strategic Value of Activity:==
* Improve understanding of best practices for using PDF/A-3 in digital preservation activities
* Enhance consistency and improve long-term viability of digitally preserved content
* Provide guidance to those considering PDF/A-3 as a long-term archiving format

==Required Resources:==
* Time of working group members
* Publishing venue(s)
* Communication channels

==Roadmap:==
# Hold regular working group conference calls (monthly, between NDSA Standards WG calls)
# Draft document and review
# Invite broader NDSA member feedback
# Publish document (digitalpreservation.gov, others?)

==Dissemination of Knowledge:==
* Publish report on digitalpreservation.gov
* Write a blog post
* Announce on NDSA member organization communication channels
* Present at conferences that members (and non-members?) are attending

==Signifiers of Success and Outcomes:==
* Completed guidelines document published on digitalpreservation.gov
* Guidelines document referenced on related Wikipedia pages
* Guidelines in use or recommended by NDSA participating organizations or others
* Publication at other conferences/other journals

==Questions to Ask and Answer==
*Talk about background (what is pdf/a-3 and how is it different from earlier versions of PDF/A)
*Iterate categories of materials/use cases/concrete examples where it makes sense to use A-3 and other categories where it doesn't make sense. Example: if you're sending a video file don't put it in a PDF! If you had a certain kind of a journal article that had a static version of the spreadsheet in the doc but a malleable version embedded perhaps that argues for it.
*Risks to the format (scenarios in why this might be bad and why)
*Possibilities of the format (scenarios in why this might be good and why)
*Have list of defined terms in our document. How do these relate to the terms in the ISO spec. Leverage NDSA Levels of Preservation glossary. Link to glossary.

==PDF/A-3 Use Case Scenarios==
Add them here! We can create a separate page as necessary.
----
Example: Federal agency with a document management system puts an MPEG video file (and nothing else) into a PDF/A-3 file to store and then, later, to submit as an SIP (Submission Information Package) to NARA for long-term management.

Example: Publisher has a text-only article and puts it into a PDF/A-3 file, even though, in the past, the publisher used PDF/A-2. The article is then sent to library where it will be preserved for the long term.

Example: Publisher has an article that includes a complicated table, "frozen" in place, and puts it into a PDF/A-3 file, along with the Excel file from which the table was generated, in order to make it easier for a future researcher to have a malleable version of the table for use when writing another article on the same subject.

==Members==
*Caroline Arms, Library of Congress (caar@loc.gov)
*Don Chalfant, NARA (Donald.Chalfant@nara.gov)
*Kevin DeVorsey, NARA (Kevin.DeVorsey@nara.gov)
*Chris Dietrich, National Park Service (chris_dietrich@nps.gov)
*Carl Fleischauer, Library of Congress (cfle@loc.gov)
*Butch Lazorchak, Library of Congress (wlaz@loc.gov)
*Sheila Morrissey, Ithaka (Sheila.Morrissey@ithaka.org)
*Kate Murrary, NARA (Kate.Murray1@nara.gov)

==Calls and Notes==

Call information:

*Call-in toll-free number (US/Canada): 866-469-3239
*Participant access code: 21408589

Next call: Tuesday Jan. 22, 2013, 2:00 P.M.

==Background Materials==

*[http://www.digitalpreservation.gov:8081/formats/fdd/fdd000360.shtml Library of Congress Sustainability of Digital Formats DRAFT PDF/A-3 site]
*[http://blogs.loc.gov/digitalpreservation/2012/11/all-in-embedded-files-in-pdfa/ Blog Post on PDF/A-3 on the Signal]
*[[NDSA:Media: TheNetworkIsTheFormat.pdf | Sheila M. Morrissey, The Network is the Format: PDF and the Long-term Use of Digital Content, Archiving 2012, pg. 200-203 (2012)]]
*[[NDSA:Media: CommentsOnISO19005-3_smorrissey.pdf | Ithaka comments on ISO 19005-3 draft]]
*Caroline's document
*In future set up calls with Steve Levinson (U.S. Courts) and Leonard Rosenthal (Adobe)