Information Playground

Research Papers Moving to the Cloud

The ACM has been publishing scholarly works since 1954. They have been diligently maintaining an online library of research papers that has continually benefited the technical community. For years much of their budget has been spent on print-based services as well.

I read an announcement by the ACM last week that they have decided to scale back their investment in print-based services and focus instead on the long-term digital preservation of ACM content as part of a public cloud.

Public clouds that specifically target digital preservation have a different set of requirements than a public cloud like Amazon EC2, for example. The focus in a "preservation cloud" is longevity, and the administrators of said cloud must think like digital curators.

It's an interesting exercise to study the (a) system and (b) curator choices that ACM has chosen. The ACM has chosen to go with CLOCKSS and Portico, respectively. Both CLOCKSS and Portico are not-for-profit organizations.

CLOCKSS

LOCKSS stands for "Lots of Copies Keep Stuff Safe", and the 'C' stands for "controlled". The LOCKSS software framework was developed at Stanford University and is a peer-to-peer, de-centralized model.

The CLOCKSS initiative is run by "the world’s leading scholarly publishers and research libraries" with a goal of ensuring "the long-term survival of Web-based scholarly publications for the benefit of the greater global research community". CLOCKSS ingest boxes are located at Rice, Indiana, and Stanford university. As libraries and researchers submit content to these "ingest boxes", they are stored in normalized, maintainable format in triplicate across the sites. Once all of the ingest boxes have cross-audited the content, the artifacts are moved to "archive nodes" spread throughout the globe. These boxes continually audit themselves and verify the authenticity of the content, and create new versions when hardware fails (very similar to a RAIN architecture).

Interestingly enough, the solution is a "dark archive". The initial content is not accessible to the general public. The ingest and preservation within CLOCKSS is initially focussed on maintaining content for the long-term. When a "trigger event" occurs, however, content is made public by migrating it to the newest format and storing it on publicly available nodes at Stanford and the University of Edinburgh.

A description of how CLOCKSS works can be found here.

Portico

Portico can be thought of as a third-party partner that supplies the people and processes behind the CLOCKSS solution. The best way to explain the process they bring to the table can be found in one of their brochures:

Diagram from the Portico brochure

The CLOCKSS and Portico solution relies heavily on migration of data formats as the years roll by. I've always wondered about the feasibility of such an approach. Can it scale to billions of documents? Does the system have to continually upgrade all documents to newer file formats as these new formats become available? One thing I picked up as I learned about their solution is that the format conversion occurs when the format goes from "dark" to "publicly available". In other words, there's no need to continually upgrade file formats.

The alternate approach to format conversion is something I've been researching here at EMC. As an entire class of documents is imported into an archive, can a Virtual Machine also be imported that is able to "read" all of these documents, both now and in the future? This would require some time of virtual machine "player" that has infinite playback capability (no easy task).

I hope to learn more about the server/storage implementation of CLOCKSS at locations around the world.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Intrapreneur

Preserving Profiles in Courage

The digital preservation effort at the JFK Presidential Library and Museum keeps rolling on, one scanned document at a time.

All of President Kennedy's office files (POFs) have been scanned into the archive, and the digital curators at the museum are now focusing on digitally preserving President Kennedy's personal papers.

I have had the good fortune to personally see some of these documents, and for me the most interesting documents are those that contain the President's handwritten notes. In a previous post I wrote about his 1962 speech regarding the US Space Program, and how JFK scribbled notes on the first draft to highlight emphasis and relevance to the audience at Rice University.

Attached below is another example of a heavily edited document from President Kennedy's personal papers. This particular document is a draft of the book Profiles in Courage. The document, along with the handwritten notes, was probably typed and edited around 1953-54 (the book was published in 1955).

NOTE: The image above is PNG format; the curators at the JFK Library use TIFF. I converted to PNG because it's 30x smaller. The curators use TIFF format because they want the highest quality scan; every pencil scratch must be legible.

This is cool stuff. It highlights the importance of digital preservation efforts. As a software engineer working for a company (EMC) that builds information storage systems and workflow management software, it's also a great opportunity to try and build complete solutions that more fully address the needs of digital curators.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Intrapreneur

Passing Through the Dust of Swift-Tuttle

Stargazers will be looking heavenward tonight, but where's the data going?

 Tonight is the peak night for getting a great look at the Perseid meteor showers (if you happen to live in the Northern Hemisphere). The Earth is currently hurtling through the debris from the comet known as Swift-Tuttle. On the morning of August 12th we'll pass through the densest amount of this debris. These meteors are called "Perseids" because they radiate outward from the constellation known as Perseus. Here are some tips for viewing the Perseids. Lie down in any large, flat area, preferably one that is not paved, and/or without orange lines running down the center.

When it comes to the study of space phenomena, the amount of data generated and stored is, well, astronomical. I've often wondered how scientists configure their storage systems to handle this amount of data.

One group of scientists that is on the job capturing data about meteors are these guys from NASA's Meteoroid Environment Office. The MEO is the "NASA organization responsible for meteoroid environments pertaining to spacecraft engineering and operations". I grabbed this picture of them from their website and I'm glad that they are working hard and wearing ties.

I started to dig around for the purpose of understanding where and how the world is storing the scientific data about outer space. Part of my motivation is to understand more about the re-use of scientific data as part of the MIT DataSpace proposal. I happened upon an effort called the Virtual Meteor Observatory (VMO), which has a goal of "improved access to astronomical data and computing resources".

In December of last year there was a meeting of the International Space Science Institute (ISSI) to discuss the architecture of a VMO. The meeting was attended by a variety of meteor observer teams, such as the Polish Fireball Network, the University of Western Ontario in Canada, and the IAU Meteor Center. The full report from the meeting can be found here.

The VMO has agreed upon a standard XML format for the description of meteoric data, which I've cut directly from the report and pasted here (for what it's worth I was unaware that Heidi Klum was involved in this effort):

<?xml version="1.0" encoding="utf-8"?>
    <vmo xmlns="http://www.imo.net">
        <fireball>
            <time>2008-11-23T15:24:13</time>
            <brightness>as the full moon</brightness>
            <observer>Heidi Klum</observer>
            <location_latitude>
                35.24351
            </location latitude>
            <location_longitude>
                -89.62907
            </location longitude>
            <country code>US</country code>
            ...
        </fireball>
    </vmo>

With the agreed-upon metadata format, the team then determined five different types of "data formats" to be stored in the VMO repository:

• Visual meteor observations
• Video and still camera data
• Forward and backward scatter radio observations
• Fireball observations
• Orbit data

Existing scientific software that generates data and metadata must be converted into the VMO format, and ultimately everything is stored in a PostgreSQL database. Discussions are ongoing on whether  all scientific data is stored in one large database or if a distributed approach will be used.

Two thoughts that I had as I read through the information about this effort:

1. The researchers are involved with the submission and ingest of scientific data, as well as the dissemination of the data for scientific analysis. This workflow is very similar to the standard workflow described by the Open Archival Information System specification (depicted in Figure 4-1). The researchers could benefit from their awareness of this standard.
2. I was unable to find mention of the scalability limits of the system and requirements on the storage in general. Given that nearly all of the data being generated is fixed content (otherwise called reference information), then the XAM specification is a good candidate to abstract away the dependency on PostgreSQL.

Get out there tonight, find a safe piece of grass, and watch the show.

Steve
http://stevetodd.typepad.com
Twitter: @SteveTodd
EMC Intrapreneur

Repertoire Politics

I came across the phrase "repertoire politics" while reading the latest online copy of ON Magazine. The section that I was reading was called Philharmoniana (page 6). The article highlights a digital preservation effort that is currently happening at the St. Petersburg Philharmonic Library in Russia.

The library itself has been part of the St. Petersburg Philharmonic Hall since 1882, and it contains over 150,000 items, such as musical scores, and concert programs. 

One of the rooms in the library has a card catalog made up of 155 drawers which contain roughly a quarter of a million index cards. These cards contain detailed notes and information about every concert that has been held in the hall since 1921! These details include:

• works that were performed
• name of the composer
• orchestra
• name of the conductor
• names of any soloists
• time it took to perform a piece
• name of the musicologist (who often gave a pre-concert lecture)

The card catalogue section of the library is known as Philharmoniana. Digital curators at the library are choosing to digitally preserve the Philharmoniana first. Allowing these notecards to be viewed online opens the door for more researchers to understand St. Petersburg history through the lens of the performances that occurred at its great hall. This brings me back to the introduction of the term "repertoire politics", which is defined in the article as follows:

'the concert information scrupulously entered into the catalogs will allow scholars to analyze the shifts in musical preferences, trends, and tendencies -- known as "repertoire politics" in the world of classical concerts -- from season to season and over the course of nearly a century.

This unique knowledge base will also satisfy the curiosity of music lovers who seek to know who was conducting what and when, who were the famous soloists invited to Russia, and what music was performed during the 900-day siege of Leningrad in World War II.'

Currently 10,000 cards from 17 drawers have been digitized onto a CLARiiON AX4 as part of a grant made available from the EMC Heritage Trust Project, a project that provides funds to safeguard cultural resources around the world.

One more note of interest: the author of the article, Dr. Ludmilla Leibman, is the Executive Director of the Educational Bridge Project. The EBP's goal is to provide educational and artistic exchanges between Russia and America. This group has arranged for seven representatives from the leading educational and cultural organization in St. Petersburg to visit Boston area cultural sites during November of 2009.

Many of my co-workers are in St. Petersburg and in fact I visited the city in 2007 on business for EMC. One thing that I learned is that the city of St. Petersburg (founded in 1703) is actually younger than the city of Boston! I hope to learn more about St. Petersburg come November.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

Content Management and XAM

When I started writing a paper last year about the advantages of using a technology like XAM for the long-term curation of digital artifacts, I was looking forward to hearing feedback on the recommendation.

Well, last week Craig Randall forwarded me an article highlighting the fact that others are arriving at the exact same conclusion on their own.

In the above-referenced article, Alan Pelz-Sharpe, who covers Enterprise Content Management technologies and practices, duly notes that the work the SNIA has done on the XAM proposal provides strong benefits in an ECM environment where document preservation is a paramount requirement.

The thrust of the article is that XAM is an excellent "contract" when it comes to an API between archival applications and storage systems. It has strong "longevity" characteristics when it comes to the ability to seamlessly migrate archived content from one system to another over the course of many years.

The article ends with a call for customers to add XAM to the list of request for proposals (RFPs) to content technology providers.

While Pelz-Sharpe focuses on the XAM benefit of "migration between storage devices", I'd also like to point out a few more XAM benefits that augment the point:

• XAM is defined for massively scalable object storage in a flat object space. Traditional archiving applications that use file systems often experience a high cost of ownership in regards to maintaining said file system. XAM devices offer the potential to hide file system storage nuances (if the XAM device uses a file system at all), and minimize or remove high administration burdens.
• In a similar vein, some archiving applications that utilize underlying file systems will store hard-coded absolute pathnames to content. This can result in a rigid, inflexible file system structure that is very difficult to change (or migrate). Archiving applications that use XAM must reference location-independent names instead. These names are known as XUIDs.
• Archiving applications often use databases as well as file systems. Content and its related metadata can become "split" between the DB and FS. This split increases the chance of losing the "unity" between a given piece of content and its related metadata. XAM technology acts as a "paper clip" which permanently binds these two (or more) items together.
• Many times archival applications need to specify retention periods. XAM has built in support for retention. When an archival applications stores content to a XAM device it can mandate the length of time that the device must prevent deletion (including an infinite amount of time).
• Finally, archival applications often have strong authenticity requirements. Customers may need to know that the originally ingested document has not been tampered with or altered in any way. Once again, XAM has built-in support for content time-stamping on ingest as well as restriction of alteration when it comes to content (this is known as "binding" in XAM-speak).

Steve
http://stevetodd.typepad.com
Twitter: @SteveTodd

Digitizing Dickens

In my local newspaper this morning there was an article that caught my eye. There's a fairly significant digital archiving project occurring at the Worcester Polytechnic Institute to preserve some of the original works of English novelist Charles Dickens. This, of course, is interesting to me in the context of previous posts I've written about the digital preservation occurring at the JFK Presidential Library and Museum in South Boston.

The George C. Gordon Library at WPI has a very significant selection of historical books and they invest a large amount of their time caring for and preserving the physical copies of old works.

So why go to the effort of digitally preserving Dickens' works when certainly most of his novels can be found online already?

Well the answer lies in the fact that the particular books at WPI's library are sequential portions of the entire novel. I looked up this announcement (from 2008) of the WPI Dickens digitization project and learned that the original release of his novels became available as a series of printed installments.  Once these serialized versions are digitized it will be the first time they'll be available on the web.

The importance of web-based access to the serial documents is highlighted in this quote by WPI professor Joel Bratin:

“Few scholars—and certainly very few readers—have previously had access to these rare serial parts, as they can be found in only a few collections worldwide,” Brattin says. “That’s unfortunate, as they reflect, more so than any other published text, the author’s original intentions as to wording, punctuation, and so on. Plus, by exploring these installments, readers are able to experience the novels—and a bit of Victorian London life—as their earliest readers did, complete with advertisements and with illustrations that are rarely reproduced in full in today’s paperback editions.”

This reminds me of the digitization project at the JFK Library. Copies of JFK's speeches are widely available on the internet, but the original speeches that he hand-edited in his office are not. These originals clearly provide new insights to researchers.

The WPI effort uses scanners and OCR (optical character recognition) to join the scanned image together with metadata. Additional metadata is added by a WPI digital curator. The library is storing the documents as PDF/A, the archival form of PDF, which actually leaves out some of the PDF features that are not amenable to long-term archiving. The stories are being preserved along with color illustrations and will be text-searchable.

The newspaper article went on to state that WPI owns all but four of the serialized novels, and hopes to borrow the rest (and preserve them) from the Charles Dickens Museum in London.

Steve

http://stevetodd.typepad.com

Twitter: SteveTodd

Considering MIT Dataspace

The EMC Innovation Lecture that I wrote about several weeks ago went well. A webcast is available here. A few dozen people dialed in to listen to myself, Burt Kaliski from EMC, and Stuart Madnick from MIT talk about Digital Curation and a variety of related topics. Wayne Adams (EMC) from SNIA was also present during the call to talk about what's been going on lately with the XAM standard within SNIA.

Several attendees called in to ask questions and/or submitted them via chat.

When the lecture was over I found that I had a question of my own:

Can scientific research repositories benefit from archival standards?

MIT DataSpace

Distributed silos of research information don't necessarily lend themselves to external re-use. For example, if large amounts of anonymous medical X-RAYS are stored at a hospital for research and analysis, can the X-RAY data (and the results of the research) be accessed by another research team?  If environmental data is captured off the coast of Massachusetts and used to measure ocean temperature trends, can that data be re-used by other environmentalists around the world?

The answer can often be no, and the MIT DataSpace project is a proposal focused on turning that answer around.

The proposal has been submitted to the National Science Foundation (NSF). The NSF is accepting proposals as part of its DataNet initiative. If funded, MIT and a host of other academic and industry teams will work on a solution.

What is the Format of Research Data?

Back to my question. Research data in repositories around the world have a variety of different formats. Some of it exists in databases, some of it exists in file systems, and some exist in both. Let's assume that DataSpace would propose that a standard method be used to take these existing systems and provide some standardized form of access to the data.

And let's assume that DataSpace would like this research data to be available for a long time.

Keeping digital data for a long time is the job of a digital curator, and most digital curators have adopted a standard for their digital archives known as the Open Archival Information System (OAIS).

How does scientific research data map to OAIS?  Consider the oceanographic data example. The temperature measurements are the key piece of data, known in OAIS terms as a Data Object. Any Data Object, when it is contained within an OAIS archive, must be accompanied by the following mandatory metadata:

• Representation Information: how to interpret the oceanographic data. What format is it in?
• Reference Information: a persistent identifier that uniquely identifies the data.
• Fixity Information: authenticates the oceanographic data
• Provenance Information: documents the history of the data
• Context Information: metadata or description describing the data

It seems to me that the above five fields, if consistently made available for scientific research data, would be highly valuable. This leads me to conclude that the techniques used in long term, OAIS-compliant digital curation should be studied in the context of standardized access to digital research archives.

The next step that I'm interested in pursuing is finding some use cases (e.g. real-life research repositories) and beginning the process of mapping them on to OAIS.

I'd love to hear from anyone that has already thought along these lines (or is interested in doing so).

Steve

http://stevetodd.typepad.com  

Twitter: @SteveTodd

Cuban Missile Crisis: EMC Innovation Lecture Series 6/3

Tomorrow, June 3rd, at 2:00 PM EST, I'll be giving a talk about digital curation and archiving as part of EMC's Innovation Lecture Series. You're invited.

The goal of my talk is to cover research directions within EMC when it comes to XAM technology and digital curation. I briefly covered the agenda in my last post, and gave an example of a cool historical document that has already been scanned into the archive: President Kennedy's 1962 speech at Rice University regarding the United States space program.

I've received another document from the archivists at the museum that also has great historical significance. Below is a scanned portion of the speech given by President Kennedy just one month after the speech given at Rice:

This speech was given to the American public from the White House as part of what is now known as the Cuban missile crisis.

What's interesting about this speech is that the original document from the White House does NOT use the term "Cuban Missile Crisis". This label was applied after the speech. Indeed, the three countries involved (the US, Russia, and Cuba) all gave the crisis a different name. If this document is simply scanned into an archive it will not be found by a search unless a human somehow attaches metadata to the image.

The "somehow" of attaching metadata to an image is a big part of my talk tomorrow.

In addition we're hoping that Stuart Madnick from MIT will be available to participate as part of the session. Professor Madnick and MIT are proposing an industry-wide collaboration to solve the problem of uniting distributed, heterogeneous, scientific digital archives. This effort is known as the DataSpace initiative.

Please join us, registration is open to all and can be found here.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

Digital Curation Current and Future: Lecture Series

Next Wednesday June 3rd, 2:00 PM Eastern, you're invited to participate in a talk I'm giving as part of EMC's online Innovation Lecture Series. My talk focuses on the growing field of Digital Curation and the applicability of two standards: OAIS (Open Archival Information Systems) and XAM (eXtensible Access Method).

In a nutshell, here's the agenda:

• An up-to-the-minute status of the digital archiving effort going on at the JFK Library.
• A high-level overview of OAIS and XAM.
• A preview of MIT's DataSpace proposal, which was just submitted this month to the NSF.

Dr. Burt Kaliski is the host of the lecture series for EMC, and he asked me to present this topic based on this research paper presented at the Digital Curation Conference last December.

Why do I find this topic so interesting? Mainly due to the items being preserved. The JFK Library has sent me some of the more interesting documents that they have scanned into their archive. Check out this JFK speech from 1962, complete with his handwritten notes to himself:

 

My talk will cover this speech as well as another speech given to the American people at the beginning of the Cuban missile crisis. 

I hope you can dial in, please register here.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

 

For information about helping Nick, please read Mark's blog:

http://markfredrickson.wordpress.com/2009/05/21/may-20-810-p-m-new-development/

EMC Digital Archiving HW for JFK

In this series of articles about the digital preservation effort going on at the John F. Kennedy Presidential Library and Museum, I've yet to describe the actual hardware configuration that is currently in operation.

I've covered the EMC software being used, the archivist's process being used, the crucial role played by Documentum, and the conformance to standards such as OAIS, NARA, and Dublin Core.

This post more fully describes the EMC hardware architecture put in place to make it all happen.