Up to ten times the speed with Silent Bricks

Every era provides new insights: This makes it all the more important to be able to evaluate documents from the past again and again based on the current zeitgeist and state of knowledge. Archives such as the Federal Archives are essential for this purpose. "Every archive is a place of reassurance that makes individual and collective memory possible. In the age of information and digital change, archives are guarantors of national sovereignty," as Prof. Dr. Michael Hollmann, President of the Federal Archives, puts it.

A simply gigantic amount of data

The Federal Archives has the legal mandate to permanently conserve the archive material of the Confederation and to also make it usable. Files, maps, pictures, posters, films and sound recordings are stored in analog and digital form. Around 2300 employees currently work at the Federal Archives, about 100 of them in IT. Digitization is a real Herculean task, with between five and ten petabytes of storage volume being added every year. There are many analog sources; at the end of 2020, for example, there were 428 linear kilometers of written material and over one million film reels alone. Since mid-2021, the Stasi files of the Gauck Office have also become the responsibility of the Federal Archives: Here, there is a need for further digitization.

In addition, all films that benefit from film funding in Germany are stored in the Federal Archives. Most of the contents, however, are files, including, for example, protocols of the federal cabinet and of administrative agencies. The archived documents date back to the Holy Roman Empire of the German Nation. Secret documents are also stored in the Federal Archives.

What can be done about bottlenecks in the digitization process?

Initial thoughts regarding the project began in the fall of 2018. "We were faced with the problem that we were receiving a lot of data from external service providers, and the trend was increasing. At that time, this data came to us on USB media or hard drives, and the transfer involved a lot of effort," reports Timo Dommermuth, Head of IT Operations at the Federal Archives. First, the data had to be downloaded to a special storage area so that it would be available for the next processing steps.

"The copying from these data carriers, mostly USB hard drives, only worked at 80 megabytes per second - in the best case. That slowed down processing considerably, because we don't just process a few terabytes, but up to ten petabytes a year," explains Dommermuth, who is responsible for the project. It took around two weeks to validate the data on such a hard drive and to ingest it completely into the archive. This is because the data was validated and transformed into the TAR format from the dedicated storage area. These data packets were then transferred to the tape archive storage. The question was therefore how this process could be significantly streamlined, simplified and accelerated.

Copy process skipped and transfer rates significantly accelerated

This was not easy, as there seemed to be practically no solution on the market that made sense. Finally, the IT experts remembered a presentation of FAST LTA that they had seen some time ago. At the time, it had been about the classic concept of Silent Bricks as an archive solution that could act as energy-efficient cold storage. However, given the volumes of data that the Federal Archives has to deal with, that would have been utopian, says Timo Dommermuth. However, the experts came up with a brilliant idea: How about simply using the technology to exchange data with service providers? "We remembered that Silent Bricks can be pushed and mounted directly into controllers. This might make the ingest process and copying much faster," Dommermuth summarizes.

At the end of 2018, a proof of concept (POC) was launched together with IT service provider and FAST LTA partner Bechtle AG. The start was made with Silent Bricks HD in two sizes and an SSD variant as well as the associated controllers. "In the POC, we found that we could manage transfer rates of 500 to 700 megabytes per second, with large file formats, with the Silent Bricks: This has resulted in an immense advantage and a significant process streamlining," says Dommermuth. Throughput times have been more than halved, and the information is now available digitally in the archive correspondingly faster. The data is written directly onto the Silent Bricks by the service providers, and after transport in the Federal Archives, it is simply pushed into the controllers and mounted via NFS: The copying process is completely eliminated. In the first step of the project, twenty Silent Bricks and two controllers, each with five slots for five Silent Bricks, were purchased. Because of the higher performance, the SSD variant was chosen.

Integrating digitization partners easily

The new storage technology was already tested in the POC with some service providers who had been given the task together with Bechtle. "We were able to use the FAST LTA technology off the shelf, so to speak, without any adjustments, and our digitization partners reported back to us that they were able to work smoothly with it right away," recalls the IT specialist. From the first thought to implementation, he notes, it took only half a year. "That is very quick for an IT project with a technology that had not been used before," says Dommermuth.

About a third of the digitization is done by the Federal Archives’ team itself, the rest by external service providers. At the Berlin-Lichterfelde site silent bricks are written with new data and exchanged with Koblenz every two weeks by regular transport. Particularly secure transport containers are used for this purpose. "We bet on the right horse," sums up Timo Dommermuth, "the contact with Bechtle is impeccable and the support from FAST LTA is very good, both partners are very flexible. No problems have arisen so far".

Safety as the driving factor

Another important motivation for the decision was the robustness and almost complete fail-safety. After all, many transports take place between the external service providers and the Federal Archives, and things are not always smooth. Erasure coding is used in every storage medium to prevent data loss due to failure. "The risk of data loss is practically zero due to the integrated security concept. In the past, on the other hand, documents had to be digitized again when USB hard drives failed," says Timo Dommermuth.

Digitization is not always without problems, especially when it comes to working with old nitrate films. Here, a fine touch is required in the truest sense of the word to ensure that the films are not destroyed or even that people are put in danger. The carrier layer made of nitrocellulose, also known as gun cotton, falls under the Explosives Act. The films must therefore be stored at a certain temperature to prevent spontaneous combustion. Now, storage of the digitized nitrate films on the Silent Bricks also takes place in protected rooms. The cinema newsreels from the World War II era alone already comprise 20 terabytes in digital form.

The storage system continues to grow

Over the years, the system, which can be flexibly expanded on the basis of a modular principle, has grown. The company is currently in a new procurement phase. Another six G5000 controllers and around 30 Silent Bricks, each with a 24TB gross SSD, are to be purchased soon. The trend is clearly moving towards digitization on demand: as soon as a file is touched as a result of a request, it is immediately digitized in a scanning process. This is one of the reasons why the silent-brick infrastructure continues to be expanded. The experts at the Federal Archives expect that by 2024 they will be able to digitize around 60 to 80 million pages per year. So far, 75 million pages have been digitized - which is a lot, even by international standards.

The motivation of archive users varies greatly: it is often scientists or doctoral students who need documents for their research. But there are also repeated requests for clarification of fate: the grandchild generation is increasingly interested in coming to terms with family history during National Socialism. There are also rather unusual trends, however: game designers, for example, are basing their designs on plans of battleships from World War II. They now all get their information digitally, rather than as paper copies.

Author: Daniela Hoffmann