Work in Progress-Report: Historical Searches. 
Latest Developments in the AAC-Austrian Academy Corpus

Hanno Biber, Evelyn Breiteneder, Karheinz Mörth
AAC-Austrian Academy Corpus
Austrian Academy of Sciences
hanno.biber@oeaw.ac.at,evelyn.breiteneder@oeaw.ac.at,karlheinz.moerth@oeaw.ac.at

The "AAC-Austrian Academy Corpus" is a Vienna based corpus research unit of the Austrian Academy of Sciences. The AAC is concerned with establishing and exploring large electronic text corpora and conducting scholarly research in the fields of digital text corpora and digital editions. The texts of the AAC collections date from the last 150 years, are predominantly German language texts and have been selected into the collections for their historical and cultural significance. The AAC has incorporated thousands of literary objects by thousands of authors, representing an astonishing range of different text types from all over the German speaking world. Among the sources, which cover manifold domains and genres, there are newspapers, literary journals, novels, dramas, poems, advertisements, essays, travel accounts, cookbooks, pamphlets, political speeches as well as plenty of scientific, legal, and religious texts, to name just a few. Thus, the AAC can provide a great number of reliable resources for investigations into the linguistic and textual properties of these texts. Well beyond 400 million running words of text have already been scanned, converted into machine-readable text and carefully annotated. The annotation and mark-up schemes of the AAC are based upon XML related standards. Most of the projects in the first phase of corpus build-up were dealing with issues related to the application of basic structural mark-up and selected thematic features of the texts. In the next phase which is planned to be focused on application development, the AAC intends to intensify their efforts towards deeper thematic annotations, exploring problems of linguistics and textual scholarship. This paper is to give a short overview of latest developments in the AAC’s corpus build-up activities and the ongoing efforts to improve their corpus exploration tools.

Since its foundation, the AAC has been committed to text technology, a field of research virtually non-existent in the traditional academic canon. Text technology - as we understand it - is situated at the crossing points of computing in the humanities, corpus linguistics and the philologies. Reaching beyond the traditional sphere of Academia, text-technology is concerned with a range of divergent fields such as information science and communication technology, philological methodology and web design.
The AAC's text technology working group has been working on issues of digitisation of historical language data, establishing efficient workflows as well as developing usable software. While the original objectives of the build-up phase of the project were focused on issues of corpus creation, the second phase will see more work on analysis and exploitation of textual resources. All objectives formulated at the outset of the program have been fully met, in many areas the working group exceeded what was originally planned.
The AAC's work in the years to come will be to adapt existing resources, to develop new resources where necessary and to document and describe these resources in a manner that enables users from different backgrounds and disciplines to do research with textual resources. These endeavours can be regarded as basic research in humanities methodologies. Our perspectives parallel - in many respects - those formulated in the pan-European project CLARIN which has been set up to "create, coordinate and make language resources and technology available and readily usable".
Over the past few years, the foundation for the department's future work has been laid by a continuous build-up of a digital corpus of German language historical texts. The AAC has developed methods and tools to allow scholars to access these texts and other comparable resources. In pursuing this agenda, the AAC has always proceeded from very text-oriented concepts attaching great importance to a rather holistic perspective that does not allow the reduction of such resources to collections of merely words or sentences. They have strived to keep the doors open for scholars of the various history disciplines, the traditional philologies, philosophy, literary studies, sociology, musicology, law, cultural studies as well as linguistics.
The work of the next phases was planned in a manner complementary to the previous steps. Research will have to focus on methods and resources for making large amounts of texts accessible. The AAC will make efforts to use available and suitable tools, attempting to add to them wherever necessary, to provide and to promulgate relevant expertise while building up what in the department's nomenclature has been dubbed the AAC Container, the central access point to the holdings of the corpus. However, before realizing this task, a number of preliminary steps have to be taken.
To fully grasp the AAC's approach to encoding digitised historical printed material, it is necessary to take into consideration that –institutionally - the AAC developed as a spin-off of the then Department of Functional Literary Text Types (Kommission für literarische Gebrauchsformen). The working group's motivation to develop a digital corpus derived largely from experience gained during their text-lexicographic work on the "Wörterbuch der Redensarten". It was with that project that they started to make use of digital resources, embarking on their first digitisation experiments and undertaking their first more challenging software development projects.
In contrast to other corpus-oriented projects, the working group proceeded from literary and text lexicographic premises. Corpus research and the creation of large electronic text collections have traditionally been the domain of (corpus-)linguists who have pursued this path for several decades. Literary digitisation initiatives were often restricted to particular writers, many of these projects did neither produce large amounts of data nor pursue research on methods of how to tackle the problems involved in working with such data. Being aware of the need of digital resources in many fields of the humanities, the AAC's working group started to work on applications, tools and methods geared towards a wider range of applications trying to pursue a path of text-oriented computing. While the needs of linguists have not been ignored, they tried to work towards applications that also offered access to coherent texts, being convinced that for many applications it will be indispensable for scholars/researchers to have access to the text as such.
From the start, the AAC has relied on XML (a general-purpose specification for creating custom markup languages) and cognate technologies as the foundation of their corpus build-up activities. The AAC's experiments with XML began right when the standard emerged in 1998. It based all its projects upon this standard. 
While a large number of varying XML "dialects" to define XML vocabularies have been developed, XML Schema appears to have emerged as one of the more successful ones. Most of the AAC's projects use XML Schema as a means of validation. The AAC tools support this technology which allows both the controlled application of markup as well as automated validation of large amounts of data.
The AAC applies an encoding scheme characterized by a combined approach to capture both structural features of the texts as well as a certain amount of data describing the physical appearance of the original texts. This approach has led to a hybrid system of markup not only representing the basic semantic structures of the texts but also a certain amount of layout information. In digitising historical data, semantic and presentational data - of necessity - will remain inextricably intertwined. When working on large amounts of such texts, it is considerably easier to capture formal data than to translate innumerable typographic idiosyncracies into consistent structural markup. Although the AAC's system proved to be workable in many of their projects ‑ its practicability was established in the digital editions of "Die Fackel" and "Der Brenner" ‑ presentational data is being applied in reduced fashion now.
The AAC's digitising activities are characterized by a strong connection to the physical objects of digitisation. This may be seen as one of the motives behind what in the department has been called the "one-page-one-file principle", which implies that each page of a printed publication is stored as a separate digital object in the digital medium which ‑ by means of implicit and explicit linking ‑ form larger digital objects that in turn represent coherent texts. By doing so, the AAC attaches importance to the semantic structures of the text as well as to the physical appearance of the text.
To ensure an efficient working environment that uses floating staff, the AAC has set up a detailed documentation of the markup to be applied. The documentation is fully XMLified. The output is visualized via XSLT in standard browsers and accessed in the help sections of our encoding tools. It contains precise specifications and explanations of the elements and attributes that make up the system, and furnishes numerous examples intended to help users to correctly apply markup.
Metadata describing production processes and details about the physical sources of the digitised objects are the backbone of a digital data collection's usability. While the importance of this kind of data is widely acknowledged in the community today, it is often a secondary task.
The AAC collects two types of data that fall into the above categories. The first consists in descriptive metadata concerning the digitised objects. This information has been drawn-up on a regular basis when the physical objects were scanned. It is stored in a relational database containing c. 5500 records (early 2009) holding all relevant information about the physical objects so far ingested into the corpus. In many cases this data is much more detailed than regular library records. The record fields in this database were designed in a way that they can be easily mapped onto the fields of TEI headers. Mechanisms exist that allow to associate these with the relevant documents, either by storing them as separate documents, or inserting them as TEI headers into the respective XML documents.
In addition the working group has been gathering data that describes the production process of the digital objects. This data is generated by a content management system used to allot particular tasks to encoders. This software, dubbed taskEditor, was designed and implemented in-house at the outset of the project. It has since been used to organise the workflow and to achieve versioning and archiving of the data processed by the AAC. This system is entirely built on XML too.
The Austrian Academy Corpus is not publicly accessible on the Internet but can be used at present in-house for research. Although the build-up phase of the AAC's programme did not imply the disclosure of the data, the department's researchers needed access to the data. To make this possible, applications were developed that, from the beginning of the project, were called corpusBrowser. This is that part of the system that interacts directly with the user, with the researcher reading, interpreting analyzing the texts. There have been two generations of corpusBrowsers. Each has been continually improved and over time their functionality has been considerably enhanced. As there are no out of the box solutions for text collections such as the AAC, all of this work has been largely exploratory, experimental in nature, a path of trial and error.
While the first generation of corpusBrowsers worked with relational databases, the second generation was built around an indexer known by the acronym DDC (DWDS/Dialing Concordance). While DDC is a linguistic search engine, it is also capable of performing general purpose word form searches. DDC also offers neatly definable range operators. It is comparable to SARA, the search engine built to access the British National Corpus. As DDC was primarily designed as a linguistic search engine, the data it is best suited for is lemma and part-of-speech data. However, it can handle any types of token attributes. The tag set we have been using for this particular purpose has been STTS (Stuttgart-Tübingen Tagset).
To access and display the underlying data in a comfortable manner, it was necessary to find an adequate display mode. Over time, the working group came up with multi-window solutions which could be easily adjusted to individual needs. The primary window is used to trigger queries and to organise the search results. There are additional forms to displays the XML text, to display the facsimiles, one to display metadata and a special form to display the book resp. magazine as a whole. This is the component of the program which offers random access to the resources containing the results of queries as well as any other part of the corpus.
Having established a working infrastructure for the digital texts available at this point, the AAC is currently working on more sophisticated methods of utilising large scale corpora doing experiments with various database systems as well as XML-aware indexing tools to establish standard procedures of accessing large XML text repositories. In its various projects the AAC makes also extensively use of traditional database management systems. In all of this work, the focus has been on developing efficient methods for the integration of relational databases and document oriented XML data. Databases are used for a number of quite varied purposes. In the first instance, they serve as indexing tools, making use of the databases' full text search capabilities. For smaller and medium-sized projects this method of indexing can yield quite satisfactory results, although it does have a number of obvious limitations. Two AAC projects have to be highlighted in this regard, both of which make use of a common interface linking textual corpus data to respective database entries. The first one is a large-scale prosopographical project, the edition of "Der Brenner" served as a pilot application for this data. As in most AAC projects, the methodological approach in this project has been very text-oriented, first collecting textual data concerning persons mentioned in particular parts of the corpus. Then this data has been enriched from external resources and finally made available as part of digital online editions. The other project to be mentioned here deals with references to periodicals in a historical journal. The proceeding in collecting this data has been similar. After drawing up a thorough survey of all periodicals mentioned in a particular journal, the data was disambiguated and enriched with externally available data.
As has been stated before, the AAC has been experimenting extensively with both traditional databases and linguistic indexers. In the BRENNER ONLINE and AAC-FACKEL editions they opted for database approaches. As these projects have to be regarded as experiments that were conducted consecutively different solutions were selected. In the case of the Brenner-project, all querying is performed making use of MySQL's full text search engine which given the comparatively small amount of data being searched did scale quite well.
In the AAC-FACKEL we pursued a more sophisticated approach. Again a ready tool, in this case MSSQL, was put to use, however in a rather heterodox manner. As the Fackel data had been furnished with linguistic information in the preprocessing phase we had to find a way to search these without giving up the other information stored in the XML documents. The quite effective full-text engine of the database did not prove to be the adequate tool for the purpose. In the end, the task was accomplished by creating database records for each token which in turn did not only contain linguistic data but also information as to the position within the original document. In this way, it was not only possible to refer from the search results back into the texts but also to allow complex queries including range operators. Preparation and import of data into the database was performed by functions built-into Corpeduni, the AAC’s main encoding tool. However, it has to be mentioned here that this approach will not work in a satisfactory manner for amounts of text larger than what we have been working with in the Fackel-project as particularly range queries which are resolved by the database through the creation of subtables often tend to be very slow. For future projects with amounts of text tokens that go into six-figure numbers the AAC is conducting experiments with indexers such as Xaira and the afore mentioned DDC, both being quite versatile tools which need quite some expertise when setting up a more complex application. Concerning the interface, we have observed a few very important principles. All of the content displayed in the digital editions is dynamically created from XML data. Output is produced by means of XSLT stylesheets. This holds for the text section as well as the contents overview and the result lists of the two editions. We have adopted this approach in order to be able to cope with future challenges and to ensure the viability of our data for as long a period as possible. The communication between the server and the clients is performed by Ajax (Asynchronous JavaScript and XML) and simple CGIs that do some formatting of data coming from the database.
The AAC-FACKEL was published on 1 January 2007 and BRENNER ONLINE followed in October 2007. The interface is organized five synchronised frames which are placed within one single window. The "Paratext"-section situated within the first frame provides background information and essays. The "Index"-section gives access to a variety of indexes, databases and full-text search mechanisms. The results are displayed in the adjacent section. The "Contents"-section has been developed, to show the reader the whole range of the journal ready to be explored and provides access to the whole run of issues in chronological order. The "Text"-section has a complex and powerful navigational bar so that the reader can easily navigate and read within the journals either in text-mode or in image-mode from page to page, from text to text, from issue to issue and with the help of hyperlinks. These digital editions will function as models for similar applications. The AAC's scholarly editions of "Der Brenner" and "Die Fackel" are to be regarded as an important contribution to the development of historical digital resources for linguistic and literary research.

REFERENCE: http://www.aac.ac.at