LibGuides: Digital Humanities: Data

Getting Started

Planning

University of Otago Library data management plan (DMP)
Whether your data is numerical, pictorial, textual, or something else entirely, always have a plan to ensure it is managed throughout the research lifecycle and beyond.

Finding

Find
Open data is data that can be freely used, re-used and redistributed by anyone - subject only, at most, to the requirement to attribute and sharealike. Find sources of open data for digital humanities projects.

Cleaning

OpenRefine
Looking for a free, powerful tool to help you clean your data? Try OpenRefine.

Advice on research methodology : Ben Daniel

Ben Daniel | HEDC Coordinator & Associate Professor
Ben is available to talk to any humanities researcher about research methodologies, data and content analysis, and educational technology.

Experimenting with the British Library’s content and data - DHA Praxis project

Be inspired!

In February 2016, the University of Nottingham and British Library Labs organised the workshop “Experimenting with the British Library’s Content and Data” as part of the British Library Labs Roadshow 2016 and the Digital Humanities and Arts Praxis Project. Researchers from the Horizon Digital Economy Research Institute and the British Library Labs team presented experiences of creative use of digital collections and data. The workshop ended with the ‘Ideas Lab’ encouraging participants to explore, experiment and think of ideas of what they might do with the British Library’s digital content and data. This video presents a synthesis of the event.

Big? Smart? Clean? Messy? Data in the Humanities
This overview addresses questions like:
-What is the relation between the data we have and our objects of study?
-What types of data are there in the humanities, and what difference does it make?
-What new ways of dealing with data do we need to adopt in the humanities?
Defining data for humanists: text, artifact, information or evidence?
What is data to a humanist? This article addresses: data as constructed artifacts, data as interpretable texts, and data as processable information.
Digital Humanities Data Curation (DHDC)
Intended to help students and those new to the field, the DH Curation Guide also provides a quick reference for teachers, administrators, and anyone seeking an orientation in the issues and practicalities of data curation.
A Gentle Introduction to Data Cleaning
Sometimes your data spreadsheets will contain errors. Learn how to spot and fix them in the online learning module "A Gentle Introduction to Data Cleaning".

Public data lab
A new initiative to engage researchers, civil society groups, journalists, developers, designers and public institutions around data infrastructures and data practices.
RDA and the Digital Humanities
The Research Data Alliance (RDA) provides a neutral space where its members can come together through focused global Working and Interest Groups to develop and adopt infrastructure that promotes data-sharing and data-driven research, and accelerate the growth of a cohesive data community that integrates contributors across domain, research, national, geographical and generational boundaries.
Association for computers and the Humanities: Digital Humanities Q&A
A community-based Q&A board for digital humanities questions that need (just a little) more than 140 character answers. Also see: https://twitter.com/dhanswers

Do more

University of Otago Solutions

ITS: Syncplicity Sync & Share
Syncplicity Sync & Share is a file synchronisation and sharing service provided by ITS which allows University of Otago staff (and other designated users) to securely share files with others and across different desktop computer systems and mobile devices.
ITS: High-Capacity Central File Storage (HCS)
The University of Otago provides a free high-capacity central file storage (HCS) service to University departments. HCS is an ideal solution for ongoing secure storage of research data.

Some Examples of Registries and Repositories of Open Data

Other sources of data specific to your discipline are often included in Subject Guides.

National Library of New Zealand: Open Data
Download or get API access to data from the Library and other organisations, and turn it into new apps, analysis, or reinterpretations.
https://www.data.govt.nz/
The data.govt.nz service aims to make open government data easy to find and use. Note: there is now a new beta site being developed at http://beta.data.govt.nz/
re3data
re3data.org is a global registry of research data repositories that covers research data repositories from different academic disciplines. It presents repositories for the permanent storage and access of data sets to researchers, funding bodies, publishers and scholarly institutions.
Research Data Australia
Research Data Australia helps you find, access, and reuse data for research from over one hundred Australian research organisations, government agencies, and cultural institutions.
UK Data Archive
A catalogue of over 5000 data collections, many of which are free to use. You will need to register to access data.
New Zealand Social Science Data Service
Now hosted by COMPASS: Centre of Methods and Policy Application in the Social Sciences (Auckland University)
The Qualitative Data Repository (QDR)
A dedicated archive for storing and sharing digital data (and accompanying documentation) generated or collected through qualitative and multi-method research in the social sciences. QDR provides search tools to facilitate the discovery of data, and also serves as a portal to material beyond its own holdings, with links to U.S. and international archives.
Inter-university Consortium for Political and Social Research
An international consortium of more than 750 academic institutions and research organizations. Maintains a data archive of more than 250,000 files of research in the social and behavioral sciences.
Europeana
Multi-lingual online collection of millions of digitized items from European museums, libraries, archives and multi-media collections. Currently offers four APIs for use.
British Library Collection Guides
A list of datasets for content mining, image analysis, digital mapping etc.
Gapminder
Gapminder Foundation is a non-profit venture registered in Stockholm, Sweden, that promotes sustainable global development and achievement of the United Nations Millennium Development Goals by increased use and understanding of statistics and other information about social, economic and environmental development at local, national and global levels.
Zanran
A search engine for finding data and statistics. The search results will be graphs, charts and tables.

Also see Spatial Data (in Tools>GIS)

Digitised Primary Sources and Datasets

Also see Text Mining

Marsden Online Archive
The Marsden Online Archive provides a number of clean data sets that you can use. ‘Clean’ means that all of the metadata, page numbers and TEI mark-up has been removed so the .txt file only contains the transcripts in its purest form.
JSTOR Data for Research (DfR)
The Data for Research (DfR) service is provided by JSTOR for use by the research community. It provides a set of web-based tools for selecting and interacting with content from the JSTOR archive, an archives that spans hundreds of years and billions of words of text. The DfR service is completely free and available to the public.
Alexander Street Press (ASP)
Electronic publisher specializing in the Humanities and Social Sciences. The Library subscribes to a number of their video and primary source databases. ASP allows harvesting of their data for Digital Humanities projects.
Digital Humanities | Adam Matthew
Adam Matthew publishes unique primary source collections from archives around the world. The Library subscribes to some of this content. Adam Matthew will provide free of charge access to data mine collections provided their formal request process is used.

Can I harvest data from any other Library Database?

Many of the databases the Library subscribes to do not allow unsolicited text/data harvesting in their licenses. Some will allow researchers to negotiate access (sometimes for a fee). Please consult with your Subject Librarian before attempting automated data scraping and large scale access to journal articles and databases as part of your project.

Digitalised Newspapers

Library Newspapers
Newspapers available online through the University of Otago Library.

What is Open Data?

Managing your research data
A University of Otago Library guide that provides information and links to support data management in five major areas of data-related activity.

Otago ITS supported software for qualitative analysis:

NVivo
NVivo is designed for researchers who need to combine subtle coding with qualitative linking, shaping, searching and modeling. NVivo is ideal for those working with complex data such as multimedia, who want to conduct deep levels of analysis.
ATLAS.ti
ATLAS.ti is a powerful workbench for the qualitative analysis of large bodies of textual, graphical, audio and video data.

Otago ITS supported statistical analysis applications:

SPSS
SPSS is a very easy-to-use statistical package that runs on Windows, Macintosh and UNIX platforms.
JMP
JMP is a statistical analysis application by SAS Institute, Inc. It dynamically links statistics with graphics to interactively explore, understand, and visualise data. This allows you to click on any point in a graph, and see the corresponding data point highlighted in the data table, and other graphs.
Stata
Stata is a complete statistical software tool that incorporates data analysis, data management and graphics. Stata is available for Windows, Mac and Linux computer platforms.

Using R to Analyze your Data

Humanities Data in R by Taylor Arnold; Lauren Tilton
Publication Date: 2015-09-23
RStudio
RStudio is an integrated development environment (IDE) for R. It includes a console, syntax-highlighting editor that supports direct code execution, as well as tools for plotting, history, debugging and workspace management.
R - Statistics Overview (Otago ITS Training)
R is a language and environment for statistical computing and graphics. Go through the steps of a data analysis using sample data. Set up and modify data files, run analyses, and plot graphs.
Video: R: The most powerful and most widely used statistical software
In the last ten years, the open source R statistics language has exploded in popularity and functionality, emerging as the data scientist's tool of choice. Today, R is used by over 2 million analysts worldwide, many having been introduced to its elegance and power in academia. Users around the world have embraced R to solve their most challenging problems in fields ranging from computational biology to quantitative finance, and to train their students in these same fields. The result has been an explosion of R analysts and applications, leading to enthusiastic adoption by premier analytics-driven companies like Google, Facebook, and and the New York Times.

Need More Computing Power?

NeSI: Computational Support
Make use of High Performance Computing (HPC) via NeSI. NeSI team members can work within your research team to improve the performance of your scientific software codes, enhancing memory and disk performance, optimising compilation and job management and scheduling, and in other advanced ways improving capability to make efficient use of HPC.

Also see:

Select a Data Repository

Where to Keep Research Data (DCC)
A checklist for evaluating data repositories.

Upload and Share Your Data

Figshare
Figshare is a repository where users can make all of their research outputs available in a citable, shareable and discoverable manner. Includes DOI.
Mendeley Data
Mendeley Data is a place where researchers can upload and share their research data for free. Datasets can be shared privately amongst individuals, as well as published to share with the world. Includes DOI.
Heurist
HEURIST’s research-driven data management system allows any confident researcher or data manager to design, create, manage, analyse and publish their own richly-structured database(s) within hours, through a simple web interface, without need of programmers or consultants. Sophisticated searches (including faceted and rule-based) can be saved, organised and rendered as layered interactive maps, timelines, network visualisations, cross-tabulations, lists, custom reports, XML and XSLT transforms, all of which can easily be printed or embedded (live) in a website. You can migrate later to a personal or institutional server, or access data directly via SQL. Use Sydney University’s free service to build a new database in minutes and a complete application in hours. For support, contact Ian Johnson

Need to Control Who You Share Data With?

ITS: Syncplicity Sync & Share
Syncplicity Sync & Share is a file synchronisation and sharing service provided by ITS which allows University of Otago staff (and other designated users) to securely share files with others and across different desktop computer systems and mobile devices.

Publish in a Data Journal

ANDS: Data Journals
This guide is for librarians, researchers and data managers who are involved in the deposition, publication, curation and citation of published data. It includes examples of data journals and outlines data submission criteria.

Get a DOI for your Data

DataCite
DataCite is a leading global non-profit organisation that provides persistent identifiers (DOIs) for research data. Their goal is to help the research community locate, identify, and cite research data with confidence.

Move Large Amounts of Data

HCS High Speed Data Transfer
Researchers who store data on the ITS high capacity storage (HCS) service can securely transfer large research data sets at the highest available network speeds to NeSI high performance computing facilities around New Zealand and to other research facilities around the world.

How to Cite Data

ANDS: Data Citation
Data citation refers to the practice of providing a reference to data in the same way as researchers routinely provide a bibliographic reference to outputs such as journal articles, reports and conference papers. Citing data is increasingly being recognised as one of the key practices leading to recognition of data as a primary research output
DCC: How to Cite Datasets and Link to Publications
This guide will help you create links between your academic publications and the underlying datasets, so that anyone viewing the publication will be able to locate the dataset and vice versa.

Get a DOI for your Data

DataCite
DataCite is a leading global non-profit organisation that provides persistent identifiers (DOIs) for research data. Their goal is to help the research community locate, identify, and cite research data with confidence.

Cleaning Data Guides

The Quartz guide to bad data
An exhaustive reference to problems seen in real-world data along with suggestions on how to resolve them.
A Gentle Introduction to Data Cleaning
This School of Data course is a gentle introduction to reducing errors by cleaning data. It was created by the Tactical Technology Collective and gives you a clear overview what can go wrong in spreadsheets and how to fix it (if it does).

OpenRefine (formally Google Refine)

OpenRefine
OpenRefine (formerly Google Refine) is a powerful tool for working with messy data: cleaning it; transforming it from one format into another; and extending it with web services and external data.

OpenRefine: Digital Humanities focused tutorial

Learn the basics

Sage Methods Space: Learn the Essentials of Data Visualization
In this one-hour webinar, the presenters address the basics of data design and demonstrate various ways for showing different types of data.

Tools to create beautiful visualisations:

See Visualisation (in Tools)

See Create Maps (in Tools>GIS)

Be Inspired

The beauty of data visualization
David McCandless turns complex data sets, like worldwide military spending, media buzz, and Facebook status updates, into beautiful, simple diagrams that tease out unseen patterns and connections. Good design, he suggests, is the best way to navigate information glut -- and it may just change the way we see the world.
The best stats you've ever seen
With the drama and urgency of a sportscaster, statistics guru Hans Rosling uses an amazing new presentation tool, Gapminder, to present data that debunks several myths about world development. Rosling is professor of international health at Sweden's Karolinska Institute, and founder of Gapminder, a nonprofit that brings vital global data to life. (Recorded February 2006 in Monterey, CA.)

Sources of Open Statistical Data

Other sources of statistical data specific to your discipline are often included in Subject Guides.

https://www.data.govt.nz/
The data.govt.nz service aims to make open government data easy to find and use. Note: there is now a new beta site being developed at http://beta.data.govt.nz/
Statistics New Zealand
Official New Zealand statistics on the Census, Business, Education, Agriculture, Households, Industry and much more.
UN Data
Brings UN statistical databases within easy reach of users through a single entry point. Search and download a variety of statistical resources, and worldwide statistics, including populations, demographics, trade, commodities, and more.
OECD iLibrary
The Organisation for Economic Co-operation and Development's statistical databases.
UNESCO Data Centre
The UNESCO Institute for Statistics (UIS) collects the data for more than 200 countries from Member States and international organizations. Contains over 1,000 types of indicators and raw data on education, literacy, science and technology, culture and communication. You can download predefined tables or build your own. Summary data (profiles and global rankings) are also available.
World Bank Open Data
Free and open access to global development data. Keyword search or browse by country or indicator. Homepage includes links to other datasets such as World Development Indicators and the Global Consumption Database.

Integrated Data Infrastructure (IDI) and Confidentialised Unit Record Files (CURFs)

The Integrated Data Infrastructure (IDI)
The Integrated Data Infrastructure (IDI) is a large research database containing microdata about people and households. Data is from a range of government agencies, Statistics NZ surveys including the 2013 Census, and non-government organisations. The IDI holds over 166 billion facts, taking up 1.22 terabytes of space – and is continually growing. Researchers use the IDI to answer complex questions to improve outcomes for New Zealanders.

Contact access2microdata@stats.govt.nz for further information on data location, security, privacy and confidentiality. From 1 July 2014 data access costs will be reduced and new remote access locations set up. Overview page - Access our microdata.

Confidentialised unit record files (CURFs)
Confidentialised unit record files (CURFs) are unit record data that have been modified to protect the confidentiality of respondents while also maintaining the integrity of the data. Modification involves applying methods such as top-coding, data swapping, and collapsing categorical variables to the unit records. The end result is a dataset that can be made available to researchers.

Learn more about the IDI, CURFs and other sources of New Zealand Statistics, by visiting Statistics for Research on the Library's NZ Statistics guide.

You can also consult Thelma Fisher, Contact Officer for Statistics NZ at the University of Otago.

Data Mining / Text Mining

Data or Text mining is where computer software applies automated analytical techniques to interrogate data sets for patterns, trends and other useful information that typically would be incredibly labour intensive or difficult to conceive by traditional human research.

Learn more

Data Mining and Text Analysis
Data mining in the digital humanities usually involves performing some kind of extraction of information from a body of texts and/or their metadata in order to ask research questions that may or may not be quantitative.

Digitised Primary Sources and Datasets

Alexander Street Press (ASP)
Electronic publisher specializing in the Humanities and Social Sciences. The Library subscribes to a number of their video and primary source databases. ASP allows harvesting of their data for Digital Humanities projects.
Marsden Online Archive
Search the digitised letters and journals of the Reverend Samuel Marsden, along with other early NZ missionaries.
JSTOR Data for Research (DfR)
The Data for Research (DfR) service is provided by JSTOR for use by the research community. It provides a set of web-based tools for selecting and interacting with content from the JSTOR archive, an archives that spans hundreds of years and billions of words of text. The DfR service is completely free and available to the public.
GALE: Data Mining and Textual Analytics
We do not currently subscribe to this resource. Please contact your Subject Librarian if you would like to arrange a trial.
Adam Matthew
Adam Matthew publishes unique primary source collections from archives around the world. The Library subscribes to some of this content. Adam Matthew will provide free of charge access to data mine collections provided their formal request process is used.
HathiTrust Datasets
HathiTrust makes the texts of public domain works in its corpus available for research purposes. The works fall into two categories: non-Google-digitized volumes, which are freely available, and Google-digitized volumes, which are available through an agreement with Google. Within each category there is a distinction between public domain works available only in the US versus public domain works available anywhere in the world.
Internet Archive: Free eBooks and Texts
The Internet Archive and Open Library offers over 10,000,000 fully accessible books and texts. Information on downloading in bulk provided in the official blog: https://blog.archive.org/2012/04/26/downloading-in-bulk-using-wget/
Michigan State University: Datasets for Digital Research
Datasets for text mining or other "non-consumptive" research, that is, research conducted by computational methods which does not reproduce significant portions of text for personal or public display. Most, but not all, are available to non-MSU users.

Can I harvest data from any other Library Database?

What tools are available for text analysis and topic modeling?

See Text mining & analysis

New to APIs? Learn more:

Video: What is an API?
API’s provide a standard way of accessing any application, data or device whether it is shopping from your phone, or accessing cloud applications at work.
An Introduction to APIs
In this course, we walk you through what it takes for companies to link their systems together. We start off easy, defining some of the tech lingo you may have heard before, but didn’t fully understand. From there, each lesson introduces something new, slowly building up to the point where you are confident about what an API is and, for the brave, could actually take a stab at using one.

New Zealand API examples:

OUR Heritage OAI-PMH Repository
A window into the cultural, heritage and historical material within the University of Otago's Hocken Collections and Special Collections. Harvesters can access metadata from this site at: http://otago.ourheritage.ac.nz/oai-pmh-repository/request
DigitalNZ for Developers
The data available through DigitalNZ describes the digital items held by content partners across all walks of NZ life. You can use the APIs to search across the likes of titles, descriptions, dates, and creators of the material; as well as requesting thumbnails and the full metadata record for each item.
Using data.govt.nz APIs
You can access both dataset metadata and machine-readable datasets in data.govt.nz using the JSON format.
National Library of New Zealand: Open Data
Download or get API access to data from the Library and other organisations, and turn it into new apps, analysis, or reinterpretations.

Other API examples:

Trove API technical guide
A basic overview of the Trove API can be found here. Trove helps you find and use resources relating to Australia bringing together content from libraries, museums, archives, repositories and other research and collecting organisations.
Case Studies | Create Something with Trove
Trove's API has been central for people building new tools, interfaces and visualisations. We hope you can gain inspiration from those who have already created with the API.
Our APIs | Europeana Pro
Multi-lingual online collection of millions of digitized items from European museums, libraries, archives and multi-media collections. Currently offers four APIs for use.
HathiTrust APIs
HathiTrust has developed a Data API that makes it possible to retrieve page images, OCR text, rights information, and a variety of other data about objects in the repository. There is also a Bib API that returns bibliographic, copyright, and volume information (including permanent URLs) when queried with a variety of standard identifiers (e.g., ISBN, LCCN, OCLC, etc.).

API Directory:

The ProgrammableWeb: API directory
Search the largest API directory on the web. Since it was founded in 2005, ProgrammableWeb has been amassing the Web's most relied-on directory when it comes to discovering and searching for APIs to use in Web and mobile applications.

If you are interested in whether a specific Library database has an API please contact your Subject Librarian.

We want your feedback!

This guide continues to evolve, and we really welcome your feedback so we can continue to improve it. Please let us know if you find:

Incorrect or irrelevant details, tools that don't work, dead links or otherwise unhelpful information
Helpful details, tools, links or information that you think need to be on the guide, but aren't currently.

We'd also love to hear from you if you want to have your project featured on the guide, or would like to be profiled on the Connect&Collaborate@Otago page. Email Alexander Ritchie, with any comments or suggestions on how we can improve it.

Viva Digital Humanities!