The social sciences and the web: From ‘Lurking’ to interdisciplinary ‘Big Data’ research

Abstract

‘Big data’ is an area of growing research interest within sociology and numerous other disciplines. The rapid development of social media platforms and other web resources offer a vast and readily accessible repository of data associated with participants’ activities, attitudes and personal information on a scale and depth that would have previously been difficult to access without substantial resources. However, as well as offering opportunities to social researchers, this medium also presents a significant range of challenges. Ethical issues are one much debated area where social scientists are having to reassess their longstanding modus operandi, given questions regarding access to personal data and ambiguities regarding its status and legitimate usage. In addition, the scale of data accessible and the possible skills required to collect and analyse it is also a critical issue, and is an area that, arguably, has received lesser attention. In its infancy, online research could be fairly rudimentary, employing simple techniques to gather information from weblogs, forums and so on. However, the possibilities now presented by large-scale social media platforms has created the potential for more sophisticated research that often requires specialist technical expertise, involving collaborative work by computer and social scientists working together. This is a scenario that raises its own concerns, not least in terms of forging areas of shared understanding between these disparate disciplines sufficient to facilitate such projects. This article addresses such issues, providing a reflection on the theoretical and practical experience of engaging in online research, from fledgling involvement to embarking on a current collaborative interdisciplinary project. The aim is to provide some insights to other social scientists with respect to some of the potential advantages and pitfalls of web research, while a flavour of the current project, exploring Scottish Referendum and UK General Election related Twitter data, is also presented.