Predicting Innovative Firms Using Web Mining and Deep Learning
12 Pages Posted: 23 Jan 2019 Last revised: 7 Feb 2020
Date Written: January 1, 2019
Innovation is considered as a main driver of economic growth. Promoting the development of innovation through STI (science, technology and innovation) policies requires accurate indicators of innovation. Traditional indicators often lack coverage, granularity as well as timeliness and involve high data collection costs, especially when conducted at a large scale. In this paper, we propose a novel approach on how to create firm-level innovation indicators at the scale of millions of firms. We use traditional firm-level innovation indicators from the questionnaire-based Community Innovation Survey (CIS) survey to train an artificial neural network classification model on labelled (innovative/non-innovative) web texts of surveyed firms. Subsequently, we apply this classification model to the web texts of hundreds of thousands of firms in Germany to predict their innovation status. Our results show that this approach produces credible predictions and has the potential to be a valuable and highly cost-efficient addition to the existing set of innovation indicators, especially due to its coverage and regional granularity. The predicted firm-level probabilities can also directly be interpreted as a continuous measure of innovativeness, opening up additional advantages over traditional binary innovation indicators.
Keywords: Web Mining, Web Scraping, R&D, R&I, STI, Innovation, Indicators, Text Mining, Natural Language Processing, NLP, Deep Learning
JEL Classification: O30, C81, C83
Suggested Citation: Suggested Citation