A labour of love: 10 years of labour market research at Nesta

Nesta's data science team shares insights from five years of building their own job advert scraping algorithm and developing open source tools for labour market analysis.

The team invested six months in 2020 to create their own job scraping system rather than purchasing data, which has provided ongoing flexibility for experiments and research projects. Working with large datasets of unstructured job postings became a catalyst for developing advanced data science capabilities across machine learning, natural language processing, and cloud engineering.

• Making their skills extraction algorithm open source led to valuable user feedback that improved cross-platform compatibility and identified critical bugs
• Managing diverse job advert data enhanced team expertise in entity recognition, semantic mapping, clustering, and dashboard development
• External users prompted important improvements like switching to Poetry for dependency management and hosting models on HuggingFace
• The investment in custom scraping tools paid off through reduced data costs and increased freedom to explore innovative methodologies