Automating Insights

Building a full stack data solution to process TypeForm responses

Context

A couple of years ago, I started working remotely as a digital nomad. My wife and I embraced the flexible working environment, with its rewards of seeing the sights, eating the local food, and meeting new people during evenings and weekends. The digital nomad community is diverse and inclusive, lending itself to many discussions about ups and downs of our chosen career paths.

I wanted to explore the digital nomad lifestyle and help others understand the community. Having designed a survey about work, accommodation and finances, I developed a full-stack data solution to capture and analyze the valuable insights that were generated. In this article, I’ll walk you through the step-by-step process of building this solution, from survey design to deploying a user-friendly app to access the insights.

Want to take the survey yourself?

Take survey

DESIGNING THE SURVEY WITH TYPEFORM

To capture the experiences of the digital nomad community, I turned to TypeForm—an engaging and user-friendly survey platform that has built-in web hooks (this will be important later).

The survey starts with a few introductory questions to establish basic demographics. It then delves into certain core topics – namely people’s work preferences, travel habits, and accommodation choices. The intention is to add more topics over time, as I learn more about digital nomads’ needs and pain points. I’ve attempted to structure the survey with a logical flow, so that respondents are only answering questions about their specific areas of interest.

ANALYZING THE SURVEY DATA

python
Analyzing survey data isn’t straightforward. There are multiple question types that may have more than one answer, causing a lot of nested data. Data analysis on semi-structured data can be done, but most business intelligence tools work best with data in a relational table. For this reason, I wrote some custom Python code that has two outputs:
  1. It groups and transforms responses into a table that can be loaded into a relational database;
  2. It creates a meta data table that describes the questions and provides essential information for carrying out analysis.
After writing the code and testing it locally, I needed to find a way to store that data and make it publicly accessible.

USING DETA SPACE TO TRANSFORM DATA

With the survey designed and Python code tested, I needed a seamless solution to capture, transform, and store the data in order to be able to visualize it. Enter Deta Space—a powerful hosting platform that provided the ideal infrastructure for my data transformation needs. Leveraging Deta’s microservices capabilities, I set up a Python script that would listen for new survey responses (via Typeform’s web hook) and automatically trigger a data transformation process. This script parsed the raw survey data, extracting relevant information and structuring it in a format suitable for analysis. I now had a smooth and efficient data pipeline to automatically process the data and make it accessible for analysis and visualization.

CREATING WEB APP VIA STREAMLIT

streamlit

To make the insights accessible to the digital nomad community, I decided on an easy-to-use cloud application called Streamlit. This web-based app provided an intuitive and interactive interface for users to log in and explore the visualized data. However, I wanted to drive people to fill in the survey, so I decided to require a password login to view the results. This way, only people who fill out the survey receive login details. Anybody else who is curious can always send an email to request access. By deploying this app, I bridged the gap between data analysis and user accessibility. And in the process, I hope to help the digital nomad community by providing insights to inform their lifestyle choices and enhance their nomadic journeys.