Automation of Insights:

Data Pipelines

  • An effective Data Pipeline can dramtically speed up refeshing data and performing analysis
    I have created a simple pipeline that provides automated udpates to interative dashboards (diagram below).

    Data Pipeline Process


    Step 1: Data is extracted with a laptop using a Python script connecting to the Federal Reserve Economic Data
    Step 2: Python script loads data to Google Sheets
    Step 3: Tableau Server has a live connection to Google Sheets and any udpates will be reflected on the dashboard
    Step 4: The interactive dashboard is automatically updated

    This flow brings together over 150 tables.
    The 3 data sources from the Ferderal Reserve have spererate tables for each of the 50 states as well as
    the District of Columbia and Puerto Rico.

    Both dashboards get updated with one script
    1. Tableau dashboard to explore US Unemployment data set.
    2. Tableau dashboard to explore GDP, Unemployment and Voting at a state level.

    This process could be made even more effecient by using a server that is always on to execute this data pipeline on a scheduled basis,
    resulting in a fully automated process that does not require any one to execute the Python script in Step 1.