Skip to main content

Importing data

There are many ways to get data into Patterns. We'll discuss a few here at a high-level here and go into specifics elsewhere. At a high-level there are 5 ways to get data into Patterns.

  1. Import from a connected database
  2. Import using a python node (CSV, API, etc.)
  3. Receive data via webhooks
  4. Use a pre-built component

Database

You can connect and add data from Postgres, BigQuery, or MySQL databases. Support for other databases is on the way. Connecting to a read-only database will make that database accessible for copying data. That is, when you write queries against read-only databases, Patterns will extract query results and insert them into a new table withing the configured storage for your app. Unless configured to use a different database, Apps use the Patterns Postgres DB by default for writing intermediate results.

Python

Use Python nodes to get data from external systems using python requests or read from a hosted file using pandas

  1. Read CSV - as demonstrated in our Quick Start tutorial
  2. GET from API - as demonstrated in this ISS Current Location Graph
get-iss-current-location.py
from patterns import State, Streamfrom datetime import datetime, timedeltaimport requestsdetect_freq = timedelta(seconds=5)desired_data_points = 60# Connect to a stream in write mode:iss_locations = Stream("iss_locations", "w")# Use State to track how many runs we havestate = State()data_count = state.get_value("data_count", 0)if data_count >= desired_data_points:    # If we have enough data, exit    exit()# Fetch the current location of the International Space Stationresp = requests.get(    "http://api.open-notify.org/iss-now.json",)# Write the location to the streamiss_locations.write(resp.json())

Webhooks

Patterns webhook nodes generate an endpoint on the Patterns platform for you to POST data to. Add a webhook node by dragging the node into your graph. After a few moments, the webhook will generate a url that you can then send form-encoded or JSON data to via HTTP POST. Your webhook will be automatically connected to an output stream where webhook request payloads will appear.

Component

Explore components from our marketplace that are easy to configure and provided extended functionality for importing data from third-party APIs and databases.