Skip to main content

7 posts tagged with "release"

View All Tags
Chris Stanley

Public beta release

We're excited to announce that after a year of heads-down product development, a beta version of Patterns is open to the public to self-serve.

Create an account here.

This is an important milestone on our mission to make data more accessible. One of the hardest parts about data science is knowing where to get started, what tools to learn, what business problems to study, etc.

I remember in 2014, when I left my job in finance, and set out on a journey to reskill myself to join a tech company, I learned SQL through Alan Beaulieu's Learning SQL, learned data science through Harvard's CS109 Data Science course, and eventually landed a job a Square.

During my whiteboard SQL interview there, I wrote my answers in a single run-on line becasue this was the only way to write SQL in a terminal:

select customer_id, date, sum(amount) as amount from payments group by customer_id, date 

Lucky for me, my boss was understanding of my inexperience, and saw my commitment to learning technical subjects. To get this far I had to overcome setting up a local db, setting my bin/bash profile (I still don't know what I'm doing here), download and run python (never have the right version)... and this is just 1/10th of what's required to set up a bare bones analytics stack at a company. It. Should. Not. Be. This. Hard.

With Patterns, it's no longer this hard.

A new way to add nodes

We want using our product to be as enjoyable as playing a video game. To that end, we copied a common UX pattern in video games such as Command and Conquer for adding new items to a gameplay canvas. See below for show

Linear version history

While it's not quite git style version control (which we support by managing graphs via our devkit), a linear version history, and the ability to revert back to a prior version, is a powerful feature set for managing development on complex projects. After viewing the state of a prior project you can easily revert the current app back to that prior state, download the zip file, or clone the version into a entirely new app.

Posthog App

We use Posthog internally for a ton of different analytical and operational use cases. We support receiving events from Posthog via a Posthog App, integratef within their product. We also support extracting data from Posthog's API and have built a component within Patterns for this.

Chris Stanley

The Graph is Alive

User interface and interactivity is one way we differentiate our product. To that end, this month we implemented websockets and animated our graph with events coming from the server to communicate the state of our data graph at any point in time. With a client app, api server to manage, and cloud infrastructure where the functions actually execute, this was a cross-functional feature in the making for the past few months. Never again be left in the dark about the state of your pipelines.

Introducting Apps

As we continue our development on dashboards, it didn't seem write that we were calling everything a Graph, we need a new term to encapsulate the idea of a directed graph and dashboard as user interface objects --- we arried at App.

IDE modal

It's hard to keep an entire system in your head at the same time. Graphically laying out your data flow is incredibly helpful for understanding complex system. Even more helpful is being able to have multiple IDE windows open at the same time.

New node designs

The graph is an incredibly powerful interface, and can help with communicating complex ideas if make intuitive enough. In Patterns the graph can contain python or SQL functions

OAuth and Shopify

We layed foundational work for secure OAuth connections to authenticate access to push/pull data to external systems such as SaaS APIs. The first one we implemented was Shopify.

Chris Stanley

Marketplace, dashboards, connections, and our new name... Patterns

  • Marketplace - A core value prop to our product is the ability to share, discover, and clone entire data solutions. This can be either a single component to your product, like a Python script to ingest Stripe Charges, or a whole use case like an ML model to score new inbound leads and sync'ing this data with Salesforce. With our Marketplace now live, anyone can discover, share, and clone components and data apps.
  • Dashboards - Patterns is now great for sharing your analytical work with external stakeholders with dashboards. Now you can drag and drop views of your markdown, chart, or table nodes into a view only dashboard for sharing with external stakeholders via public or private url.
  • Connections - There are now two ways to authenicate with external services. From day 1 we supported secrets, used in authentication calls to external APIs and services. Now you can authenticate with OAuth and use the provided token in API
  • Patterns, our new name - After much deliberation and thought, we've decided to rebrand to Patterns. We think this is a better name to communicate our full vision of building a data warehouse operating system, with reuseable components and entire data apps (patterns...)
Chris Stanley

This month our team focused on improvement of existing feature sets, squashing bugs, and performance optimizations:

  • Storage configuration - We improved the UI/UX for linking nodes in your graph. Now, edges are implied by your function code and if stores don't exist, the app highlights this configuration detail and asks you to add streams/tables if they don't exist.
  • Parameters - Paramater configuration just got 10x easier. We now provide users with a drop-down menu of all avalaible parameters instead of requiring them to follow a specific syntax rule eg. ${{parameter_name}}.
  • Code completion - Both Python and SQL IDEs now have code completion for core Patterns objects such as tables, streams, and parameters. First time users can now begin using core Patterns objects almost without reading any documentation.
  • 10x speed increase - We've implemented optimizations for graph saving and versioning, which accelerated some UI features by 10x.
  • Bug fixing - We've had a backlog of bugs from new feature development that this month we began to solve. Many of the improvements above automagically fixed bugs, but there were a host of others that required specific attention that we attended to this month.
Chris Stanley

This week we added a number of core features, please see below for details.

  • Explicit Stores - Before, a node's storage (table or stream) was contained within the node abstraction. This made it difficult for a user to understand 'where' their data lives and 'how' their node wrote data to those stores. We made the decision to make node stores explicity objects in the graph and UI to make this more clear to the user.
  • Isolated Envs - Customers can now sign-up for Patterns and are granted isolated compute and storage environments in GCP. This improves individual and aggregate scalability of the Patterns platform.
  • Chart nodes - Developers can now utilize Vega-lite chart capabilitiesto specify visualizations for table data.
  • Webhook nodes - Developers can now utilize webhooks in their graphs. After you drag+drop a webhook, you receive an endpoint that you can send arbitrary data to.
  • Organizations - Teams can now collaborate together in Patterns with our organization settings feature.
  • Social Auth - Login wih your Google account using Auth0.
Chris Stanley

For the month of March we set our focus on the Studio App development experience and shipped a number of foundational features that make it easy for anyone -- data engineers, scientists, and even less technical analysts -- to build production computational graphs in Patterns.

  • In app code editing, saving, and deployment of graphs - This feature set is a game-changer. Most data science products in the market require separate tooling for development and deployment of data pipelines. Now with Patterns anyone can deploy solutions from our web UI without having to touch a single piece of cloud infrastructure.
  • Support for Components - No one likes writing code from scratch that they know someone has already written. This is particularly true for data engineers who often need to integrate common data sources, such as APIs from SaaS vendors like Stripe or Salesforce. Now you can use one of our many pre-built components by just dragging/dropping a component into your graph.
  • Try Runs - When developing a node, it's common to only want to 'test' a node, and not write any of it's output data to storage. Now you can select for a node to 'run' or 'try' which allows you to test code before it's deployed and set to execute on production data stores.
Chris Stanley

We had a foundational month in February. After adding three members to our team and having a focused January, we happy to announce a series of product updates that you can read below:

  • Shipped V1 of the Patterns devkit - this allows users to devleop Patterns graphs locally.
  • Shipped V1 of the Patterns Studio app - this enables users to view the execution and structure of their graphs. This is a foundational step towards enabling users to view and edit their graph in the UI.
  • SOC-2 This month we began implementing requirements that get us closer towards SOC-2 certification.

Stay tuned - there is much to come!