The Missing Operating System
To get value from your data today requires a dozen separate systems, one for data ingestion, one for data ETL and transformation, one for data visualization, another for orchestration and automation, another for "reverse ETL", yet another for machine learning operations, one for data lineage, one for data quality -- you get the idea. This is the "Homebrew computer club" days of cloud data -- you have to patch these systems together, be an expert in each, constantly battle incompatibilities, and re-discover every small solution at every step.
All these problems were solved in the PC era by traditional operating systems. Patterns is built from first principles to be the new Operating System for your Data Warehouse. It abstracts over cloud data warehouses, storage, and compute to allow businesses to build and run full-stack data applications on a single unified platform. With Patterns, you can ingest with Python, transform with SQL, train with PyTorch, dashboard with Vega, and automate with webhooks, all in one place, with unified events, lineage, monitoring, permissioning, and a consolidated bill.
Just like a traditional operating system, Patterns manages the messy interface and orchestration so application developers can focus on business value and end users can get incredible, powerful solutions with a few clicks.
Traditional operating system:
- Abstracts over hardware with "drivers" - harddisk, cpu, peripherals
- Defines standard data and program formats - exe, stdout
- Orchestrates program execution and resource management
- Handles user input and output, provides standard user interface
Patterns - Data warehouse operating system
- Abstracts over cloud providers - databases, compute, storage
- Defines standard data and program formats - schemas, python and SQL functions, table and stream stores
- Orchestrates pipeline and data flows, manages cloud resources
- Provides user interface, development environment, and visualizations and interactives
The Data Warehouse App Store
The transformational power of operating systems comes from the developer ecosystem and marketplace of applications that they enable. At the heart of the Patterns vision is a rich ecosystem of components and full-stack apps that empower modern businesses.
To enable such an ecosystem requires solving the problem of "exploding pipelines": there are hundreds of vendors that are required data sources and sinks for modern businesses -- ingest your data from Stripe, Facebook ads, Zendesk, etc, export your analysis back to Salesforce, etc. The list of vendors to support is huge. In addition, each business vertical has its own requirements and needs for the data from these systems. So you end up with 100s x 100s x 10s of different data paths that need to be supported.
Patterns solves this combinatorics problem with canonical schemas for core entities -- like Customer, Transaction, etc -- along with translations from every source and sink, so application developers and users can work with gold-standard, quality-assured data and build solutions for the entire market, not just a specific vendor.
For instance, an application developer on Patterns can build a robust Forecasted Customer Lifetime Value app using the latest statistical modeling and machine learning techniques, building it off of common Transactions and CustomerAttributes schemas. The Patterns ecosystem provides standard ETL transformations for dozens of data sources into the standard Transactions schema -- Stripe, Shopify, Square, Amazon -- all can be conformed to a standard schema so now the user has access to the Forecasted Customer Lifetime Value app, along with hundreds of others and the developers don't have to build the NxN custom solutions.