Snowflake Openflow in Practice

This post was written by an engineer at QueryPlane. QueryPlane is an app builder for your database: bring your own postgres db and you can create interactive applications to share with other developers, coworkers or even your customers. If you’re interested in trying it out, get started here.

---

For years the answer to “how do I get this data into Snowflake?” was a stack of tools: Fivetran or Airbyte for SaaS sources, Debezium plus Kafka plus a Snowpipe Streaming connector for database CDC, custom scripts for everything else. Each tool came with its own control plane, its own pricing model, and its own pile of YAML.

Snowflake’s Openflow collapses most of that into a single managed service running inside the Snowflake account boundary. It is a managed Apache NiFi — visual flow design, hundreds of processors, plus a curated set of Snowflake-authored connectors for sources like PostgreSQL CDC, MySQL CDC, Kafka, and Google Drive. The Snowflake-hosted deployment that runs on Snowpark Container Services went GA on November 4, 2025, which makes it credible for production work rather than a preview to file under “interesting.”

This post walks through Openflow as it actually behaves in production: the deployment model, the Postgres CDC connector (because that is where most teams will start), the cost mechanics, and where the sharp edges are.

In this post, we’ll cover:

• What Openflow actually is — the NiFi heritage, the runtime/deployment/connector model
• BYOC vs Snowflake Deployment — which deployment type to pick and why
• Setting up a Snowflake Deployment — the roles, grants, and compute pool
• The Postgres CDC connector — snapshot + logical replication, schema evolution, the soft-delete model
• The Kafka connectors — bidirectional patterns and how they compare to the classic Snowflake Kafka Connector
• Cost and scaling — compute pool sizing, the always-on management cost, autosuspend
• When to choose Openflow vs Snowpipe / Snowpipe Streaming / dbt
• Production patterns and pitfalls

What Openflow Actually Is

Openflow is “Apache NiFi as a service,” but the layering matters because the terminology shows up everywhere in the UI. There are three concepts to internalize before you click anything:

A deployment is the data-plane container. It is either a Snowflake-managed compute environment (running on Snowpark Container Services) or a Kubernetes cluster you operate in your own AWS account. The deployment is the boundary at which compute is provisioned.

A runtime lives inside a deployment and is the execution unit for one or more pipelines. A runtime maps to a NiFi instance. When you run multiple unrelated pipelines, you typically spin up multiple runtimes inside the same deployment so they scale independently.

A connector is a curated, versioned NiFi flow template authored by Snowflake. The Postgres CDC connector, the Kafka connector, and the Google Drive connector are the most commonly used. Connectors run inside a runtime. A runtime can also host raw NiFi flows you build yourself on the canvas — the connectors are not a separate runtime type.

The thing that makes this practical, as opposed to “host your own NiFi cluster,” is that everything is wired into Snowflake’s auth model. Runtimes authenticate to Snowflake using a Snowflake Managed Token — a short-lived, auto-rotated credential — rather than a static keypair you have to provision and rotate yourself.

BYOC vs Snowflake Deployment

There are two deployment types, and the choice is mostly about where the data plane physically runs.

Snowflake Deployment runs the data plane inside Snowflake on Snowpark Container Services. You do not see the underlying nodes, you do not configure VPCs, and the cost shows up on the Snowflake bill alongside warehouse credits. This is the deployment type that went GA in November 2025, and it is available across AWS, Azure, and GCP commercial regions. It is the default starting point for almost every team.

BYOC (Bring Your Own Cloud) runs the data plane in your own AWS account using a CloudFormation template that Snowflake supplies. The Snowflake-managed control plane still hosts the UI, the catalog, and the orchestration, but the runtime nodes execute inside your VPC. BYOC is available in AWS Commercial regions and you pay AWS for the EC2/EKS bill plus a Snowflake control-plane charge.

Choose BYOC when sensitive data needs to be preprocessed before it leaves your VPC (PII redaction at the edge), when on-premise sources require private connectivity you already terminate in your AWS account, or when you need to use connectors that depend on internal services not reachable from Snowflake’s network. Otherwise, default to Snowflake Deployment — there is materially less operational surface area.

Setting Up a Snowflake Deployment

The setup is mostly SQL grants in your Snowflake account plus a few clicks in Snowsight. The official quickstart calls it a 25-minute one-time setup; in practice the SQL side takes ten.

The roles and grants follow Snowflake’s standard “create an admin role, grant it the integration privileges” pattern:

USE ROLE ACCOUNTADMIN;

CREATE ROLE IF NOT EXISTS OPENFLOW_ADMIN;
GRANT ROLE OPENFLOW_ADMIN TO USER <openflow_user>;

-- Account-level privileges Openflow needs
GRANT CREATE OPENFLOW DATA PLANE INTEGRATION ON ACCOUNT TO ROLE OPENFLOW_ADMIN;
GRANT CREATE OPENFLOW RUNTIME INTEGRATION ON ACCOUNT TO ROLE OPENFLOW_ADMIN;
GRANT CREATE COMPUTE POOL ON ACCOUNT TO ROLE OPENFLOW_ADMIN;

-- The Openflow login flow refuses ACCOUNTADMIN as a default role
ALTER USER <openflow_user> SET DEFAULT_ROLE = OPENFLOW_ADMIN;
ALTER USER <openflow_user> SET DEFAULT_SECONDARY_ROLES = ('ALL');

The DEFAULT_ROLE switch is not a style preference — the runtime login UI explicitly rejects users whose default role is ACCOUNTADMIN, which is a reasonable guard but trips up almost everyone the first time.

From there, navigate to Ingestion » Openflow in Snowsight, accept the terms as ORGADMIN, and create a deployment. Picking the deployment type is the only meaningful question; the rest is region and a name. Once the deployment exists you create a runtime inside it. The runtime size determines the instance family the underlying compute pool uses:

Runtime size	vCPUs	Memory	SPCS instance family
Small	1	2 GB	CPU_X64_S
Medium	4	10 GB	CPU_X64_SL
Large	8	20 GB	CPU_X64_L

The compute pool autoscales between a minimum and maximum node count you set, and the runtime is what owns the pool’s lifecycle. The non-obvious part is that Openflow also runs a management compute pool on a single CPU_X64_S instance regardless of whether you have any active runtimes — that pool is always on, and it shows up on the bill even when nothing is flowing. Plan for that as the baseline cost of having Openflow installed.

The Postgres CDC Connector

The Postgres connector is the one most teams reach for first, and it is the strongest illustration of how the connectors are designed. It targets PostgreSQL versions 11 through 18, including AWS RDS, Aurora, GCP Cloud SQL, and Azure Database for PostgreSQL.

The replication model is three-staged: schema introspection (the connector reads pg_attribute to discover columns, validates them against Snowflake type limits, and creates the destination tables), an initial snapshot via COPY, then continuous logical replication using a replication slot and the pgoutput plugin. The output table looks just like the source plus three metadata columns: _SNOWFLAKE_INSERTED_AT, _SNOWFLAKE_UPDATED_AT, and _SNOWFLAKE_DELETED.

The data-retention behavior is the most opinionated part of the connector, and it is worth understanding before you turn it on. Customer data is never automatically deleted on the Snowflake side:

• Deleted source rows are soft-deleted — the row stays in the destination with _SNOWFLAKE_DELETED = TRUE set.
• Dropped source columns are soft-deleted — the column is renamed with a __SNOWFLAKE_DELETED suffix.
• Journal tables retain every change indefinitely unless you put a retention policy on them.

That model is great for analytical and audit use cases (“what did this row look like a week ago?”) and surprising if you assumed CDC meant a 1:1 mirror. If you need a hard-delete mirror, you build a downstream view that filters on _SNOWFLAKE_DELETED = FALSE.

A few prerequisites bite teams who haven’t read the fine print. The runtime must be Medium or larger, and it must be configured as single node (min_nodes = 1, max_nodes = 1). The connector cannot horizontally scale a single Postgres source because the replication slot has a single consumer. Each replicated table also needs either a primary key with replica identity DEFAULT, or a unique index with replica identity USING INDEX:

-- Required on every table you want to CDC from
ALTER TABLE orders REPLICA IDENTITY DEFAULT;

-- Or, if no primary key but a unique index exists
ALTER TABLE orders REPLICA IDENTITY USING INDEX orders_id_uniq;

Without either, only INSERT operations replicate — UPDATE and DELETE will arrive in the WAL but the connector cannot identify which row to modify in the destination.

Schema evolution is automatic but worth knowing the rules of. New source columns are added to the destination automatically and start replicating going forward, with NULL in existing rows. Column drops become soft-deletes. Column renames are interpreted as a drop + add — the destination ends up with both the old column (suffixed) and the new column, and historical data lives in the old column. If your team renames columns in Postgres regularly, plan for a periodic destination-side cleanup pass. The connector explicitly does not handle two changes: modifying the source primary key, and modifying the precision or scale of a numeric column. Both require pausing the connector, dropping the destination table, and re-running the snapshot.

There are a few more limits to keep in your head. The connector does not replicate columns whose values exceed Snowflake’s type limits — the default cap is 16 MB per value, extendable to 128 MB with an account parameter. By default the connector’s Oversized Value Strategy is Fail Table — a value over the limit marks that table permanently failed and stops replicating it (you can switch the strategy to replace oversized values with NULL instead), so this fails loudly rather than dropping rows silently. TOASTed values that did not change in an UPDATE arrive in the journal as the placeholder string __previous_value_unchanged rather than the real value (this is a Postgres limitation, not Snowflake’s — the WAL just doesn’t carry unchanged TOAST values). And out-of-range date/time values (Postgres allows years beyond Snowflake’s supported range) get clamped to placeholder values rather than failing the row.

The Kafka Connectors and the Snowflake-to-Kafka Reverse Connector

Openflow ships two Kafka-shaped connectors that look symmetric and aren’t quite.

The forward Kafka connector reads from Kafka topics and writes into Snowflake tables via Snowpipe Streaming. This is functionally equivalent to the older Snowflake Kafka Connector but uses the new Snowpipe Streaming integration that pushes throughput up to 10 GB/s per table with five-second-to-query latency. The migration story from the older connector is straightforward — point Openflow at the same topics, mirror the topic-to-table mapping, then cut over.

The Snowflake-to-Kafka connector runs the other direction: it consumes a Snowflake stream and emits the CDC records to a Kafka topic. The interesting use case is sending data out of Snowflake to operational systems that need a low-latency feed — typically downstream microservices, reverse ETL into operational databases, or fanout to other warehouses. The stream-based pattern means the connector reads the same change history Snowflake’s streams and tasks feature would expose, but does the egress for you instead of you writing a Snowpark Python procedure.

A practical note: the reverse connector is the right tool for low-volume operational fanout (maybe events per second up to thousands per second). For sustained high-throughput egress it can get expensive, because Openflow pays for SPCS compute the whole time the consumer runs — at that scale teams typically run a dedicated consumer against a Snowflake stream rather than leaving the runtime hot. (Note the older Snowflake Connector for Kafka goes the other direction — it is a Kafka-to-Snowflake sink — so it is not an option for Snowflake-to-Kafka egress.)

Cost and Scaling

Openflow’s cost model has four moving parts: SPCS compute, infrastructure (storage and network for the data plane), data ingestion (Snowpipe / Snowpipe Streaming credits depending on which destination processor you use), and telemetry data (charged in credits per GB).

The piece that most surprises new users is that compute pools auto-scale to zero only after 600 seconds of inactivity, and the management pool never scales to zero. Credits are billed per second with a five-minute minimum, so a runtime that wakes up briefly every ten minutes for a small CDC catch-up is paying the five-minute minimum each time it wakes. For low-volume CDC against an idle source, it can actually be cheaper to leave the runtime running continuously than to let it bounce. For high-volume continuous CDC, the autoscale window is a non-issue because the runtime is always above zero anyway.

Suspending a runtime explicitly is the way to take a development environment to zero data-plane cost — the management pool still costs a CPU_X64_S instance-hour, but everything else stops. Most teams put development runtimes on a schedule and resume them with tasks tied to working hours.

When to Choose Openflow vs Snowpipe / Snowpipe Streaming / dbt

The four options solve overlapping problems and the choice matters because picking wrong adds a tier of complexity you do not get back.

Plain Snowpipe — file-based, micro-batch (one to two minute latency), the cheapest option per GB. Use it when sources land files in S3/GCS/Azure Blob on their own schedule and you do not need second-level latency. The pattern is documented in Snowflake stages, COPY INTO, and Snowpipe in practice. Most lake-style ingest paths still belong here.

Snowpipe Streaming — row-level, ten-second latency, predictable per-GB pricing. Use it when you have a producer that can write rows directly via the SDK (your application, a Kafka Connect cluster, or a custom Spark job). It is the fastest path for streaming and the most cost-predictable, but you own the producer.

Openflow — visual pipeline builder + curated connectors. Use it when you want managed CDC from Postgres/MySQL/Kafka and you do not want to operate Debezium, or when your sources are SaaS systems that fit the connector catalog. Under the hood Openflow uses Snowpipe Streaming as one of its target processors, so you are not picking a different runtime engine — you are picking a different operational story.

dbt and Snowflake streams and tasks — transformation, not ingestion. They run on data that is already in Snowflake. Even teams that pick Openflow for ingestion typically still use dbt for the transformation layer.

The non-obvious answer is that these are stacked, not exclusive. A real production pipeline might have Openflow ingesting Postgres CDC and Kafka, dbt transforming the raw tables, and Snowpipe Streaming carrying a separate application-emitted event stream — all in the same Snowflake account. The right question is “which tier owns each source,” not “which tier do we standardize on.”

Production Patterns and Pitfalls

A handful of patterns are worth applying from the first runtime you create.

One runtime per source-system family. Putting the Postgres CDC connector and the Kafka connector into the same runtime makes them share an autoscale boundary, which means one noisy source can keep the pool warm and bill the other. The cost overhead of multiple runtimes is one CPU_X64_S management charge per deployment, not per runtime — multiple runtimes share the same management pool.

Set Postgres wal_level = logical and bump max_replication_slots on the source before you start. The Postgres CDC connector consumes one replication slot per runtime, and the default max_replication_slots = 10 is the cap that bites teams running CDC alongside other consumers like Debezium or read replicas.

Monitor replication slot lag from the Postgres side. The connector creates a replication slot that pins WAL until the consumer is caught up. If the runtime is paused or fails for a few days, the WAL can grow until the source database’s disk fills. The metric to alert on is pg_replication_slots.confirmed_flush_lsn vs pg_current_wal_lsn — anything more than a day’s worth of WAL is a sign you have a stuck connector you should know about.

Build the soft-delete filter into your views, not the source tables. Resist the urge to materialize “current state” tables that filter _SNOWFLAKE_DELETED = FALSE — they break the incremental processing model. Build them as views, and let dbt or downstream consumers materialize if they need to.

A few common pitfalls to avoid:

• Defaulting users to ACCOUNTADMIN. The Openflow login UI rejects this. Set DEFAULT_ROLE = OPENFLOW_ADMIN before the user logs in for the first time, not after they hit the error.
• Running the Postgres CDC connector on a Small runtime or multi-node configuration. Both are unsupported. The runtime size enforcement happens at flow start time, not at runtime creation, which makes the failure look unrelated.
• Forgetting the management pool cost. The first month’s bill always surprises someone because the always-on CPU_X64_S is non-obvious. Run SHOW SERVICES IN COMPUTE POOL <pool> (or query SNOWFLAKE.ACCOUNT_USAGE.SERVICES) to see what is actually running before you investigate.
• Renaming source Postgres columns mid-stream. The connector treats this as drop + add, leaving the old column data orphaned. If you can’t avoid the rename, plan a manual destination cleanup.
• Mixing connector versions across runtimes. Snowflake ships connectors as versioned templates. A schema-evolution bug fix in v2.4.x is not automatically applied to flows created with v2.3.x — you upgrade the connector in-place per flow.
• Skipping REPLICA IDENTITY. Without it, updates and deletes silently do not propagate, and the destination table slowly drifts from the source. The Postgres CDC connector does not surface this in the runtime logs at any obvious level.

Wrapping Up

Openflow is the first credible managed-NiFi service that compounds with the rest of Snowflake instead of bolting onto it. The Snowflake Deployment GA in November 2025 made it possible to run the whole stack — control plane, data plane, destination warehouse — inside one account, billed in one place, secured by the same role model. For teams that were stitching Debezium + Kafka + the Snowflake Kafka Connector for Postgres CDC, the operational simplification alone usually justifies the move.

The honest caveat is that “managed NiFi” is not the same as “no NiFi.” If you need to extend a connector with a custom processor, customize a flow beyond what the connector defaults expose, or debug a stuck pipeline, you end up in the NiFi canvas reading the flow XML. The curated connectors handle the 80% case beautifully; the 20% case still benefits from someone on the team who has run NiFi before.

For inspecting and operationalizing the data Openflow lands in Snowflake — building a customer 360 view from the CDC tables, an internal ops dashboard from the Kafka stream, or a metrics tool from the journal tables — try the QueryPlane Snowflake integration. It pairs naturally with the CDC outputs because the soft-delete model wants a view layer, and QueryPlane’s app builder lets you assemble that layer plus the UI without a separate frontend project.

Frequently asked questions

What is Snowflake Openflow? Snowflake Openflow is a managed data integration service built on Apache NiFi that ingests data from databases, message queues, SaaS systems, and file sources into Snowflake. It runs as a fully-managed service either inside Snowflake on Snowpark Container Services (Snowflake Deployment) or in your own AWS account (BYOC).

When did Snowflake Openflow become generally available? The BYOC deployment type reached GA earlier in 2025; the Snowflake Deployment (SPCS-hosted) variant went GA on November 4, 2025. The Snowflake Deployment is available across AWS, Azure, and GCP commercial regions, and BYOC is available in AWS commercial regions.

Is Snowflake Openflow the same as Apache NiFi? Openflow is built on Apache NiFi and uses NiFi’s processors and flow execution engine, but adds a Snowflake-managed control plane, a curated catalog of connectors authored by Snowflake, and integration with Snowflake’s auth model (the Snowflake Managed Token). You can also build raw NiFi flows on the canvas if a connector does not cover your source.

What is the difference between Openflow BYOC and Snowflake Deployment? Snowflake Deployment runs the data plane on Snowpark Container Services inside Snowflake. BYOC runs the data plane in your own AWS account using a CloudFormation template Snowflake supplies. Choose Snowflake Deployment for simplicity; choose BYOC when sensitive data needs to be preprocessed before leaving your VPC or when on-prem connectivity has to terminate in your own AWS account.

How does the Openflow Postgres CDC connector work? The connector runs a three-stage replication: schema introspection (creating empty destination tables), an initial snapshot via COPY, then continuous CDC via a Postgres logical replication slot and the pgoutput plugin. Deleted rows are soft-deleted in the destination via a _SNOWFLAKE_DELETED column rather than removed.

Does Openflow Postgres CDC require primary keys? Each replicated table needs either a primary key with replica identity DEFAULT, or a unique index with replica identity USING INDEX. Without either, only INSERT operations replicate — UPDATE and DELETE arrive in the WAL but the connector cannot identify which destination row to modify.

What PostgreSQL versions does Openflow support? PostgreSQL 11 through 18, including AWS RDS for PostgreSQL, Amazon Aurora PostgreSQL, GCP Cloud SQL, and Azure Database for PostgreSQL.

Should I use Openflow or Snowpipe Streaming? Snowpipe Streaming when you own the producer and want the fastest, lowest-cost row-level ingest path; you write rows directly via the SDK. Openflow when you want a managed CDC or SaaS connector and do not want to operate Debezium or build custom producers. Under the hood Openflow uses Snowpipe Streaming as one of its target processors, so the throughput ceiling is the same.

Does Openflow replace Snowpipe? No. Plain Snowpipe is still the right answer for file-based, micro-batch ingest from object storage. Openflow uses Snowpipe and Snowpipe Streaming as backend ingest mechanisms depending on the target processor you choose; it is a higher-level orchestration tier, not a replacement.

How much does Openflow cost? You pay for compute pool runtime credits (per-second, five-minute minimum), Snowpark Container Services infrastructure (storage and network), the actual data ingestion via Snowpipe or Snowpipe Streaming, and telemetry data ingestion in credits per GB. There is also an always-on management compute pool that runs a single CPU_X64_S instance regardless of whether you have active runtimes — plan for that as the baseline cost.

What is the Openflow management compute pool? A control-plane compute pool that runs continuously to host the runtime orchestration and UI services. It uses one CPU_X64_S instance, scales to zero only when the entire Openflow deployment is removed, and is not the same as the per-runtime compute pools (which do scale to zero after 600 seconds of inactivity).

Can Openflow connectors be customized? Connectors ship as curated, versioned NiFi flow templates. You can clone the flow into your own runtime and modify it on the NiFi canvas if you need to extend a processor or change the flow logic, but you lose the auto-upgrade path for that specific flow. For most teams the default connector flow is sufficient.

What runtime size do I need for Postgres CDC? The Postgres CDC connector requires a runtime of Medium or larger and must run as a single node (min_nodes = 1, max_nodes = 1). The single-node constraint exists because a Postgres replication slot has a single consumer, so adding nodes would not parallelize anything.

Can I run Openflow against multiple Postgres databases? Yes, but each Postgres database needs its own runtime instance because the connector consumes a replication slot per source. Plan for that in your max_replication_slots budget on each Postgres instance, and use separate runtimes inside the same deployment rather than chaining multiple connectors into one runtime.

How do I monitor Openflow pipelines? Openflow writes telemetry to a Snowflake event table you specify when creating the deployment. The standard pattern is to point dashboards at that event table and alert on processor-failure counts, replication slot lag from the Postgres side, and runtime compute pool credit usage from SNOWFLAKE.ACCOUNT_USAGE.METERING_HISTORY.