The Databricks Photon Engine in Practice

This post was written by an engineer at QueryPlane. QueryPlane is an app builder for your database: bring your own postgres db and you can create interactive applications to share with other developers, coworkers or even your customers. If you’re interested in trying it out, get started here.

---

Photon is the native, vectorized query engine that ships with the Databricks Runtime. It first appeared as a preview in 2020, went generally available in 2022, and by 2026 it sits underneath every Databricks SQL warehouse and every Photon-enabled cluster on the classic and serverless tiers. The Spark APIs on top of it are unchanged — the same DataFrame and SQL surface you wrote against in 2019 — but the work underneath is now executed by a C++ engine that vectorizes per-batch, reads Parquet and Delta with native code paths, and has a measurably different cost model than the JVM-based execution path it replaces.

The trouble most teams have with Photon is not whether to use it but understanding what it accelerates and what it doesn’t. The DBU multiplier is real and the operator coverage is partial; a job that’s 80% Photon and 20% JVM ends up paying the Photon price for the whole run and getting Photon-level speedup on only part of it. A handful of patterns — Python UDFs, certain MAP and STRUCT operations, some streaming sinks, regex-heavy string work — silently fall back to the JVM and the wall-clock gain evaporates. Knowing which side of that line your query lives on is the difference between a 2-4× cost reduction and a wash.

This post walks through what Photon is doing underneath the Spark API, how to read the operator coverage in the Spark UI and the SQL warehouse query profile, the DBU multiplier and when it pays for itself, the 2025-era changes (Photon on serverless, broader operator coverage, ongoing native-execution improvements), and the patterns that show up when Photon is part of a real production deployment.

In this post, we’ll cover:

• What Photon is — the C++ engine, vectorization, and what changed in the JVM → native split
• When Photon is enabled — DBSQL warehouses, Photon-enabled clusters, and the runtime selection
• Operator coverage — what runs natively, what falls back, and how to read the partial-coverage signal
• The DBU multiplier — the 2× factor, when it pays off, and how to measure
• UDFs and the JVM bridge — why Python UDFs evaporate Photon’s gains and what to do
• Reading the query profile — Photon execution percentage in the Spark UI and DBSQL profile
• Production patterns — when to turn Photon on, when to leave it off, and how to size warehouses
• Pitfalls — the handful of footguns that aren’t obvious from the docs

What Photon is

The Spark execution model that shipped with Databricks Runtime up through 2020 was a JVM-based code generator: query plans were compiled into Java bytecode by Whole-Stage Code Generation, the JIT compiled hot loops, and execution was a tuple-at-a-time pipeline. It worked, and it scaled, but two architectural choices left performance on the table: data was processed one row at a time within each stage, and the JVM’s object model imposed pointer chasing and garbage collection overhead on every primitive value.

Photon replaces that path with a C++ engine designed for columnar, vectorized execution. Inside a Photon stage, data is held in column vectors of a few thousand values each, operators are written as tight loops over those vectors, and the SIMD instructions available on modern x86 and ARM cores are used directly. The original VLDB paper “Photon: A Fast Query Engine for Lakehouse Systems” walks through the design — the punchline is that for the SQL and DataFrame operations Photon covers, the speedup over the JVM path on the same hardware is typically 2× to 8× depending on the query shape.

The Spark API is the user-facing surface; Photon is the executor for that surface on a Photon-enabled cluster. You don’t write Photon code — you write Spark SQL or DataFrame code, and the Photon planner decides which operators in the physical plan run in C++ and which fall back to the JVM. The fall-through is operator-level: a single stage can be entirely Photon, entirely JVM, or a mix, and the engine handles the data conversion at the boundary.

Three things are worth internalizing about the C++ side. The Photon engine has its own Parquet reader that decodes columnar data directly into Photon’s column vectors — skipping the conversion to Spark’s InternalRow format that the JVM path requires. It has its own native expression evaluator that compiles SQL expressions to C++ at plan time. And it has a separate memory layout: a Photon stage allocates its working memory off-heap from a different allocator than the JVM uses for shuffle buffers, which is why Photon clusters generally need more memory headroom than non-Photon clusters of equivalent CPU.

When Photon is enabled

Photon is the executor on three different compute surfaces in 2026, and each turns it on differently.

Databricks SQL warehouses run on Photon by default and there’s no way to turn it off. The Pro and Serverless tiers are Photon-only — if you ran a query on a DBSQL warehouse, Photon executed whatever portion of the plan it can cover. The Classic tier is also Photon by default. This is the right tier for any BI or analytics workload: the platform makes the engine decision for you and you don’t need to think about it.

Photon-enabled classic clusters require an explicit checkbox or a runtime_engine setting in the cluster API. The checkbox is on the cluster create page under “Use Photon Acceleration”; the API equivalent is runtime_engine: "PHOTON". Photon is supported on most modern instance families (AWS m6g/c6g/r6g and m5/c5/r5 series, Azure Eav5/Dav5, GCP n2 and c3) and is not supported on smaller t3 / D-series general-purpose instances. The job-cluster API has the same flag.

Photon-enabled serverless compute is the 2025 default for new serverless jobs and notebooks: when you run a workload on serverless compute Databricks transparently picks a Photon-enabled runtime, and the pricing for serverless is already factored to assume Photon is on. There’s no checkbox — the platform decides.

Where this matters in practice: the same Spark job, written once, may run with or without Photon depending on which compute target picks it up. If the same notebook runs interactively against a non-Photon cluster and then against a job-cluster with runtime_engine: "PHOTON", the wall-clock and DBU numbers will not match. Standardizing on Photon for any cluster that runs scheduled ETL is the right default in 2026 unless you’ve already measured a specific workload where Photon doesn’t help.

Operator coverage

Photon does not run every Spark operator. The coverage has expanded with every DBR release, but at any point in time some operators are native, some fall back to the JVM, and which is which is the single most important thing to understand if you care about cost.

Broadly, the operators that are native cover almost all of the Spark SQL functional reference for analytic workloads:

• Scans of Parquet and Delta tables run through the Photon native reader; reading Iceberg/UniForm-managed tables is supported from DBR 14.3 LTS and above.
• Filters and projections over primitive types — including INT, BIGINT, DOUBLE, DECIMAL, STRING, DATE, TIMESTAMP — are fully native.
• Aggregations (SUM, COUNT, AVG, MIN, MAX, APPROX_COUNT_DISTINCT, percentile sketches) including GROUP BY over any combination of primitive keys.
• Hash joins, including broadcast and shuffled hash joins. Photon also commonly rewrites sort-merge joins into native hash joins.
• Window functions including ranking, lag/lead, and rolling aggregates over a Photon-native partition column.
• Sorts and limits, including top-K.
• Common string operations — substring, length, case, and the simple regex functions (regexp_extract, regexp_replace for literal patterns).

The operators that fall back to the JVM in 2026 fall into a few buckets:

• Python and Pandas UDFs of any flavor run on the JVM and require a row-by-row conversion at the Photon/JVM boundary. This is the single most common source of Photon underperformance.
• Scala UDFs are slightly better but still trigger a fallback for the stage containing them.
• Complex regex with non-literal patterns, lookahead/lookbehind, or non-Unicode locale handling falls back.
• Some MAP and STRUCT operations — transform_keys, transform_values, aggregate on arrays — are partially covered as of DBR 15; verify with the query profile.
• Streaming sinks other than Delta — Kafka producer, JDBC sink, custom foreach — run on the JVM. Streaming sources including Auto Loader are Photon-native; the conversion happens at the write side.
• Some file formats — JSON parsing for unstructured documents has only partial coverage; CSV with quoted multi-line records falls back; Avro is partial.

The query profile is the source of truth for which side of the boundary your query lives on. When you click into a query in the SQL warehouse profile or the Spark UI for a job cluster, each operator in the plan is labeled Photon or shows a “Photon ↦ Spark” boundary marker where the engine handed off. Aiming for >90% of stage time inside Photon nodes is the right target for an analytics workload; anything less and the DBU multiplier starts working against you.

The DBU multiplier

This is the part most teams get wrong. Photon-enabled clusters consume DBUs at a higher rate — often roughly 2× on classic clusters, though the exact factor varies by instance type, so check the per-SKU pricing. The higher rate exists because Photon-enabled clusters use the same VMs but bill more per hour for the engine.

The payoff math is straightforward. If Photon makes a query run 3× faster end-to-end, paying 2× the DBU rate gets you the result in one-third the wall time at two-thirds the cost — a 33% net savings. If Photon makes the query 1.5× faster, you pay 2× the DBU rate for 67% of the wall time, which is 1.33× the cost — a loss. The break-even is at a 2× speedup, and the practical guidance is: only run Photon on workloads where you’ve measured at least a 2.5× speedup, or where the workload is interactive and wall-clock latency matters more than cost.

Three workload shapes consistently hit the 2.5× threshold and above:

1. Scan-heavy analytics over Parquet or Delta with selective filters — TPC-DS-style queries, BI dashboards, ad-hoc warehouse analytics. Photon’s native Parquet reader and vectorized filter pushdown routinely hit 4-8× speedups.
2. Aggregations over large fact tables — daily rollup jobs, sessionization with window functions, materialized view incremental refresh. Vectorized aggregation is one of Photon’s strongest paths.
3. Joins between large tables when one side is broadcast-eligible — Photon’s native hash join build is markedly faster than the JVM equivalent.

Workloads that consistently fall below the break-even:

1. Heavily UDF-driven pipelines — anywhere a stage is dominated by Python UDF time, the JVM overhead dominates and Photon does nothing for that part of the work.
2. Tiny jobs — anything that runs in under 30 seconds tail-to-tail has too much fixed overhead for Photon’s gains to be visible.
3. CSV-heavy ingest — quoted, multi-line, badly-encoded CSV trips Photon’s partial coverage and the speedup disappears.
4. Wide-shuffle workloads — when the shuffle phase dominates wall time (and not the compute), Photon’s per-stage gains are buried under network and disk I/O.

The right pattern in a Databricks account is to default to Photon on DBSQL (where the decision is made for you) and to A/B classic clusters with and without Photon on the specific workload, comparing total job DBUs not just wall time. We’ve seen teams flip Photon off on a single ingest job that was 70% Python UDF time and save a meaningful fraction of monthly DBUs.

UDFs and the JVM bridge

Python UDFs are the single most common reason a Photon migration disappoints. The mechanism is simple: when a query plan contains a PythonUDF node, that node executes in a Python worker over an Arrow-encoded batch shuttled out of the JVM. Photon cannot run Python; the data leaves the C++ engine, is converted to a JVM internal row format, then to Arrow, then to Python; the function runs; and the result walks back. Every UDF call carries a fixed cost per batch plus the per-row Python work.

Three patterns avoid the trap:

Replace UDFs with SQL or DataFrame primitives whenever possible. Most Python UDFs we see in production are 5-line functions that could be a CASE expression, a regexp_extract, a map_filter, or a built-in higher-order function. The Spark SQL function library has grown substantially in the last three releases; a substring + concat that took a UDF in Spark 3.0 is now a one-liner in DBR 15.

Use pandas_udf for unavoidable UDFs. A pandas_udf operates on Arrow batches instead of single rows, which amortizes the conversion overhead. It’s still on the JVM-Photon boundary and still falls back, but the wall-clock damage is 5-20× smaller than a row-by-row Python UDF.

Use the native built-in SQL functions for JSON, regex, and string work. from_json, get_json_object, and regexp_extract with literal patterns are Photon-native; the temptation to wrap a Python json.loads call into a UDF for “more flexibility” is a Photon killer.

The query profile shows the cost cleanly: any operator with a non-trivial wall-clock time and a “Spark” label adjacent to Photon operators in the plan is a candidate for replacement.

Reading the query profile

The Spark UI for a job cluster and the SQL warehouse query profile for a DBSQL warehouse are the two surfaces where Photon’s contribution is visible.

In the Spark UI, the SQL / DataFrame tab shows the physical plan as a tree of operators. Photon-native operators are highlighted in orange; the JVM operators they hand off to stay the default color. Hovering a node shows its row counts and timing, so you can see how much work was done natively versus on the JVM. The most useful single number is the share of stage time spent in Photon operators — anything above 90% means Photon is doing real work; anything below 60% means the JVM bridge is dominating.

In the DBSQL query profile, the Execution Details panel reports the percentage of task time spent in Photon, and Photon operators are rendered in purple. The profile also surfaces per-operator timing, so you can see where the JVM is being entered. A common pattern: a query that’s 95% Photon at the scan and aggregation level but drops to 40% during a window function because the partition key is a STRING cast from a STRUCT field, which forced a fallback. Rewriting the upstream cast to keep the partition key in a Photon-native shape often recovers the bulk of the gap.

For job-cluster workloads, inspect the physical plan directly with EXPLAIN FORMATTED <query> (or df.explain("formatted") on a DataFrame) and look for the Photon operator nodes versus the JVM nodes they hand off to. There’s no dedicated “Photon explain” mode — the formatted plan plus the query profile’s Photon timing are how you audit a query. Run this on the longest-running jobs in your account as a regular hygiene check.

Production patterns

For a Databricks account running real production workloads, the patterns we’ve seen converge on a small set of defaults.

Default to Photon on every cluster that runs scheduled SQL. ETL jobs that read Parquet/Delta, aggregate, and write back are the textbook Photon win. Set runtime_engine: "PHOTON" in the job cluster spec and audit DBU usage after a month.

Use DBSQL warehouses for BI and ad-hoc analytics. They’re Photon-only, the scale-up is automatic, and the cost model is straightforward. Don’t run BI on Photon-enabled job clusters when DBSQL is the right product — the DBU per query is similar but the warehouse handles concurrency better.

Audit UDF-heavy pipelines individually. Anywhere a stage’s wall time is dominated by Python UDFs, run the same job once with Photon and once without and compare total job DBUs. If non-Photon wins, leave it off until you can either replace the UDF with native SQL or rewrite it as a pandas_udf.

Right-size cluster memory. Photon-enabled clusters need more memory headroom than the JVM equivalent because of the off-heap vector buffers. The rule of thumb is to either pick a memory-optimized instance family (r5, r6g, Eav5) or to over-provision the standard family by 20-30% relative to a non-Photon baseline.

Track native-execution coverage on serverless. Photon remains Databricks’ proprietary engine, but the broader Spark ecosystem now has open-source native engines (Apache Gluten and DataFusion Comet); on Databricks serverless compute, newer runtimes pick up native execution for some operators that historically fell back. Verify the coverage against your specific job in the query profile — the operator coverage matrix changes per release.

Common pitfalls

A few recurring failure modes that aren’t always obvious from the docs.

Confusing Photon with adaptive query execution. They’re independent features. AQE reshapes the plan at runtime; Photon executes whatever plan AQE produces. A query can be Photon-native end to end and still be slow because AQE picked a sort-merge join when broadcast would have been better. The two should be diagnosed independently — check the query profile for both.

Assuming Photon helps streaming workloads uniformly. Streaming sources (Auto Loader, Kafka, Delta CDF) are Photon-native; streaming sinks other than Delta typically aren’t. A pipeline that reads Kafka, transforms, and writes Kafka will see Photon-level speedup on the transform but the write side will run on the JVM. Match Photon enablement to where the wall time is spent.

Using Photon on tiny ad-hoc clusters. A 1-2 worker cluster running a 30-second query rarely benefits — the planner overhead and DBU multiplier eat the gain. Reserve Photon for clusters that are doing real work and where wall-clock and cost both matter.

Inferring schema on every run. Photon’s Parquet/Delta scan is fast because the schema is known upfront and the column vectors are pre-sized. A schema-inference path (raw JSON, CSV without inferSchema=false) bypasses the optimization and converts most of the read on the JVM side. Pin schemas explicitly for any production ingest.

Not measuring before and after. Photon’s DBU multiplier means every cluster decision should be backed by a measurement. The system tables billing schema (system.billing.usage) is the right source of truth — aggregate DBU per job for two-week windows before and after enabling Photon and verify the total cost actually dropped.

Ignoring the serverless cost model. Serverless compute pricing is a flat per-DBU rate that already factors in Photon; comparing classic-cluster DBUs against serverless DBUs is not apples-to-apples because the serverless rate includes the platform’s resource management.

Using Photon on a workload dominated by I/O wait. A pipeline whose stages spend most of their time waiting on S3 reads is not bottlenecked on compute. Photon helps the compute portion of the wall clock; if compute is 10% of the stage time, the maximum possible savings is 10%, and the DBU multiplier will negate it.

Wrapping up

Photon is the right default for scan-and-aggregate analytics on Databricks — the C++ engine, vectorized execution, and native Parquet path are real and the speedup on the workloads it covers regularly clears 2.5× on the same hardware. The trick to using it well is knowing where the JVM bridge is in your specific plan and either eliminating it or measuring whether the partial coverage still pays for the DBU multiplier.

A few things to take away. Default to Photon on DBSQL and on scheduled ETL clusters; leave it off only when measurement shows it’s costing you. Replace Python UDFs with native SQL or DataFrame primitives wherever you can; the bridge to Python is the single largest source of unrealized Photon gains. Use the query profile and the percentage-of-task-time-in-Photon metric to audit production pipelines on a regular cadence; the operator coverage matrix expands with every DBR release and yesterday’s “this falls back” might be today’s “this is native”. And measure DBU cost, not wall-clock, when deciding whether to flip Photon on or off — the multiplier is real and the wall-clock improvement is not always enough to justify it.

The companion posts in this series — Auto Loader in practice and Lakeflow Declarative Pipelines in practice — cover the ingestion and orchestration sides of the same stack. Auto Loader is the source side that feeds Photon-native streaming tables; Lakeflow is the framework that runs both on Photon-enabled compute by default.

If you’re running this stack and want a SQL editor and an app builder that connects directly to a Databricks SQL warehouse so you can inspect bronze tables, write ad-hoc queries against the medallion gold layer, and build internal tools on top of your Lakehouse, QueryPlane is built for that workflow. We connect to the same warehouse Photon runs underneath, give you a fast SQL editor and Python-free chart and table builder, and let you share the result with other engineers without standing up a separate BI seat.

Frequently asked questions

What is the Photon engine on Databricks? Photon is the native, vectorized query engine that ships with the Databricks Runtime. It’s written in C++, executes the same Spark SQL and DataFrame plans the JVM engine would, and routinely runs 2-8× faster on scan-and-aggregate analytics over Parquet and Delta. The Spark API doesn’t change; only the executor underneath does.

When should I turn Photon on? Default to Photon on DBSQL warehouses (where the choice is made for you) and on scheduled SQL ETL clusters reading Parquet or Delta. Audit individually any job that’s heavy on Python UDFs or that reads CSV/JSON with complex parsing — these workloads commonly fail to clear the 2× DBU multiplier.

Does Photon cost more than non-Photon? Photon-enabled classic clusters charge roughly 2× the DBU rate of equivalent non-Photon clusters, though the exact factor varies by instance type. The net cost is lower whenever the wall-clock speedup is above 2× and higher when it’s below; serverless compute already factors Photon into the flat DBU rate. Always measure with system.billing.usage before and after, not just wall-clock time.

Why are my Python UDFs not faster with Photon? Python UDFs run on a Python worker outside the Photon engine and require an Arrow-encoded round trip out of the C++ engine, through the JVM, into Python, and back. Photon doesn’t accelerate the UDF execution itself — it only accelerates the operators around it. Replace UDFs with built-in SQL functions or pandas_udf (which batches the conversion) wherever possible.

How do I check how much of my query ran on Photon? In the DBSQL query profile, the Execution Details panel reports the percentage of task time spent in Photon, and Photon operators are shown in purple. In the Spark UI SQL / DataFrame tab, Photon operators are highlighted in orange. Aim for >90% of task time in Photon on analytics workloads.

Is Photon the same as the open-source native Spark engines? No. Photon is Databricks’ proprietary C++ engine. The open-source ecosystem has its own native/columnar Spark engines — Apache Gluten and DataFusion Comet — which share concepts with Photon but are separate projects. Photon stays proprietary to Databricks and continues to ship coverage and performance improvements ahead of the open-source equivalents.

Does Photon work on Iceberg tables? Reading Iceberg/UniForm-managed tables under Unity Catalog is supported from DBR 14.3 LTS and above, with filter pushdown and column pruning. Writes go through Spark’s Iceberg writer. Check the query profile to confirm how much of a given Iceberg read runs in Photon on your runtime.

Does Photon support GPU acceleration? Not as of DBR 17. The Photon engine is CPU + SIMD only on x86 and ARM; GPU-accelerated Spark workloads use a different stack (typically the RAPIDS Accelerator for Apache Spark, which is a separate plugin). Photon and the RAPIDS plugin are mutually exclusive on the same cluster.

How does Photon interact with Adaptive Query Execution? They’re independent. AQE reshapes the physical plan at runtime based on stage statistics — picking broadcast vs sort-merge joins, coalescing shuffle partitions — and Photon executes whatever plan AQE produces. A query can be Photon-native end to end and still slow because AQE picked a suboptimal join, and vice versa. Diagnose them independently in the query profile.

Can I tell ahead of time whether Photon will help my query? Inspect the plan with EXPLAIN FORMATTED <query> (or df.explain("formatted")) and look at which nodes are Photon nodes versus JVM nodes — there’s no dedicated Photon explain mode. For an existing job, audit the SQL warehouse query profile or the Spark UI for the historical run; the percentage of task time spent in Photon is the single number that predicts whether the cluster’s DBU multiplier will pay off.