Performance Engineering on Microsoft Fabric

Introduction

The experiment tested whether Microsoft Fabric could run an enterprise-scale data engineering estate: approximately 3,000 objects, more than 4 billion rows of initial data movement and a single dependency graph spanning Spark and Warehouses.

The initial load completed in 54 minutes 44 seconds on an F64 trial capacity. Repeated incremental runs completed in approximately 13 to 18 minutes. Reaching those times depended on controlled concurrency, engine-specific load design and making unchanged data cheap to process.

1. Experimental design

1.1 Engineering requirements

The experiment is motivated to include the backend workload necessary for serious data engineering, including staging, hashing, change detection, quality thresholds, audit columns, CRUD statistics, bookmarks and execution logging.

It also tests dependency-aware recovery: a failed branch blocks its dependants without stopping independent work. Five execution paths are exercised:

The benchmark measured one successful representative workload. It did not attempt exhaustive failure injection.

1.2 Dataset design

The source distribution follows the long-tailed shape documented in a study of production workloads across the Amazon Redshift fleet (van Renen et al., 2024): a few large tables, many small ones and many sources with little daily change.

The generator produced approximately 1,000 source objects:

The sources simulate reproducible daily changes, including inserts, updates and deletions. Their change frequencies vary according to size, with approximately one source in seven changing on a typical simulated day.

A separate estate of approximately 3,000 objects consumes these sources through several transformation layers distributed across one Lakehouse and two Warehouses.

The estate contains incremental merges, physical deletes, full-source upserts, append-only loads, static tables and views. More than 2,000 tables use incremental behaviour.

The extra dependencies model scheduling pressure rather than expensive analytical joins. Source generation and daily source changes were kept outside the measured estate-load interval.

1.3 Scope and exclusions

The experiment isolates data movement, incremental reconciliation, validation, operational state and orchestration. It does not cover complex analytical joins, interactive query performance, Power BI rendering, theoretical maximum throughput or comparisons between Fabric SKUs.

2. Performance engineering strategy

Three changes mattered: schedule independent work together, reduce the cost of each load and make unchanged objects finish quickly.

2.1 Concurrency and asynchronous execution

Several thousand objects cannot run sequentially, but unlimited concurrency creates contention. The scheduler therefore used separate resource limits for Spark and each Warehouse while releasing nodes as soon as their dependencies cleared.

Spark concurrency

A separate Spark application per table would spend too much time on startup and executor allocation. The benchmark instead ran up to 24 concurrent Python or Spark SQL operations inside one application.

Shared execution required isolated temporary objects and safe runtime state. FAIR scheduling alone did not solve the problem; controlled concurrency did.

Warehouse concurrency

Warehouse procedures ran through independent SQL connections, with up to 12 concurrent procedures per Warehouse. These lanes did not consume Spark slots.

Separate limits allowed Lakehouse and Warehouse branches to progress together without treating every operation as equivalent.

SQL analytics endpoint synchronisation

A completed Delta write is not always immediately visible through the SQL analytics endpoint. Waiting synchronously after every write would stall unrelated work.

The DAG therefore models endpoint refresh as its own operation. Only consumers of the refreshed state wait; independent branches continue.

Critical-path scheduling

The scheduler optimises elapsed time across the full dependency graph. A branch waiting for Spark, Warehouse or endpoint readiness does not stop another branch whose prerequisites have cleared.

2.2 Optimising individual load operations

Concurrency cannot compensate for waste inside every node. A reliable incremental load still has to stage data, calculate signatures, detect inserts, updates and deletes, apply quality policy and preserve audit fields.

The implementation minimises repeated scans, round trips and materialisation. Spark may benefit from retaining an intermediate relation; Warehouse T-SQL may favour CTAS, a calculated change set and a targeted MERGE.

CRUD counts should come from the same reconciliation work where possible. Separate queries for every count repeat expensive distributed work.

2.3 The fast no-op

Unchanged objects are common in large estates. The Redshift fleet study found that most observed tables had no inserts or deletes during its observation period.

Fabric still pays fixed costs for an empty operation: Spark submission, Delta metadata access, distributed SQL startup and transactional bookkeeping. Small costs become material when repeated thousands of times.

The benchmark reduced this path to three things: detect an empty change set with minimal engine work, move non-critical bookkeeping off the critical path, and advance an explicit bookmark after a clean load.

Bookmark design

Bookmarks are persistent cursors that record how far each load has processed its sources. They are significant because they let the next run start from a known point instead of rediscovering its position by scanning target data or orchestration history.

Using the maximum timestamp in a target table as that cursor is fragile. It cannot describe several upstream sources, filtered changes may never reach the target, and calculating the maximum can scan a large table.

The benchmark stores one bookmark per loadable object. It records the instant immediately before the last clean load began and advances only after success. Changes that arrive during a load remain eligible next time.

Late or backdated source records still need source-specific handling, such as an overlap window or change-data-capture sequence. Some generated transformations use a target Epoch maximum, so the benchmark contains both patterns.

3. Performance measurements

End-to-end wall time is the headline metric. Row counts, execution outcomes and node-duration distributions provide the correctness and tail checks.

The initial load read and inserted 4,045,265,040 rows across 2,185 executable nodes. The unchanged run executed the same graph, read 2,293,000 rows for change detection, and produced no inserts, updates or deletes. The three daily incrementals changed a median of 7,719,640 rows in 17 minutes 29 seconds.

Across repeated runs of the deterministic workload, incremental loads completed in approximately 13 to 18 minutes. The three runs in this campaign were tightly grouped between 17 minutes 14 seconds and 17 minutes 35 seconds. Every load retained the same node shape and exact row totals, and all four nine-test convergence checks passed.

Incremental timing varied with Spark executor allocation and whether another session still held resources. Processing efficiency and capacity availability are separate effects.

3.1 Resource utilisation

Wall time alone does not explain where the run spent its resources, so the campaign retained four evidence layers.

These layers are not interchangeable: capacity-unit seconds are not CPU time, and Spark efficiency does not describe Warehouse execution.

3.2 Telemetry findings

The campaign retained seven Spark applications, 325,924 Query Insights records across five Warehouses, 396 capacity samples and 64,100 capacity-operation events. No Spark application exceeded capacity; Spark core efficiency ranged from 89.5% to 95.3%; every Warehouse request record succeeded.

The capacity-wide 30-second overview samples averaged 18.3% utilisation and peaked at 26.9%. They recorded no overage, while the capacity-operation records reported no throttling delay. The incremental duration was therefore not explained by F64 saturation.

Spark was efficient while active. The remaining time was more consistent with session overhead, dependency waiting, data movement and coordination than with inefficient Spark computation. This is also visible in the operation mix: 42,405 of the 43,657 capacity-operation events associated with the seven campaign items were OneLake operations.

Warehouse request duration showed a long tail rather than a broad failure pattern.

The Core Warehouse contained the longest individual request, but the stable end-to-end load times and node-duration distributions show that this was a tail event rather than a general slowdown.

Other items were active on the capacity during the measured window. The campaign items accounted for 68.1% of capacity-operation events, yet the capacity remained well below saturation and recorded no throttling. The result therefore did not depend on obtaining a completely isolated capacity.

3.3 Interpretation

The tested workload completed its initial load in under 55 minutes and incremental loads in under 18 minutes on F64.

This is evidence for this class of engineering workload, not a general Fabric benchmark. The transformations were deliberately simple, the estate was synthetic and the untuned estimate was not a controlled before-and-after measurement.

4. Implementing these techniques in your own environment

The same patterns apply outside the generated workload:

1. Schedule from dependencies.Keep the graph as queryable metadata. Release work when prerequisites clear, with separate resource limits for Spark and each Warehouse.
2. Treat incremental state as infrastructure.Give every loadable object reliable change detection, an explicit bookmark, validation, reconciliation and an operational record.
3. Implement the contract for each engine.Keep one load contract, but use the physical strategy that suits Python, Spark SQL or Warehouse T-SQL.

These capabilities are already included in Weaver, with concurrency parameters that can be chosen for each site. Everything else, including asynchronous execution, endpoint barriers, incremental bookmarks, reconciliation and operational recording, is handled automatically.

Reproducing the experiment

The public stress-test generator creates the source estate, measured estate and deterministic daily changes. The runner records each build, load and test in machine-readable form.

The same workflow can run at a smaller scale, such as 300 objects and 1% of the data volume.

5. Conclusion

Microsoft Fabric completed the 3,000-object initial load in 54 minutes 44 seconds and the three measured incremental loads in 17 minutes 14 seconds to 17 minutes 35 seconds.

The result came from the loading architecture: separate execution lanes, explicit endpoint barriers, engine-specific reconciliation and a cheap no-change path. At this scale, orchestration and operational overhead matter as much as row volume.

The implementation: Weaver

Weaver is the open-source implementation used by the experiment. Metadata declarations sit alongside ordinary Python and SQL transformations to manage dependencies, incremental loading, auditing, rejection policy, bookmarks and concurrent execution.

The benchmark generator, methodology and runner are available in the public stress-test source code.