Would you like to take advantage of the latest Apache Spark version for your Dataverse Synapse Link? Well, you’re in luck, because I’m going to explain how you can update your Synapse Link profile to use Spark 3.3 with Delta Lake 2.2, without losing your data or current configuration.

Apache Spark 3.3 with Delta Lake 2.2

Prerequisites

  1. Synapse Link Profile: You need to have a Synapse Link profile for Dataverse delta lake already running with a Synapse Spark 3.1 version.
  2. New Spark Pool: You must create a new Synapse Spark 3.3 pool. This pool should have the same or superior node hardware configuration within the same Synapse workspace. And note, this Spark pool must be created independently from the current 3.1 pool.

Procedure

It’s very easy and quick, you just need to follow these steps:

  1. Sign in to Power Apps and select the environment where you already have Synapse Link installed.
  2. In the left navigation panel, select Azure Synapse Link. If you don’t see it, select …More and then choose the option you want.
  3. Open the Synapse Link profile and select upgrade to Apache Spark 3.3 with Delta Lake 2.2
  4. Select the available Spark pool from the dropdown menu and click Upgrade.

And that’s it! This is how simple it is to update your Synapse Link to the latest version of Spark, which offers you better performance, greater stability, and more features.

Keep in mind, however, that the upgrade only occurs when a new Spark job is triggered to convert your Delta Lake. So make sure you have at least one data change after clicking the upgrade button.

I hope this article has been useful to you and that you enjoy your improved Synapse Link. If you have any questions or comments, feel free to leave them below. See you next time!