Would you like to take advantage of the latest version of Apache Spark for your Dataverse Synapse Link? Well, you're in luck, because I'm going to explain how you can update your Synapse Link profile to use Spark 3.3 with Delta Lake 2.2, without losing your data or your current configuration.
Prerequisites
Synapse Link Profile: You need to have a Synapse Link profile for Dataverse delta lake already running with a version of Synapse Spark 3.1.
New Spark Pool: You must create a new Synapse Spark pool with version 3.3. This pool must have the same or higher node hardware configuration within the same Synapse workspace. And be careful, this Spark pool must be created independently of the current 3.1 pool.
Procedure
It's very easy and fast, you just have to follow these steps:
Sign in to Power Apps and select the environment in which you already have Synapse Link installed.
In the left navigation pane, select Azure Synapse Link. If you don't see it, select …More and then choose the option you want.
Open the Synapse Link profile and select upgrade to Apache Spark 3.3 with Delta Lake 2.2
Select the available Spark pool from the drop-down menu and click Update.
And that's it! It's that simple to update your Synapse Link to the latest version of Spark, which offers better performance, greater stability, and more features.
Of course, keep in mind that the update only occurs when a new Spark job is triggered to convert your Delta Lake. So make sure you have at least one data change after clicking the refresh button.
I hope you found this article useful and that you enjoy your improved Synapse Link. If you have any questions or comments, don't hesitate to leave me below. Until next time!