I’ve stumbled upon a brand new feature in Azure Data Factory. The new timeout setting under advanced when you implement a custom script task in your pipeline. The UI As with all other things from Data Factory, the UI is pretty straight forward. Under the advanced tab in your script task, you now the option to set the timeout for the execution of this specific script. The information modal when you hover the (i) is:
I’ve stumbled upon an error a few weeks ago when working with a client and trying to read some Parquet files from a new Blob container from the build-in Storage account in an existing Synapse Workspace. If you’ve only worked with CSV untill now, you should also try out the Parquet files - they know how to perform fast when done right - but that’s possibily another blogpost. The database has been setup months ago to read data from “old” Blob containers and things was working out pretty nice.
Do you want to learn Databricks and do you want to read only one book about it? Then you should read the book by Vihag Gupta (Li/Tw) - Business Intelligence with Databricks SQL. The writer Vihag Gupta is Solutions Architect in Data & AI at the Databricks company. He shows a great knowledge in the book around Databricks and the way of working with SQL in the services. With a good match of text, downloadable demos and easy to understand illustrations, Vihag Gupta makes it an easy read.
Yes, you read it correctly. You can travel in time on your data using Azure Data Bricks and DeltaLake. The feature is quite easy to use and comes with a lot of very nice features and business value. Even a value so powerfull that the business does not need to make descissions up front when you are building your data warehouse. A little word about the background Azure has abopted DataBricks into the Azure Synapse portfolio with seemless integration between all the services from within the umbrella of Azure Synapse.