Just recently I had to have a big datawarehouse solution to test some performance optimization using BIML. I could use the AdventureWorks2012 database, but I needed the clean datawarehouse tables in order to have minimum data maintennance when testing the BIML scripts.
I could not find it, and figures out it was faster to make my own.
So heavily inspired by this post from Jonathan Kehayias (blog), I’ve made a script that can be used to enlarge the dbo.FactInternetSales table.
The script creates a new table called dbo.FactInternetSalesEnlarged and copies data from dbo.FactInternetSales into it with a randomizer. Exploding the data to a 100 times bigger table – est. 6 mio rows.
If you want to remove the time element in a datetime object, you should cast it to DT_DBDATE. But because that datatype is very inconvenient to use, you should cast it back to the original datatype. That will set the time to 0:00.
Time from datetime
If you want to remove the date element in a datetime object, you should cast it to DT_DBTIME. And optional cast it to a string.
With all the BIML code done and the production scenarios start to emerge, it would be great to have the BIML code automatic expanded and create new packages if there are changes to the underlying metadata. Either configured manually or directly from the sources. Also known as BIML expansion.
With the MIST application from Varigence – this is possible through the command line util that ships with the installation.
When the installation of MIST has finished, you’ll find a new .exe-file in the installation folder called bimlc.exe.
This file is the core engine of the command line util.
According to the online documentation found here – the command line util is pretty straight forward. The observant reader will notice that the link points to a documentation for hadron.exe – this is the old name for the same tool. See this documentation from Varigence.
The syntax is
bimlc.exe -s="<sourcefile>" -t="<targetfolder>"
I’ve made a quick demo biml-project with below code:
This file is called AutoCompile.biml, and I want to put the generated files in c:\AutoCompilePackages. Then the commandline would be:
bimlc.exe -s="c:\AutoCompile\AutoCompile.biml" -t="c:\AutoCompilePackages"
When I then press ENTER the compiler starts up and does it’s thing:
The output tells us that there are 9 packages created in each of its own projects.
I can find the compiled projects and packages in my defined output folder.
And the content of each folder.
Thats it. Easy job 🙂 thanks to Varigence and their MIST product.
In these times of data warehouse automation and fast track SSIS, BIML has become a huge contributor to my daily work. On one of my most recent projects we had a custom script task in SSIS that we needed to implement in the same package several times.
The code of the custom script is in its own biml-file in the project and referenced from the main biml-file.
We kept getting a mysterious error when executing the package in debug mode from Visual Studio. (No warning upon building the packages).
After alot of debugging and hair pulling hours, we finally got our arms around the bug.
The error was that the ProjectCoreName of the new 3 script tasks in the same package had the same name.
My learning from this, and yes, I’m not a .NET developer, is that upon executing the custom script task, the SSIS engine must compile the code and store it for usage upon handling the data. This storage is shared with the whole package, and therefore cannot contain the same .NET project name. [Written from my personal understading – correct me if I’m wrong].
So for any future BIML users who wants to add the same custom script task to the BIML project – remember to add a variable to the ProjectCoreName of your biml-script.
This can be done fairly easy, as your biml-scripts can relate to the variables calling the biml-file. Just add the standard
to he ProjectCoreName-tag of your biml-file containing the custom script.