I just finished creating a new version of the Enhanced Data Generators for use with Visual Studio Team System Database Edition (aka Data Dude), GDR R2.
This new version has the added ability to generate Chinese first and last names and also valid IBAN numbers. Also quite a bit of performance tweaks and bug fixes.
Instead of just writing blog posts about this, I have setup a mini website, with a couple of html pages that I can update and use for providing information about these components.
You can find the new website at
You can also find my older posts about these components at
Please use the contact form to provide feedback about this component.
As some of you may have noticed I have been writing a number of custom Data Generators for the VSTS Database Edition over the last couple of months.
During this time I have learnt quite a bit about the way these work and about writing these components. There is scarce documentation and I couldn't find too much information around writing these in the blogosphere too. Any information out there seems to be put out when VSTS Database edition was released and obviously a lot has changed since then with the release of multiple updates (with the latest being GDR R2).
One of the things that makes a lot of sense to me is to try and reuse some the Generators that Microsoft has written to create more complex Generators. One of the most interesting Generators is the RegEx Generator. I did quite a bit of searching around and am yet to find any code/freeware that does the equivalent and from a data generation perspective it is absolutely important to have the ability to generate random data based on a pattern.
The class that does this work is called RegExString. (I look at the Microsoft.Data.Schema.Generators.Extensions.xml found in the VSTSDB\Extensions directory will give you this information). If you search around for information on this class you will find a lone MSDN entry and most probably one of my forum posts.
If you just add the dll specified in the Microsoft.Data.Schema.Generators.Extensions.xml to your reference you will find no RegExString Class that you can use. A little bit of sleuthing will show you that the namespaces between the Dll and the MSDN entry are wrong. Well this is because, the namespaces have changed with GDR R2(I am not sure if it changed with R2 or GDR R1) and now all Generators are internal (or friend) and not public. So only Visual Studio can create instances of these classes.
Well now what about the MSDN documentation, which does not talk about the class being internal. Well that is for the older DLL, Microsoft.VisualStudio.TeamSystem.Data.Generators.dll. You should still be able to find this dll in the DBPRo\Extensions directory inside Microsoft Visual Studio 9.0. This is the older version where the Generators are still public.
So I decided to use those in one of the newer Generators I am writing (hope to get a new release out sometime later this week). I know this is a risk as most probably this is going to disappear when moving to VS 2010, but as of now, this is the easiest way for me to generate some pattern based data.
This Dll also requires a reference to the Microsoft.VisualStudio.TeamSystem.Data.dll, which can be found in the DBPro directory.
Once this is referenced you should be able to create an instance of the class. So I created an instance and took the class for a spin,
regx.Expression = exp regx.MaximumLength = length regx.GenerateNextValues() value=regx.Result
Well this looks like this should work. If you try this you will get an "Object reference not set to an instance of an object" error. Searches and forum posts did not turn up any solution.
I had almost given up on this when I actually found the solution when playing around with the code searching for something else. The secret is to Initialize the Generator:
So once the Generator is initialized, everything should work fine. This actually is true for some of the other inbuilt generators I tried too…
I really hope Microsoft makes the inbuilt Generators public again so that it is easier for people to develop custom generators and not reinvent the wheel every time.
One of the most problematic part of any new development is to make sure that you test your application with data that is near real life. 90% of the cases developers just test with a couple of rows of data that they enter and the data is rarely diverse enough or similar to real life data.
There are a number of data generator tools out there in the market, but they are costly and development teams rarely get the budget to buy these tools. There are a number of random data generation free-wares, but most of these are developed over a weekend hacks and though they generate random data, I don’t think they are representative of most real life data (Who actually has a name like “$ssoghetn2u#a”).
Few people know that Visual Studio Database Edition has a set of tools to generate test data. Called Data Generators, you can add a data generation plan to your database project and use that to generate test data. It ships with multiple data generators and you can also write custom data generators very easily. Since the Database Edition is now available with the developer and some of the other editions, it is easy to use by most developers.
I actually started writing a data generation application that can generate real life representative data a couple of years back. As is the fate of a number of those weekend projects, this went into hibernation once I ran into time issues and then was forgotten.
Then a couple of months back I started playing around with the Visual Studio Team System (VSTS) Database Edition and then remembered I had some code that I had written for the original data generator and so started looking into writing custom data generators for VSTS DB. There is not too much data on writing custom generators and most of the blogs are out of date. So if you want to write custom generators, I would point you to MSDN.(That information is quite good and you should be able to get up and running quickly)
I was surprised at how the VSTS DB custom generator interface was so similar to what I had come up with and so started writing a couple of data generators based on the code I already had. After a couple of weekends, I had my first couple of custom data generators.
Note: These are written for VS 2008 SP1 and Database Edition GDR R2. I am not sure if this will work for GDR 1. (I suspect it will as the interface for custom data generators have not changed).
This EnhancedDataGenerator currently contains two DataGenerators:
1. FirstNameGenerator – Generates real-life First Names
2. LastNameGenerator – Generates real-life Last Names
These are also locale aware and currently support two locales en-US and de-DE. So based on the locale this generates different kinds of First and Last Names. (Locale is a property in each generator, so you can generate English first names and German last names, if that is what you want). If you have data for common First and Last names in other cultures, please send them to me and I will update the database to handle those cultures too. (The maximum time I actually spent was to search for and get the base data into a format that I can import into the database)
The FirstName Generator is also Gender aware.
A couple of screenshots of these at work:
EnhancedDataGenerator Setup – Setup File that will deploy the files to the default VS install location (Don't use if you had used a custom path to install Visual Studio)
Other Things to Notes
1. Error Handling is not great. I have not yet found a way to report errors to the IDE user. So as of now if there are any errors, nothing happens. E.g. if you set the locale to anything other than en-US or de-DE, you just get empty data back. So if no data appears, that means you have an error .
2. The setup kit has not be fully tested. So if this does not work for you, use the DLL kit and do a manual deploy
3. Retain the directory structure inside the zip file when doing manual deploy
3. If your Visual Studio is installed to a custom path (not <prog Files x86>\Microsoft Visual Studio 9.0, then you will have do a couple of more things.
a. Use the manual depoy method and get the Extensions deployed in the VSTSDB\Extensions
b. Change the config file setting DBPath to point to the Database directory inside your custom extensions directory.
4. The .Extensions.xml file is already created with the correct data, so you will not have to create those again when doing a manual deploy.
A few months back I wrote and application called the PDC downloader and Viewer and released it at CodePlex.
Over the last couple of weekends, I kind of updated the application, so that it can also be used to get the Mix 2009 videos. So now it is a more of a generic Event video downloader and viewer. If MS continues to use similar naming conventions, this application can be updated quite easily to download and view videos from other future events too.
The new version 220.127.116.11 of the updated Event Video downloader and viewer is now available for download from Codeplex.
It has been quite sometime since I actually coded TSQL. So I had to kind of brush up the hidden knowledge over the weekend when I wrote a small stored proc for a personal project I am working on.
(expect to see something in Codeplex, if I can get it finished before the new year)
I declared a variable and wrote out some code as follow:
declare @tempstring as nvarchar(max)
<More code here>
select @[email protected] + ' ' + @word
I was kind of left scratching my head as @tempstring always seem to end up being empty, even when @word did have some data.
Then I remembered something from my start up SQL classes... anything+null is always null. So I initialized @tempstring to an empty string and viola, there the data was, correct and ready to use.
Having been working so much in VB that it actually took me quite a while to figure out those small klinks that are specific to TSQL.
This is the kind of situation that may really affect someone moving from C# to VB or vice versa too. :)
Over the weekend I fixed up a couple of bugs in the PDC 2008 Video Viewer and Downloader( My WPF based Viewer and Downloader for PDC 2008 Videos). This version is not available for download in CodePlex. All releases in Codeplex also contain the Source code. Do not download the source from the Source tab. The Source tab is a older version of the source and not the latest.
I have already fixed a couple of bugs in the application. To make it easier to download and manage versions I have created a CodePlex project to manager this applications.
I am not using the source code part of Codeplex. Instead, I will use the releases system to upload zips with both the source and the executables.
Version 18.104.22.168 is already uploaded.
I couple of years back I wrote a .NET tool the just download the whole set of PDC videos for offline viewing. So when the PDC 2008 videos became available I thought I would update the tool. But then I thought that this gives me a chance to kind of mess around with some newer technologies and so wrote a PDC video downloader and viewer using WPF, VS 2008 and SQL CE 3.5.
The whole source and executable are available for download below. I also added the session and speaker information that is available online at the PDC site, so it is a true Offline PDC viewing experience. I have also added a few extras like try and find out the blog of the speakers (this is based on an automated live search and so I expect that there are quite a bit of wrong links. If you can update the database and send it back to me, I would update the application).
Note that the application currently only downloads WMV-HD, so make sure you have enough free space on your HDD and bandwidth to download (from what I found searching online the videos will be 50+ GBs).
I don't have too much of error handling code, so expect crashes when things go wrong. If you find a bug, let me know, I will fix it when I get time.
It does happen sometimes that you start off a project write up a few classes and then suddenly realize that you need to break this up into multiple projects or re-organize your solution structure.
So when I saw this post on an option in Visual Studio 2008 that was supposedly called "Project From Existing Code" I was quite surprised how I never noticed it.
I always set my dev settings to the VB Developer Settings and this option seems to be not available in the menu for that setting. If you want to use a different setting and want to use this option, you will need to customize your Menu Options to do so.