Converting from StarTeam to Subversion - Part 1

Being a long time user of StarTeam (since 2000) we have been reasonable content with our StarTeam installation. We do however use a number of different operating systems and with Borland no longer giving appropriate support to OSx and the fact that the integration with Visual Studio is extremely bulky and only available on the top tier of the product  I decided it was time to switch to something new. A number of our developers had used Subversion in the past so this was the obvious choice - the repository would however need to be converted. Having gone through the conversion from MKS to starteam in 2000 I thought I would take responsibility for the StarTeam to Subversion conversion.

I found a utility written in Java called svnimport from Polarion and it did a reasonable job until it hit our main project and kept throwing out of heap exceptions. I tried all the Java tricks of setting a large heap size and aggressive heap usage but it would always die at about 1.3GB of allocated memory (I had 12 GB so I think the issue was the way the utility was written). After a bit of searching I found the source code for svnimport. I downloaded a copy of myeclipse and started debugging svnimport to see if I could possibly fix the issues or change the way it was using memory. I am however not a big fan of Java so I decided to convert the project to C# and use the Borland .Net libraries.

The first issue I discovered with svnimport was that it did not do its branching or tagging intelligently if you had 10 branches that were created at different times there would be 10 copies of everything up to the branch time. this meant that if you had a 1MB file with 10 revisions in your main trunk it would be present 10 times in every branch. This design caused one project to take 18 hours to dump and 4 days to import creating a 24GB subversion repository

The first thing I changed was to build up a list of branches and tags sorted by creation date (you need to check the attributes and see if the branch is based on a label and then get the appropriate date from the label item).

Svnimport builds up a list of actions/commits based on the revision of each file, I changed this to only process files that had a modified time > than the creation time of the branch - this cut down on the issue above of having duplicate files. now with these 2 changes I went about updating the dump file creation. The commits are sorted by date so all I had to do was insert code that would check to see if a commit date was larger than a branch or label and if so I would inset an action to create a branch/label that was copied from the current revision. Tags were easy as they are just a snapshot in time.

I did a couple of test runs - one project that was generating a 18MB dump file was now generating a 3MB dump, and the import into subversion was solid.

I could now start working on getting a more reliable conversion going.

continued in part 2...

Finding missing ClearQAM streams

Often, after installing a new Digital TV Tuner into your system Media Center will not detect any channels on it. You run 100's of  'Scan for more Channels' and just end up with 'No Channels Found. To add insult to injury you attempt to add the channels manually but you have absolutely no luck as guessing the channel numbers and sub numbers is just that - a guess.

Figure 1 The Dreaded - No channels found - screen

Well I had enough of that and I decided to write a small utility to scan every channel that media center scans and display the information in the hopes that someone else can find it useful - I know I did.
To understand why your channels are not detected you need to understand how Media Center 'detects' a unique channel. The first step is of course scanning for channels usually this starts by setting the digital tuner to channel 55.25 MHz (if you live in the USA) Media Center uses frequencies for BDA tuners to set the channel. It then jumps in increments of roughly 6 Mhz searching for a digital signal - there are some places where it jumps a different amount but this is outside the scope of this discussion.
Every time the tuner locks onto a signal the transport stream is scanned for a Program Association Table (PAT) table - the PAT table contains each program number and the PID associated with that program. After the PAT table is detected the Program Map Table (PMT) needs to be found. The PMT contains the information (PID and type) of each elementary stream i.e. video, audio, etc.
Now you are wondering what is he talking about what's all this talk about programs when we are talking transport streams…well a transport stream can carry many different streams that are multiplexed, each stream is identified by a PID which is contained in the PAT.

Now Media Center has determined that this is a digital stream and it wants to get 'Digital' information. The stream is now scanned for PSIP (Program and System Information Protocol) information. This starts off with the Master Guide Table (MGT) – this table, similar to how PAT contains information about the streams, contains information about PSIP tables. Most PSIP packets have a PID of 0x1ffb (8187) so detecting the correct table is just a matter of detecting the table field. The MGT will define if a stream contains a Terrestrial Virtual Channel Table (TVCT) used by ATSC transmissions or a Cable VCT (CVCT) which is obviously used by Cable transmissions. These two VCT tables along with the MGT are all that is used for channel detection. There are other tables – the Event Information Table (EIT) is used for guide information if a channel match cannot be found, the Extended Text Table (ETT), etc.
If everything panned out MCE would have detected your channels and you would not be reading this postJ. However there is one caveat occasionally program with duplicate program numbers may be overwritten. I have tested and verified this in a couple of times.

Figure 2 Channel information returned

As you can see in Figure 2 channel 72 two channels with PSIP information were detected and 3 without – the Call Signs are displayed along with the channel numbers that Media Center will present the channel as on the guide the other may be displayed as C72.97, C72.20 and of course C72.235. This of just depends of course on no other channels show up with the same program number and overwrite them.

Figure 3 Media Center Channel edit

As you can see in Figure 3 the information detected on the utility is reflected in Media Center. The pad-lock on the right shows that the channels are encrypted if there is no padlock you can check the box and the channel will show up on your guide without any information. You can then watch it for a while to determine what channel it is and change the listing manually.
If you do not see your program and you know more or less what channel it is you will be able to add it manually. Select Add a QAM channel enter the missing channel number which is either the PSIP number or the "channe .program number". You can select QAM256 by default it does not really make a difference as he tuner handles the demodulation not Media Center.

Figure 4 Adding Channel manually

Unfortunately Media Center will never map a xx.yy channel that it detected via a scan to a xx channel that you will see if you are using a cable card or a STB. This is because Media Center is treating every program as if is a ATSC program – scanning the internals of Media Center also reflects this.
The utility I wrote just scans the channels for now – I am busy porting some of this code to C# so that I can combine it with the code in my previous article to allow editing of the Media Center Guide listings directly. This will also give the ability to select a full channel number as opposed to a x.y number.

Here are the links to the files:
32bit ClearQAM Scanner.
64bit ClearQAM Scanner.

This files are offered without warranty, however they are not public domain please leave any comments re bugs etc. here.

To install run the registerdll.bat file then run the exe file - it will grab the first QAM capable tuner that Media Center is aware of i.e. you must have set up Media Center to use the Digital Cable Tuner before running this utility.

Detecting and fixing Channels in Media Center 7

One of the to-do's I have on my Streaming Tuner is to detect the channel call signs on the client computer for the encoded analog channels that are now digital. The second to-do was to get the channels to match up with guide information.

I managed to get the channels detected by scrubbing the transport stream (TS) created by the mpeg-2 encoder in the graph. I needed to edit the PAT and PMT - to change the program number of the stream and add additional streams (MCE does not like duplicate program numbers - I discovered after many hours of head banging). Then I needed to inject PSIP packets (namely MGT, CVCT and EIT) which would enable MCE to detect the callsigns of the channels. Needless to say I now consider myself to be somewhat of an expert in PSIP having thoroughly read the specifications and the book PSIP: Program And System Information Protocol: Naming, Numbering, and Navigation for Digital Television from cover to cover.

To get the callsigns turned out to be a much more complex the MCE 6 SDK has a few annoying features - one of which is the inability to provide channel names in cases where there are multiple sub-channels on the same channel number. As can be seen in the first image. Using the GetCallSigns method returns a serviceID and a callsign then by using the GetServiceIds method, which returns serviceID and channel number, one should theoretically be able to map the callsign to the channel number by using the serviceID and a common variable.

Figure 1 Channel 5.x on server

However as I mentioned earlier this does not work because for channel 5 only the serviceID of WPTVDT2 is returned with a channel number of 5.

I then shifted my focus to using XMLTV but that now requires a subscription for US guide data and the used would have to configure XMLTV as well as Media Center which would leave way to much space for mistakes. In addition to that I had not found a way for the client to match callsign to guide data yet.

Figure 2 Channel detection on client

Finally I decided to revisit getting the information from the media center database. I imported mcepg.dll into a project and voila I saw almost everything I needed. After a few hours I was able to pull the scanned analog lineup out of the database into an XML file.

 
 

Figure 3 Part of channel configuration file

Now all that was needed was to get the client side to edit the guide listing programmatically. I knew this was possible because you can import MXF formatted XML into media center. I had a look at that but it seemed more effort than it was worth. I decided to use the mcepg.dll again to edit the guide on the client side. This was also a pretty simple task except for the fact that I was getting exceptions every time I called the
Microsoft.MediaCenter.Guide.MergedChannel.AddChannelListings(Channel) method. I wrapped this in try catch and it worked fine - but I don't like hiding problems in code so I spent about a day finding out what was causing the exception (luckily I learnt to program in the days when asm was needed) - I narrowed it down to a inactive DeviceGroup. I tried a number of workarounds and finally managed to find one that would allow me to make my changes without corrupting the database of causing exceptions.

Figure 4 Client guide after channel scanning

Not only have I managed to programmatically remap scanned channels to a legitimate guide channel I have also managed to change a channel.subchannel into a channel number without any sub channels. The last image shows the guide after I did the first 2 channels 1.2 and 1.3. I know no one uses channel 1.x for anything so I just sent the channel on PSIP and major=1 and subchannel=analog channel number.

Figure 5 First 2 channels fixed

Next week I will finish up the installer and clean up a couple of issues so that I can make an alpha build and send it off to everyone that requested it. I think I might just also create a GUI for people to change their Guide information as well. I know I hate going through the "edit listing" menu and searching for a matching listing to map my clearQAM to my cablecard channels.

Forget the MCE Extender - just stream the tuner output - phase 2

It has been a very busy few weeks, I have learned way more about AVStream and BDA than I even wanted to know :). I have finally reached a point in the development cycle where I can release an alpha version of the code. The current implementation of my tuner driver currently supports ATSC and Digital/Analog Cable (clear QAM and analog cable). The Device managed capture shows the 2 drivers installed the first one handles ATSC and the second handles QAM. I found it easier on Windows Media Center (WMC) to have 2 instances of the driver each supporting its own tuning space as opposed to one instance supporting both.

I have not managed to have WMC detect the analog portion of Cable on the QAM tuner, I know it is scanning for the channels and the signal is strong and locked. I think this may be due to the fact that the mpeg-2 encoder is not inserting any EIT/ETT information WMC may be searching for the channel identification information in the stream. I have a strong belief that if I inject this information into the transport stream (TS), WMC will detect the analog channels. These are all the channels below channel 69 on Comcast in my area. The QAM tuner will then present itself more like a Digital Cable tuner.

Support for DVB-C should not be too difficult to implement - I just need to find someone with a compatible tuner to test it for me.

What I currently have working:

A user can install one or more tuner servers with as many tuners as they want irrespective of the technology i.e. Analog (Cable only in USA), ATSC and Clear QAM.

A client system will install 1 or more tuners depending on the tuners installed on the servers one tuner is used for each technology i.e. if you only have ATSC then install 1 tuner. If ATSC and Analog/QAM is present install 2 tuners. The system is totally fault tolerant if a tuner server goes off-line the clients will immediately start polling until the server re-appears or connect to another server if one has tuners available. The state is restored with the viewer only noticing a few seconds of missing video (if switching to a new server) or a delay as long as it takes the original server to come back on-line.

What I am working on:

• Injecting EIT/ETT into Analog stream for WMC. Done
• Support for DVB-C.
• Support for Set top boxes with IEEE interfaces.
• Support for CableCard (unprotected content for now).
• Interfacing with WMC to propogate recording requests
• Priority handling of tuners and concurrent access for shared content.
• etc.

I have a link here for anyone that is interested in testing the alpha when it becomes available. Register for Alpha.

Forget the MCE Extender - just stream the tuner output.

A number of companies that used to make MCE Extenders have stopped making them - I can understand why...you cannot play DVD's from a central server and you cannot access MCE applications like Netflix, Hulu, etc.

My solution to this is to try use the features included with Windows in order to get what you need and just create the rest. Since ATI came out with their firmware upgrade for their cable card tuner it is now possible to play recorded content on other computers on your network (except of course premium content like HBO). Content like videos, pictures, recorded tv, etc. can also be accessed by firstly sharing the folders and then adding the shares to your libraries on other systems. All this leaves is live TV.

There is a solution out there from SiliconDust called HDHomeRun but that will only give you ATSC and Clear QAM - although they have announced a cablecard solution. I bought one of the 2 port devices to play with and it seems pretty decent except for the fact that I need to keep poking holes in my firewall so it can communicate with its driver.

Looking at what is available it becomes obvious that the only solution would be to develop my own tuner streaming solution. I thought about reverse engineering the tuner portion of the extender but that seemed like way to much effort.

Phase 1

I spent 3 months delving into the AVStream class in the WDK during this time I played around with the sample tuner that is included in the DDK hacked out a lot of code and added a lot more. And finally came up with my first alpha solution.

For this I have an application on the 'tuner server' (main MCE box) and a tuner driver on the client system. when you start up a TV application on the client the tuner driver is started which communicates with the server. after that all tune requests are passed to the server and the tuner stream is passed back to the client.

I have included a capture of a Windows 7 x86 session running in VMWare (x86 is faster to kernel debug than x64 with windbg). Media Center is running in the VM communicating with the server (console box) on the left of the screen. My other client box is my Dell XPS notebook running Win 7 x64.

I experiance very little tiling but that could be because I am running on a 1Gbps network. I also spent a lot of time optimizing the network code.

This concludes Phase 1 of the project Phase 2 will include support for multiple simultaneous client and cable card tuners. I am still waiting for my Ceton 4 port tuner which I ordered in April but seems to have been delayed until July.

Part two...

Why can't a portable AC just power up automatically?

The problem:
We have a computer room that does not get central air after hours. Its gets pretty toasty by Sunday - to keep the air temperature down we installed a portable unit which vents into the plenum. Everything works fine however until there is a power spike or a blackout. We tried to find an inexpensive unit that would auto-restart but no luck.

The Solution:

The Current AC unit has an IR remote control so how about we just learn the codes and then anyone can terminal into the machine with the IR connected and run the application to send the power on command to the unit. Easy, right...

How will I get there:
I found an old Media Center IR Tranceiver lying around in the office and decided that would be the best way to go about this project. I then hit up Google tried all combinations of IR and infrared learing and programming and found basically nothing. Did find a number of references to Windows DDK.

Browsed the DDK and see what I can find - no client information but a couple of IOCTL_IR_* calls that could be useful if I can determine the device enumeration.

I did some more research on MSDN and found out that the client could communicate via IrBus - using the setupapi and the GUID for IrBus it is easy to get the device path.



Here are the functions for that:

hardwareDeviceInfo = SetupDiGetClassDevs ( &GUID_CLASS_IRBUS,...));
SetupDiEnumDeviceInterfaces (hardwareDeviceInfo,0,&GUID_CLASS_IRBUS,...))

PSP_DEVICE_INTERFACE_DETAIL_DATA pfunctionClassDeviceData = NULL;
// call this next function twice first time to determine buffer size needed
SetupDiGetDeviceInterfaceDetail (...,pfunctionClassDeviceData,...);

hDevice = CreateFile(pfunctionClassDeviceData->DevicePath,...);
SetupDiDestroyDeviceInfoList (hardwareDeviceInfo);

Now that I had a handle to the device I could start using the IOCTL_IR commands. However I discovered after many frustrating hours that the driver does not detect if the client application is 32 or 64 bit. I got a lot of funky results until I decided to compile my app as 64 bit.

With the 64 bit application it was easy to capture the power button on the AC remote.

9050 -4400 650 -500 600 -500 600 -500 650 -500 600 -500 600 -1650 600 -550 600 -500 600 -1650 600 -1650 600 -1600 650 -1650 550 -1650 600 -550 600 -1600 600 -1650 600 -550 550 -1650 600 -550 600 -500 600 -550 600 -500 600 -550 550 -550 600 -1650 550 -550 600 -1650 600 -1650 600 -1650 550 -1700 550 -1650 600 -1650 600 -39750 9000 -200 600 -1000000
However sending the command it is not as easy. When the remote learned the command it did not detect the frequency. I am currently trying to decide what the best parameters are for transmitting this block of data.

Why is my transmit not working...

After thinking about this last night and not being able to figure out a reliable way to calculate the carrier frequency with the data I had the only thing left was using pulse mode. To test that the emitter was actually working I learnt a button on my XPS noteboook remote and sent that back to the XPS to see if it worked - I chose an arbitrary pulse of 50 microseconds. This worked but still I could not transmit the power button of the remote control to the AC...I guess this remote is not working in pulsed or DC mode so I need to take more drastic options.

I guess the only option I have now is take it appart :).



This is a pretty simple circuit looks like a IR led/keyboard controler chip with a transistor, two caps and a resistor.
Google search for RSM2221-001 just ended up with a lot of chinese sites - using bing to do the same search I at leat got a 'translate' button. I discovered that this is actaully a knock off of a NEC μPD6121. I managed to fins a datasheet for this chip at http://www.datasheetcatalog.org/datasheet/nec/UPD6122G-002.pdf. The datasheet says the chip operates between 400 and 500 khz. Cool now I can just calculate the carrier frequency which will be the osc freq/12 so between 33 and 41khz. What I thought was a large cap just turns out to be a ceramic oscillator a CRB455E so I guess we are running at 455khz so IR carrier is 38khz. Now I can calculate the carrier period for the IOCTL_IR_TRANSMIT call, everyone knows period = 1/freq so period = 26.31 micro seconds so 26. Lets plug this in and give it a go again. And Eureka it worked - AC switched on just as I had hoped.
Now to get everything back to where it should be and see if I can switch it on remotely.

Further thoughts:
Now that I have everything back in the computer room...it dawned on me that the power is a toggle interface so I would have to know that it is hot (this info I can get from the UPS web interface) and that the AC is not already on. For the latter portion I will need a webcam and a form of illumination - but I will leave that for part 2.