A bug that is so deadly that can kill you...

A short story that I wouldn't want anyone else to live. Yesterday I was creating new animations with the Mystic Framework and found a small bug that I wanted to fix, nothing  important but was related to a UI glitch of the animation editor, so I decided to put some breakpoints and find the culprid class that created that annoying glitch and suddenly Visual Studio showed an error telling me that the source code is not matching the current build, ok I said , I will stop and rebuild so I can continue debugging, soon after it stopped around 2000+ errors popped up on the screen. This doesn't looks good, anyway, I am tired now so the best way is to quit and reload, and probably no errors will show up since I haven't closed VS for the last 3 days, well after loading the application to my surprise surprise! errors were still there, and after trying to figure out what was going on, I noticed that magically nearly 1.5 megabytes of code was gone from many classes, I don't really know how is that possible but many files were affected and the code was gone, nowhere to be found, only being able to see some code that maybe I wrote about a year ago and few pieces of code here and there was there!, I went from very angry to very sad in a second of my life.

Not all is lost, luckily, I made some periodic backups, I used to have a backup utility but it stopped working long ago, but didn't care to find a new one because I wanted to keep coding, so I only did the minimum and keep doing manual backups and zip the files, saddly the most recent back was made about 15 days ago, but not all is lost, keeping a log file in another place with the tasks I made since I started allowed me to find what I did, and restoring the old files plus using some code that I was able to salvage restored maybe 98% of the missing code or even a little bit more, and I only had to spend 2 hours more verifying that the code worked with minimum patching since the code was quite fresh in my mind I could easily fix it.

Now, I am a little bit worried that this may happen again, about 4 to 5 months ago something similar happened but was not serious just a few lines of code was lost, but this time it was almost everything I worked on the in the past 6 months or maybe more, so my weekend task is going to be "find a good backup utility", I haven't been able to find yet an "incremental" backup tool with zip capabilities so I don't get my hd full of backups, I saw a few ones here and there but most of those tools just synchronize files without compression, and without scheduler, I don't need a 5 minute backup tool, just once a day is more than enough so I can backup different folders and compress the new output, if none are available , something I doubt, I may have to write one but I don't want to spend time on it since I prefer to keep doing testing and fixing some other bugs in game as well as adding more gameplay features.

The bug in VS happened on all the form files I was using, I have one main form but its code is divided into several files for readability, so at some point of time VS may have gotten confused and broke in pieces and taking my code with it. Hope this thing doesn't happen to anyone else in the future.

Killing bugs!

I've been having small pixel errors in the shader while converting colors, and this piece of code fixed it finally, took me a few weeks to find this bug, still not quite sure why my values went over the limits but the commented code didn't do it before.

h += 360 * (-floor(h / 360.0));    // convert to absolute 0 to 360 degrees
//if( h < 0 ) {
//    h += 360;
//}
//if (h > 360) {
//    h -= 360;   
//}

pretty happy that I was able to find the issue and fix it in 10 minutes while testing the shader editor, I didn't see that bug until I used the real time capabilities to get instant feedback, trying to do the same in Visual Studio and XNA directly in HLSL is very complicated since I am not an expert in shaders yet and takes forever to see something on-screen.

Updated development video - shader and palettes editor

Finally I was able to make some time and published a brief short video showing up the shader editor I made during the last weekend to help me fix more shader coding bugs that I had for a few months and was taking a lot of time to debug. Also the color palette manipulation in the framework, color palettes is something that is not available with modern hardware but necessary for 2d games, so this is just an example of a color palette manipulation and how to take advantage of it to reduce the memory footprint by reusing textures and by changing a few paremeters you can change how your character looks in-game. The shading layer can also be manipulated while in game to adjust the shade strenght and I am thinking if I should enable shading coloring at this stage.

Now with the shading layer working and most of the framework working, it is time to redo all the animations of the game using this new platform!

Shaders! let's break the cycle

Shaders do make things come to life in the 3d and 2d, but it is quite challenging to have a Jack of all trades do it all platform that can help write good shaders since shaders are usually tied to game engines. I tried Nvidea and AMD solutions plus a few others without much luck.
So the cycle continues, I've been trying to improve and fix my shader for the past month, and still I haven't been able to properly finish it, it works but the quality I am looking for is still not there, it takes quite a lot of time to write a good pixel shader, especially when it takes some time to think and write code without knowing fully the syntax, try to compile it and get some syntax/coding errors, fix them and then again compile the shader, then compile the game editor or game viewer to test the result, if it doesn't works I have to go back and modify the shader, every single test takes from 3 to 5 minutes with a few more minutes of waiting for the compiler to finish plus review the outputs so it takes around 10 minutes for each test. I think I wasted too much time waiting, so what do you do when you can't find something that you need? well... I usually do something for myselft when something like that happens, and I wrote one for myself, and spent this weekend doing it, just a day and a half and I just made a realtime shader editor/compiler so no more wait! no more trying to find a tool that can handle my needs of mixing textures and many other things that I was looking for, now the sky is the limit! and here I go!.

I will post some videos of the real time shader editor with the new code for palette color handling soon! I am quite excited by the results of this week and looking forward to fix the last ones at the shader level.

Speed! Speed! I need to be faster...

This was a very difficult week, I was able to finally integrate and run the Mystic framework in XBOX360, finally!, and I was expecting to spend most of this week trying to improve performance of the framework, but after so many tries, it seems that XBOX doesn't have enough horse power to deliver all real time calculations I have in the framework as fast as I was expecting, sometimes I had constant framerates and sometimes from no where dropped quite a lot even in Windows.

I got stuck for about 2 days just optimizing the code as fast as I could and still enjoying the real-time calculations, but finally found a road block and couldn't find a way to improve it more, so now step 2 is to start caching values and make sure that the animations run perfectly. Luckily, I came up with many last minute optimizations and a few more are still in my mind, but I am getting now solid 60 FPS in-game even in XBOX, and this is without using multi-threading in the animation modules, when I added the multi-threading support I started getting even lower framerates from time to time, due to the local variables created per thread, so sometimes caching and re-using the same variables can really help improve the speed of the calculations but makes it impossible for some threaded blocks of code, specially for vertex calculations since I am not using the GPU at this moment for that due to the physics calculations per vertex. I will publish a video of the framework running in-game pretty soon, and it is working very nicely.

Just a quick screnshot, the character in the middle is being rendered using the Mystic Framework, the others using the old skeleton model, I will post a video of the small details achieved with the Mystic Framework animation tool.

Still need to investigate why in XBOX the color replacement technique is not working in the same way as in Windows, but in this case the color of the character in the middle was dynamically changed to green/blue for testing.

Sometimes code is so secure that I can't even access my own code...

Being trapped with security issues in the .Net framework got me stuck with no simple solution for quite many days, while trying to test the animation framework on XBOX 360 I got quite many warnings and many show stoppers, one that I couldn't figure out for almost 3 days was the confusing security implemented for classes in .Net, I know for some tools security is a must, but for a game I don't think a high level of security is needed, anyway, the content deserializer was able to properly decode all the animation and play it flawlessly in Windows but it didn't run on XBOX since I couldn't setup the same security level directly in the assembly code, so it was just telling me that the security methods that I was trying to setup didn't exists in XBOX so I couldn't set it at all, after many days of frustration found that by simple adding into each dll the following:

using System.Security;              // <-- this adds security
[assembly: SecurityTransparent]  

will clear up the dll requirements needed to be able to use reflection and call List<List<T>>
This is not rocket science nor something close to anything revolutionary, but I needed this to keep the framework simple and run it fast, finally, I was able to compile with the directives and the framework finally started to work in XBOX which really excited me. Saddly I am stuck now in another issue with the shader code which I hope to solve soon and be able to test the whole solution together. If this fits as expected I will post a video with the very smooth animations achieved with the Mystic Framework, it is not a revolutionary 3d framework since there are many, maybe hundreds or more made by tens or hundreds of people, so there is no way for a single person to compete against that, probably most people think that 3d is the future and that's where everyone is going, but I feel that there is a lot to gain also in the 2d arena using 3d technology, so I hope to make a niche new market with this solution. Rayman Origins by Ubisoft did a great step towards this direction, so I hope to achieve the same level at the Indie level, if not, well at least I tried.

The color palette of the 21st century...

Not so many years ago, previous hardware used color paletter to display colors and it was relatively easy to swap colors and create animations like rainbows, well with technology evolution it came as a thing of the past and today it is quite cumbersome and complicated to replicate the same effect, even color replacements can be quite difficult to achieve as in old generation hardware. I spent a week thinking how to achieve the same and searched everywhere for a good implementation that will have the less impact as possible, well as a matter of fact I couldn't find any code but just a bunch of ideas. Even something as simple as creating blending effects became a nightmare in a matter of days, like linear burn A+B-1 blending, can be quite complicated to achieve in modern hardware and I am still unable to perform this in a way that memory, bandwidth and speed are not compromised, since the framework has to be very fast I came up with some solutions in my mind and will post some videos soon. Sometimes I think I just only need one command to do whatever I want, but that command doesn't exists in the pipeline, or is available in the next generation hardware.

So far color replacement is working, but I haven't tested it yet in XBOX in game, for only one color, adding more colors shouldn't be that difficult but there are quite many restrictions with the technology available in XNA and pixel shaders v2 with XBOX, like the number of parameters that can be passed through the shader, so using an image to remap color palette seems to be better solutions but doing it in RGB color space seems to be pretty bad if you are using different shades of the same color, so HSV comes to the rescue, but the number of operations performed to convert each pixel to HSV can be quite taxing to the GPU of the XBOX, so hopefully I can squeeze as much juice as possible from the GPU and implement a color palette, stay tunned.