2.5D or 2D? normal maps quest for a better game

As usual, for any new game I do, I try to make something new on the tech side and learn new skills and usually increase the development time and make things ugly in the long term and delay the end result. But is a challenge I like to take just to make something different, something refreshing and something that I haven't done before, I did it with the Mystic River animation framework which works very well but it does not fit this project... yet , I have several plans for it but that will be for the next version. So what is the 2.5D challenge then? Well I've spent the last couple of days thinking how to improve the look of the game, and one of the key features is graphics, eye candy. So instead of trying to make super nice sprites with fixed shading I think I can make them even better by adding shading effects to add depth, that will make it look better without having a major impact on the development "time"...   well I was wrong but not by much.

2.5D brings some new challenges, but finally I was able to resolve them today, the simple use of normal maps to address the shading in 2D have been explored before in other games, like Legend of dungeon and also many others, but all of the implementations I found are for platformer type, or platformer camera style, with all sprites standing vertically aligned without rotations. Why do I mention without rotations? well if you use normal maps with lights, and use sprites in a batch the normal map information is baked in the normal map texture, and the light calculation is performed per batch usually, so the problem with this approach is that once you start rotating the sprites, the light information will also rotate to the same direction giving you wrong illumination. To illustrate the problem, the image below shows a bunch of asteroids and all are supposed to recieve the same light from a star located top right, but after rendering with rotations all the shades were off.

 

 

The green arrows show the actual shade of an asteroid which is pointing the to the wrong angle given the light source in red.

After trying for 2 days to align the shades, I found a not so nice solution. The thing I was trying to do in the pixel shader is not possible between the scope of spritebatch in XNA, the reason for this is because how everything gets batched, I had to think how the spritebatch batch works to understand why the light rotation is not working at the pixel shader to compensate for the rotation of the sprite.

The actual code that is not visible to the normal XNA user behind spritebatch is quite interesting, as per documentation there is a limitation 2048 sprites that spritebatch can batch at a single time, if more sprites are drawn more batches do happen, why? because it has a dynamic vertex shader with a fixed number of quads which is 2048, with a total of 2048x4 vertices. So how does the batch works then? it actually creates 1 quad per sprite and put each sprite into the dynamic vertex buffer, the transformations are performed in the spritebatch batch process in the CPU and then the end result is passed to the shader (yes up to 2048 sprites in one batch goes to vertex+pixel shader), which is what I was receiving in my shader, so I am getting a total of 2048 sprites in one go all of them already transformed, so I can't transform back the light source for each sprite because simply a maximum of 2048 sprites is already clumped together in one big mesh! so there is no way to move the light back for each sprite at this stage, it is too late for that, and as per my testing I didn't get each sprite vertex information while doing deferred spritebatch, that's why I wasn't able to resolve the issue at the vertex or pixel shader and the normals are not real normals, just baked normals, they all moved with the sprite, so the light source as seen on the asteroid image. While the vertex shader is getting all the vertex information also a tranformation matrix is sent with the vertex data, that means the whole screen transformation that happens for the camera view, which can't be used to compensate the light source for each sprite.

Then why does it works with 3d and not with spritebatch? with 3d you simply go and draw one mesh and do all the transformations for each piece and calculate the light source position for "each" piece, with a deferred spritebatch you can't do that because as I mentioned before is only one mesh of 2048 sprites with different transformations, but there is only one case in which this works, and this is when all the sprites are aligned exactly the same, meaning for example all are rotated 0 degrees, just a plain tileset or like what you see on Legend of Dungeon.

So what now? I can't adjust the light source for the whole list of sprites because simply it is only just one big mesh, but there is a solution for this but it is not nice, I just tested it today and it works. The solution is to actually rotate the light source for each sprite before it is drawn, even if you use spritebatch.begin .... spritebatch.end  you can still change the light source position passed to the shader, but the only way to make this to work is to make the batch sort to immediate, so every sprite is drawn after spritebatch.draw( .... ) so you can adjust the light position for each sprite just before drawing and voila! you have normal maps with rotation rendered at the proper location but I am not sure how slow the end solution will run yet, I just tested with a couple of sprites only, so the next step is to actually try with several hundreds and implement it in Junkcraft Armada.

To improve performance, only the sprites which require normal maps will be drawn using this technique, for any other sprite like bullets I will use the simple spritebatch to run as fast as possible.

I haven't seen any answer to this anywhere else, everyone mentioned how to do it in theory but I couldn't see why it didn't work, or how to implement it without having to code a lot of things at the vertex shader, I wish I could find a better solution with vertex shader implementation , but I really doubt it will work when having deferred sort with up to 2048 sprites in one spritebatch batch, in the meantime this seems to be the only workable solution.

I will post more screenshots during the weekend after I finish the normal maps drawings, I just only have at this moment very rough samples and won't really showcase the new render style of the game., but I am very excited with the preliminary results.

Just a quick sample I made a few minutes ago to test in a single module...