J.H.Work

This time less words but more images.

Today I tweaked a lot of small details to make the world appear more believable. For example, only the currently loaded chunks (32×32 meter rectangles) were visible. This resulted in oddly cut off terrain. Especially when moving the player would also see new chunks appear out of nowhere.  To make it less obvious that the sight range is limited, I added terrain fog as a post processing effect. The further away a pixel is, the more does it’s color fade into the sky color behind it.

I made a small test world in about 10 minutes, here is how it appears to the player:
(Ignore the green bars in the corner, they are a relic)

In my last post about my work-in-progress-game I talked about integrating a physics engine to allow better player and entity interaction with the world. I did hardly more than gathering a list of possible candidates for my game. So far, I still have to decide between PhysX, Bullet and ODE. Each of them has it’s advantages and disadvantages.Another option is that I might also write my own physic implementation since all of the mentioned only offer a C++ API and the game is written in C and because I somehow like the idea of doing everything myself even though I am reinventing the wheel. Anyway, I have been focusing on other stuff instead. I added some of the features that were planned from the beginning but were not added yet because of various reasons, mainly a lack of importance. This includes normal mapping, a real deferred shading pipeline and multi-threaded terrain streaming.

Normal mapping (similar to the better known bump mapping) is a technique to make flat surfaces appear as if they have a bumpy structure. Basically, this is achieved by using an additional texture, a normal map which defines an array of normal vectors. Usually, normals are only defined per vertex which are pretty low-res in most cases.  Normal mapping defines the normal vectors per texture pixel, so the resolution is independent of the actual vertex quantity. In the last step the per-pixel normal vectors are used to calculate the light for that pixel. This works pretty well most of the time, but the normal mapped surfaces still appear flat at extreme view angles. Displacement mapping would solve that problem and include a lot of other advantages, but since only modern GPUs can render large surfaces using this technique I will not implement it in near future. Compared to displacement mapping, bump mapping is very cheap, especially when using deferred shading. Which is the next big feature worth to mention. Deferred shading has an easy concept: Instead of only rendering correctly lit surfaces directly to the framebuffer, the surfaces get rendered without any lighting applied to a temporary framebuffer and there is also a second renderbuffer to store the normal for each pixel. So, not only RGBA color is rendered, but also a normal vector for each pixel which points in the direction the pixel is looking at. The last step is to process both, the RGB(A) buffer and the normal buffer and each scene light to calculate the final lighted scene. Since it can happen that the resulting pixels have a color value higher than 1.0 (255) when using multiple additive lights (at least when using the proper renderbuffer format) the colors need to be scaled down to a 0.0 to 1.0 range. This is called tonemapping, or sometimes HDR. While I have not added a tonemapping algorithm yet, the deferred shader is already working. As a nice side-effect, it simplified the normal mapping process and allowed me to easily add light refraction for water. To render in multiple passes and with multiple buffers further makes it easy to implement post-processing effects like depth-of-field or SSAO in a fast way. I plan to add both alongside tonemapping very soon in my engine, as they add a lot of realism to the game.

The last feature has nothing to do with graphics, but still is very important to the appearance of the gameworld. Terrain streaming means that only the currently visible terrain is stored in memory and sectors are loaded and dropped as necessary when the players moves. This allows to have an nearly unlimited world size. The current state is that I support terrain streaming and editing at realtime. I had to invent a custom archive file format to allow for this, since terrain sectors can have any arbitrary size and it is very likely they become larger when they are edited, thus the file format must be able to support chunks that are growing in size. This was probably the hardest to implement because I also aimed to add multi-threading support so the game does not freeze when a big amount of chunks has to be loaded at the same time.

My next task is to build a small test area using the ingame editor and then re-enable the possibility to run around as a character. I want to get a feeling of how the created world appears to the player. I also plan to present a few details about the game mechanics itself.

As you know from my previous post I participated in Ludum Dare 23. Everything went pretty good on the first day and I managed to write a basic 3d voxel engine. Well not really voxel, but rather Minecraft-block-style. On the second day, however, everything that could go wrong actually did. When I booted up my computer in the morning, I was confronted with a lot of crashing programs. After some research I found that my harddisk had randomly corrupted files overnight for no known reason. It took me the whole day to get everything running again. I even had to reinstall Windows. After all that trouble I had about one hour left to work on my LD entry. Of course, that was not enough and I was not able to complete in time and submit my entry.

The idea of the game was that the player wakes up in a strange dungeon filled with gigantic insects (to fit the theme ‘Tiny world’) and has to explore the dungeon to find out why he is there. My intention was that there are no battles and the main focus lies on exploration and story telling.

Nevertheless, here are some screenshots from the unfinished game:

Well, at the time of writing this there are only a few people reading this blog, thus I am writing this mostly for my own entertainment: I will participate in the Ludum Dare game jam this weekend. It’s the third time I compete. For those who don’t know what Ludum Dare is, it’s a comeptition that is hold every 4 months and the task is to create a game in just 48 hours. After the competition there is a 3 week period where everyone who submitted an entry can vote other games. After the 3 weeks the scores and a ranking is published. The winners don’t get anything besides some fame. There is also a theme for each LD which the game has to fit in. But usually the theme is not very specific and you can make any game you want and change it to fit the theme in the last few hours. Still, since all the assets, the code and everything else must be done within the 48 hour limit it is a real challenge. The first time I participated I didn’t score too good and landed somewhere in the average ranks but the second time I was able to get an overall rank of #15 which is quite good considering there were more than 700 submissions.

If you want to track my progress during this weekend keep an eye on my Ludum Dare journal.

Recently I took part in a local hacking contest. To keep it short, one of the challenges was to bruteforce any of three passwords given by a unix shadow file. Since I did not find any good tool that supported SHA512 crypt hashes and was compatible with Windows I wrote my own. It turned out that bruteforcing crypt SHA512 is very tedious and pretty slow even when optimized and using multiple CPUs. So I came up with the idea to distribute the workload onto multiple computers.

Basically, the bruteforcer is split into two parts: The server, which distributes workunits. And the client which receives the workunits and then trys to find the password within that workunit and sends the result to the server. Using this method it is possible to have nearly unlimited processing power and finding even strong passwords can be achieved in a realistic amount of time. The more computers are processing, the faster the password will be found.

Additionally, as a gimmick, Shadow512 features a cool highscore list. This can be quite helpful when trying to convince your friends to help you bruteforcing. It is also a way to compare your CPU power with others.

Download Shadow512 here

For the last few days I have been working on shadows which are cast by objects. I have the basics done now, but it still can be improved a lot.
Adding shadow mapping required me to change a lot of the material system which kept me busy for weeks since a lot of the already existing code had to be rewritten.
Besides that I also added some other stuff. One example would be the possibility to import Wavefront models (see house and trees in the screenshot)  As a sidenote the trees have been made with ngPlant, a very nice tool to create any kind of vegetation. The other models are provided by a friend. Although only the house plus my tree models are integrated in the game for now.

Next up, I plan to integrate physics and proper collisions into my game. Until now I always used a very lazy way to check for collisions and I didn’t even bother to add collisions for the new imported models. I also hope that physics will make the world appear more alive and real.

A few days ago I noticed that my game uses huge amounts of memory. More than 1GB of RAM was allocated for just a small scenery. Since my engine is not that fancy it is unlikely that it would actually require such a high amount of memory. So I started to investigate. The problem could mainly have two sources. The first being that simply more memory is allocated than needed, the other one is that memory is allocated but never freed again, slowly filling up the ram with unused data relics. The second one is also called a ‘memory leak’.

To solve this, I could dig through the source code and check every call to malloc, realloc or free for potential errors. But this task is very tedious and it becomes nearly impossible to find memory leaks in big and complicated applications by hand.
So, instead I came up with an idea to locate such problems more easily and with technical assistance. Basically, I wrote a few wrappers for the original memory management functions and logged every call, including it’s parameters and return value, into a file.

For every source code file in the project I include the following lines in the header:

#define MEMORY_LOGGER_ACTIVE

void* _memLogger_malloc(int size, char* file, sint32 line);
void* _memLogger_realloc(void* old, int size, char* file, sint32 line);
void _memLogger_free(void* p, char* file, sint32 line);

#ifdef MEMORY_LOGGER_ACTIVE
#define malloc(x) _memLogger_malloc(x,__FILE__,__LINE__)
#define realloc(x,y) _memLogger_realloc(x,y,__FILE__,__LINE__)
#define free(x) _memLogger_free(x,__FILE__,__LINE__)
#endif

As you can see, it will redirect every call to malloc, realloc and free to another function and will also pass the sourcecode filename and linenumber of the call as parameters.

Additionally to the code above, it is required to implement the three wrapper methods somewhere.
I suggest to log the filename, linenumber and buffer size and the address returned (in case of malloc and realloc) and all parameters to a file. Don’t forget to call #undef for all three method names right before the wrappers to avoid recursive calling.
Why write to a file and not analyze the data immediately? This has several advantages. First of all, it is possible to keep memory logs for later usage or even deliver builds that have memory logging activated to your end-users. Some memory leaks only eat up RAM very slowly, which means that you have to run the application for several hours. Another advantage is that you can avoid calling many functions in the wrappers which will not only slow down the program, but also, if the external methods allocate memory, will end you up crashing because of endless recursive calls.

So far so good, now run your application, do some stuff while every memory operation is logged. If you use a small cache and do not write to the file immediately you shouldn’t even notice any slowdown. Anyways, after that you need to analyze the log. I wrote a simple application that keeps track of memory buffers and their size and where they have been allocated (filename + linenumber).
At the end I simply group them by filename/linenumber, count them and return a list, sorted by amount of total memory used.

Here is the final output for my game: (no censoring of super secret file names!)

Size: 026642400b (0025mb) Count: 01275   -   .\eisland.cpp(584)
Size: 008388608b (0008mb) Count: 00001   -   .\maineditor.cpp(270)
Size: 005328480b (0005mb) Count: 01275   -   .\eisland.cpp(698)
Size: 005242880b (0005mb) Count: 00001   -   .\eislandbuilder.cpp(19)
Size: 000698368b (0000mb) Count: 00073   -   .\geodetailbuffer.cpp(68)
Size: 000400400b (0000mb) Count: 00091   -   .\jhlib\gfx\ematerialloader.cpp(60)
Size: 000191744b (0000mb) Count: 00012   -   .\geobuffer.cpp(137)
Size: 000129600b (0000mb) Count: 00003   -   .\wavefront.cpp(397)
Size: 000092448b (0000mb) Count: 01197   -   .\eisland.cpp(885)
Size: 000088064b (0000mb) Count: 00172   -   .\jhlib\sdata.cpp(13)
Size: 000071624b (0000mb) Count: 01279   -   .\jhlib\gfx\egeometry.cpp(303)
Size: 000071400b (0000mb) Count: 01275   -   .\jhlib\gfx\egeometry.cpp(98)
Size: 000064512b (0000mb) Count: 00256   -   .\maineditor.cpp(286)
Size: 000052160b (0000mb) Count: 00042   -   .\jhlib\gfx\ui\eui_font.cpp(217)
Size: 000033704b (0000mb) Count: 01041   -   .\jhlib\simplelist.cpp(10)
Size: 000030672b (0000mb) Count: 01278   -   .\jhlib\gfx\egeometry.cpp(765)
Size: 000024576b (0000mb) Count: 00004   -   .\eworld.cpp(61)
Size: 000016656b (0000mb) Count: 01041   -   .\jhlib\simplelist.cpp(5)
Size: 000014400b (0000mb) Count: 00001   -   .\editorcursorarea.cpp(126)
Size: 000012288b (0000mb) Count: 00001   -   .\esky.cpp(55)
Size: 000011284b (0000mb) Count: 00091   -   .\jhlib\gfx\ematerialloader.cpp(8)
Size: 000008192b (0000mb) Count: 00004   -   .\eworld.cpp(79)
Size: 000007168b (0000mb) Count: 00001   -   .\jhlib\gfx\ui\eui_font.cpp(12)
Size: 000005152b (0000mb) Count: 01278   -   .\jhlib\gfx\egeometry.cpp(768)
Size: 000004112b (0000mb) Count: 00001   -   .\eworld.cpp(7)
Size: 000003432b (0000mb) Count: 00066   -   .\jhlib\gfx\eshader.cpp(850)
Size: 000003072b (0000mb) Count: 00003   -   .\jhlib\gfx\ui\eui_textbox.cpp(47)
Size: 000002560b (0000mb) Count: 00001   -   .\esky.cpp(54)
Size: 000002496b (0000mb) Count: 00039   -   .\jhlib\gfx\ui\eui_label.cpp(23)
Size: 000002016b (0000mb) Count: 00042   -   .\jhlib\gfx\ui\eui_font.cpp(84)
Size: 000001536b (0000mb) Count: 00024   -   .\jhlib\gfx\eshader.cpp(1173)
Size: 000001460b (0000mb) Count: 00073   -   .\geodetailbuffer.cpp(62)
Size: 000001404b (0000mb) Count: 00039   -   .\jhlib\gfx\ui\eui_label.cpp(8)
Size: 000001344b (0000mb) Count: 00084   -   .\jhlib\gfx\etexture.cpp(99)
Size: 000001152b (0000mb) Count: 00024   -   .\geobuffer.cpp(304)
Size: 000001024b (0000mb) Count: 00001   -   .\maineditor.cpp(265)
Size: 000000700b (0000mb) Count: 00025   -   .\wavefrontimportdetailed.cpp(92)
Size: 000000672b (0000mb) Count: 00012   -   .\geobuffer.cpp(263)
Size: 000000400b (0000mb) Count: 00001   -   .\emodelstructure.cpp(39)
Size: 000000336b (0000mb) Count: 00007   -   .\geobuffer.cpp(19)
Size: 000000294b (0000mb) Count: 00013   -   .\jhlib\sdata.cpp(158)
Size: 000000288b (0000mb) Count: 00003   -   .\jhlib\gfx\ui\eui_textbox.cpp(10)
Size: 000000280b (0000mb) Count: 00005   -   .\jhlib\gfx\egeometry.cpp(56)
Size: 000000252b (0000mb) Count: 00001   -   .\emodelstructure.cpp(18)
Size: 000000228b (0000mb) Count: 00003   -   .\jhlib\gfx\ui\eui_frame.cpp(8)
Size: 000000224b (0000mb) Count: 00004   -   .\jhlib\gfx\egeometry.cpp(77)
Size: 000000204b (0000mb) Count: 00003   -   .\maineditor.cpp(314)
Size: 000000192b (0000mb) Count: 00003   -   .\jhlib\gfx\ui\eui_frame.cpp(30)
Size: 000000192b (0000mb) Count: 00003   -   .\jhlib\gfx\ui\eui_textbox.cpp(60)
Size: 000000168b (0000mb) Count: 00006   -   .\jhlib\streamwrapper.cpp(5)
Size: 000000168b (0000mb) Count: 00006   -   .\wavefrontimportdetailed.cpp(109)
Size: 000000120b (0000mb) Count: 00006   -   .\jhlib\sdata.cpp(38)
Size: 000000120b (0000mb) Count: 00006   -   .\wavefrontimportdetailed.cpp(104)
Size: 000000117b (0000mb) Count: 00013   -   .\jhlib\sdata.cpp(144)
Size: 000000104b (0000mb) Count: 00006   -   .\jhlib\sdata.cpp(95)
Size: 000000080b (0000mb) Count: 00004   -   .\eitem.cpp(18)
Size: 000000064b (0000mb) Count: 00008   -   .\jhlib\gfx\euiiconatlas.cpp(32)
Size: 000000060b (0000mb) Count: 00001   -   .\maineditor.cpp(251)
Size: 000000056b (0000mb) Count: 00001   -   .\jhlib\gfx\egeometry.cpp(33)
Size: 000000048b (0000mb) Count: 00003   -   .\wavefront.cpp(97)
Size: 000000048b (0000mb) Count: 00001   -   .\eislandbuilder.cpp(11)
Size: 000000048b (0000mb) Count: 00006   -   .\jhlib\filemgr.cpp(21)
Size: 000000040b (0000mb) Count: 00001   -   .\jhlib\gfx\ecamera.cpp(510)
Size: 000000040b (0000mb) Count: 00001   -   .\geobuffer.cpp(247)
Size: 000000036b (0000mb) Count: 00001   -   .\jhlib\gfx\eshader.cpp(57)
Size: 000000036b (0000mb) Count: 00001   -   .\emodelstructure.cpp(7)
Size: 000000032b (0000mb) Count: 00001   -   .\jhlib\simplelist.cpp(21)
Size: 000000024b (0000mb) Count: 00003   -   .\engine.cpp(295)
Size: 000000020b (0000mb) Count: 00001   -   .\geobuffer.cpp(192)
Size: 000000020b (0000mb) Count: 00001   -   .\wavefront.cpp(94)
Size: 000000020b (0000mb) Count: 00001   -   .\wavefrontimportdetailed.cpp(827)
Size: 000000020b (0000mb) Count: 00001   -   .\jhlib\gfx\euiiconatlas.cpp(42)
Size: 000000020b (0000mb) Count: 00001   -   .\eitem.cpp(28)
Size: 000000016b (0000mb) Count: 00001   -   .\jhlib\gfx\ui\eui_font.cpp(8)
Size: 000000016b (0000mb) Count: 00001   -   .\jhlib\gfx\euiiconatlas.cpp(13)
Size: 000000016b (0000mb) Count: 00001   -   .\eitemblueprints.cpp(28)
Size: 000000012b (0000mb) Count: 00001   -   .\eitemblueprints.cpp(7)
Size: 000000012b (0000mb) Count: 00003   -   .\jhlib\gfx\ui\eui_textbox.cpp(44)
Size: 000000012b (0000mb) Count: 00003   -   .\jhlib\gfx\ui\eui_textbox.cpp(41)

At the beginning of the post, I spoke of >1000MB memory used. Regrading to the logs it should be somewhere at 100-200MB (note that I round down the amount of memory used, so even 987KB would become 0MB in the statistics)
That is because I stopped logging right after the game booted up. No map data was loaded at all. Still, I could locate a big memory waste easily. In the very first line it tells me that 25MB are allocated in eIsland.cpp on line 584. After checking the code on the given location I noticed that, due to a bug, a lot more memory was allocated than really needed.
Thanks to this approach I got my game down from 1GB to 100MB memory consumption.

I have been working on an open world RPG for a few months now. The game is written entirely from scratch and features a self written engine using OpenGL. It’s still very early in the process and there isn’t very much to see or do. And while I know that it is a unrealistically huge task for a single person to code a game of this size, yet also create all the assets, I am still very optimistic.

I originally planned to write a big story about how I came to this, but decided to wait and roll up the history at a later time, when this project shows more potential. Since it is not 100% clear on what the final game will look like, mainly due to technical and man-power constraints, I will keep quiet about details for now.

But as a start here is a current screenshot made in the world editor:

Finally, after taking a break for about a year, this blog is online again. I am going to write about all the random stuff I am doing in my spare time, mainly focusing on game-dev.