PDA

View Full Version : How does it work? And why is it slow?



HSovieticus
February 3rd, 2011, 22:35
First off, let me say that I am not a programmer, not yet anyway. But I do like math and I am planning to take computer programming courses at my local college. I have done some scripting in Victoria 2 and Adobe Flash, I've made some buildings for SimCity, and I am currently doing some 3d modeling for Fallout 3. Anyway, Dungeon Keeper is my all-time favourite game, and I am interested in how this remake works.

As I understand it, the original executable has been converted into a dll file which is accessed by the new exe whenever it needs to. From there, the original game is being incrementally rewritten. Is the new code also in the dll file, or is it located somewhere else?

I'm also wondering how the author got the game to work in windows 7. I downloaded the very first release and tried it out, but it didn't work, so I'm guessing that certain parts of the original game had to be rewritten for it to run on win 7. What needed to be done, exactly? And was the process similar to how one would create a simple compatibility patch to make the original game files work on win 7?

I'm also wondering why the game is so slow. I have a very new and expensive computer which I built myself, but the game can run slower than any of the new games I've played on it. Why is this? And what needs to be done or could be done to make it run faster?

EDIT: seems like switching to 32bit mode has sped up the game somewhat. Feels like it did all those years ago.

P_Hansson
February 3rd, 2011, 22:52
The new code is in the executable, the old code in the DLL. For an exact explanation, see: http://keeper.lubie.org/other_docs/change_exe_to_dll.htm

Some of parts the game are changed, in particular the the graphics output portion is changed to SDL (www.libsdl.org) from earlier DirectDraw. These things could make KeeperFX work where DK Gold doesn't, and unfortunately also the reverse.

As for why it is slow: Try comparing to 0.37c; if 0.37c works better, it may be due to the recent change to SDL. We've noted that Win7 users in particular have had troubles with slowdowns.

HSovieticus
February 3rd, 2011, 23:03
I would like to try a higher frame rate, but I am unsure of how to enter in the commands.

P_Hansson
February 3rd, 2011, 23:15
I don't think manually setting the maximum frame rate to a higher value would have any effect if the game is already sluggish for you.

The problem most likely is the graphics API for some reason cannot keep up (which unfortunately is especially common on modern OSes/GFX cards that aren't optimized for old pixel copying 2D graphics).

HSovieticus
February 3rd, 2011, 23:19
Sorry if I caused confusion. I made an edit to my first post which you may have not noticed. I managed to speed up the game by switching to 32bit colour in the cfg file. I want to test a higher frame rate to see if it slows down and by how much.

P_Hansson
February 3rd, 2011, 23:32
Aha.

You can set game speed to a value greater than 25. You do this by adding a command line option. Or you can use the graphical launcher (run_keeperfxtools.bat or something like that).

HSovieticus
February 4th, 2011, 00:34
Turned on vidsmooth, set fps to 60, maxed out view distance, scaled up the resolution. The game is no slower. Very fast actually. Shouldn't the game be set to 32bit by default?

Also, I was wondering if you could give me a bit more detail about my other questions.
How far along is the rewritten code?
What sorts of things have already been rewritten and what sorts of things have yet to be rewritten?
Does the game become more compatible, stable, and faster as the new code is rewritten? Or do you sometimes rewrite the original game's code just to rewrite it?
What is the project's focus (eg. compatibility, stability, performance, gameplay/graphical improvements)?
And are you guys planning to remake the graphics at some point?

P_Hansson
February 4th, 2011, 12:09
How far along is the rewritten code?

I don't want to guess on a percentage here, Mefistotelis may have an answer since he's the one that has rewritten near everything.


What sorts of things have already been rewritten and what sorts of things have yet to be rewritten?

Examples of rewritten areas: Hardware interfaces (graphics/mouse/keyboard/audio), a lot of gameplay code, the frontend interface, and so forth.


Does the game become more compatible, stable, and faster as the new code is rewritten? Or do you sometimes rewrite the original game's code just to rewrite it?

The problem is more the game to become "portable", it must be 100% rewritten, which may never happen. Regarding "faster": Although technically the new code probably executes faster because of improved compiler technology, it doesn't matter at all in this context, since new computers do not have a problem with the amount of computation a game from 1997 requires. Unfortunately, because of some hardware problems, some things we change may actually cause game to lag (that has been the case for the new SDL version for some users).


What is the project's focus (eg. compatibility, stability, performance, gameplay/graphical improvements)?

All of those, minus graphical improvements for now. I'm adding stuff that weren't in the original: TCP/IP networking, voice commands, a different kind of AI, etc.


And are you guys planning to remake the graphics at some point?

Potentially. The thing is, doing such a thing would completely change the feel of the game. It would have to be optional to activate new graphics. Therefore I have not prioritized it and will not do so in the foreseeable future, although I'm becoming increasingly convinced it would be possible. (Because of how many of the game data structures are now exposed in C.)

jomalin
February 4th, 2011, 12:27
The problem is more the game to become "portable", it must be 100% rewritten, which may never happen.

Why will this be never happen? It is OK if this take a long time but I think someday it will be finished, isn't it?

Krizzie
February 4th, 2011, 13:37
Cause they don't have acces to the original code. They have to reverse engineer everything and than replace it with new code.

jomalin
February 4th, 2011, 13:48
Yes I know, but this only requires lot of time (and lot of knowledge), but can be done, it is not impossible

kyle
February 4th, 2011, 17:09
I don't want to guess on a percentage here, Mefistotelis may have an answer since he's the one that has rewritten near everything.



Examples of rewritten areas: Hardware interfaces (graphics/mouse/keyboard/audio), a lot of gameplay code, the frontend interface, and so forth.



The problem is more the game to become "portable", it must be 100% rewritten, which may never happen. Regarding "faster": Although technically the new code probably executes faster because of improved compiler technology, it doesn't matter at all in this context, since new computers do not have a problem with the amount of computation a game from 1997 requires. Unfortunately, because of some hardware problems, some things we change may actually cause game to lag (that has been the case for the new SDL version for some users).



All of those, minus graphical improvements for now. I'm adding stuff that weren't in the original: TCP/IP networking, voice commands, a different kind of AI, etc.



Potentially. The thing is, doing such a thing would completely change the feel of the game. It would have to be optional to activate new graphics. Therefore I have not prioritized it and will not do so in the foreseeable future, although I'm becoming increasingly convinced it would be possible. (Because of how many of the game data structures are now exposed in C.)

You know, syn redone the textures in high definition, you could possibly use those and everything would look sharper, of course it should still be an option but thats probably the easiest way.

P_Hansson
February 4th, 2011, 19:01
Why will this be never happen? It is OK if this take a long time but I think someday it will be finished, isn't it?

Because it takes insane amounts of time to reverse-engineer everything. In the same amount of time, I could be programming a game from scratch. The only reason to do it is because we want this to be Dungeon Keeper, and not something else.

May never happen = people tiring on re-writing code. Especially likely to happen if only code which isn't useful to reverse in any way remains.


You know, syn redone the textures in high definition, you could possibly use those and everything would look sharper, of course it should still be an option but thats probably the easiest way.

It's not necessarily easier. The thing is, if writing a new renderer from scratch, not much more code must be reverse engineered. If textures are to be replaced in the existing renderer, all the code in the existing renderer which pertains to said textures must be rewritten (because of what textures are used and texture size are probably hard coded some extents). I believe this is the reason why Mefistotelis hasn't enabled higher quality textures already. The DK renderer is more complex than you might believe, given that it's implemented in software (rather than being a hardware interface).

HSovieticus
February 4th, 2011, 21:31
Someone needs to get a job at blizzard and maybe "misplace" the source code. :)

Seriously though, is there not a program you can use to convert the original game code, which I assume is binary, into something more intelligible?

Is playing the game the only way you can compare the new code to the old code?

As for replacing the renderer (and I assume that "renderer" means more than just the API; ie. OpenGL, Direct3d, SDL): It seems as though a new renderer would make the game much more compatible/portable and also more efficient. So have you guys thought of using an open source game engine like crystal space or the blender game engine? Getting the game engine to use the dll shouldn't be much harder than it was to write the existing executable which uses the dll, right? And I would think that rewriting all that has already been rewritten should not be as hard as it was to rewrite it the first time. Or maybe not, as it would probably be fairly different. You would be writing code to integrate the dll into the new game engine, to synchronize or connect the game mechanics in the dll to the rest of the game (graphics, frontend interface, and hardware interfaces) which would be recreated in / replaced by the new game engine. As opposed to recreating the original game engine (minus the game mechanics) from scratch. Although you are able to isolate and rewrite one function at a time, it is mostly guesswork (as I understand it) since you do not have access to the original source code, nor do you have access to any kind of documentation. I might be wrong, and I'm just guessing, but I would think that using a good open source game engine to handle the graphics and user input might be easier than rewriting and improving those things from scratch. You would still need to modify the game engine to use the dll in the right way, similar to how you are writing the executable to use the dll, but if your goal is to make dungeon keeper work on a variety of modern systems as well as it worked back in 1997, maybe even better, wouldn't it be easier to use an engine that already works like that, and then modifying it to make use of the dll for things that should not be changed (and, of course, for things that have yet to be changed)?

jomalin
February 4th, 2011, 22:09
That's true, reverse engineering a game tooks insame amount of time, as an example, there was a spanish game called "La abadia del crimen" for msdos. Some years ago, a programmer dissasembled the game and ported it to win32 C code, with 256 color palette graphics enhacement and some graphics effects (shadows, lights, etc), that takes him 4 YEARS... The final work he brings was magnific

P_Hansson
February 5th, 2011, 02:15
Someone needs to get a job at blizzard and maybe "misplace" the source code. :)

There is an oversupply of game programmers. ;) Finding such jobs is quite hard unless you got a perfectly matching degree, many years of experience or get it via social contacts. (Or do it yourself.) Oh, I'm aware you weren't serious, but yeah. :p

Btw you could probably implement DK as a mod in Starcraft 2 to a large extent (it's being done as well).


Seriously though, is there not a program you can use to convert the original game code, which I assume is binary, into something more intelligible?

In the general case, no, because C to assembler is a many to many mapping of instructions, and its inverse is many to many. Some of those possible representations are pretty hard to understand. We do use a disassembler which actually produces pseudo-code, but the fact is this pseudo-code is hardly more readable than the assembler it represents. For distance, it can generate code for a C expression like:

game->player[i].money = 128;

into

* (unsigned long *) ((unsigned long) game_field00 + 276582 + (i + 1) * 5574) = 0x80; //(fictional addresses, but this is how pseudo-C output looks)

For the record, I spend a significant amount of my RE-time recalculating addresses to more sensible representations. And if I get a single bit wrong, a new bug is born. Either way, it makes me wonder how long Mefistotelis has spent on it in total. :D


Is playing the game the only way you can compare the new code to the old code?

Not sure what you mean here. Compare, in what way?


So have you guys thought of using an open source game engine like crystal space or the blender game engine...

Yes, of course. It's not productive to try to write a modern 3D graphics engine from scratch unless you got man-decades of sufficiently skilled work force available.

Even with a fancy high level GFX engine it's a pretty major project and therefore carries a rather large opportunity cost. (The core game can be improved instead.)


That's true, reverse engineering a game tooks insame amount of time, as an example, there was a spanish game called "La abadia del crimen" for msdos. Some years ago, a programmer dissasembled the game and ported it to win32 C code, with 256 color palette graphics enhacement and some graphics effects (shadows, lights, etc), that takes him 4 YEARS... The final work he brings was magnific

Agreed.

HSovieticus
February 5th, 2011, 03:07
Can't you just disable the game's original renderer and then have the instructions on what draw (and where and how) routed to the new renderer instead?

P_Hansson
February 5th, 2011, 03:30
Yes, and like I also said, it is not simple.

HSovieticus
February 5th, 2011, 05:32
I'll take your word for it then.

jomalin
February 14th, 2011, 12:46
Yes, and like I also said, it is not simple.

Searching in KeeperFX 0.38b source code for "field_" gives 6537 matches! So there are a "few" variables to be properly renamed (and understand first). In fact, too much work!