Depending on how you distribute and run your program, you may want to compress it using UPX, the Ultimate Packer for eXecutables.
However, this doesn’t play nicely with JclDebug, the JEDI Code Library unit that helps Delphi to generate stack traces in case of errors.
This article shows:
- How to get a stack trace from an exception,
- .. and in a readable format by adding debug information to your executable.
- How compressing the executable will break this functionality,
- .. and how to fix it by compressing the debug info separately before inserting it in the file.
- Full code in this gist
Before I start
A little disclaimer that this might not be the best solution for every piece of software. If you install your program on a PC, you may not even need UPX. Moreover, if you sell your program, you might not want to include debug information, because it can reveal information that could be used to reverse engineer your program.
In practice, I’ve never worried too much about reverse engineering (I’ve always written software for niche markets, I guess). In my current workplace we got a big executable that is loaded from a network share by hundreds of users, maybe not the best solution, but for us it works, and UPX’ing the executable just takes the edge off of the loading times.
UPX
Our program, at the time of writing, was about 110 MB without, and 265 MB including debug information. Following the steps below, that was reduced to just under 21 MB including debug information. 1/12 of the original size. Results may very, of course, depending on how well your program can be compressed. UPX typically brings the executable down to 1/4 to 1/5 of the original size. Debug information can be compressed a lot better.
Stack trace in the IDE
When you’re running a program from the Delphi IDE, it will be hooked up with the Delphi debugger.
When an exception occurs, the debugger can get the stack frames, to trace the back the chain of function calls from the exception to the moment the code started running (typically the root of the program, or the moment a Windows message arrived).
The stack trace it gets this way is nothing more than a bunch of addresses. This is hardly readable information. Contextual information is needed to map those addresses to positions in the source code.
Out of the box, a Delphi application comes with to build configurations, Debug and Release. Debug is the default one when you run your application from the IDE. The Debug config configures the compiler and linker to include a bunch of debug information in the executable. This information can be used by the Delphi debugger, or -theoretically- by the application itself when it’s running outside of the IDE:
Getting a stack trace from the program itself
Since Delphi 2009, the Exception class has a StackTrace method (returning a string), which, one would think and hope, returns the call stack at the moment the exception is thrown. But alas, by default it returns an empty string.
program CrashingApp;
uses
SysUtils,
Dialogs;
function CrashString: String;
begin
raise Exception.Create('Bang');
Result := 'Poof';
end;
procedure Crash;
begin
ShowMessage(CrashString)
end;
begin
try
Crash;
except
on e: Exception do
ShowMessage(e.ClassName + ' ' + e.Message + sLineBreak + e.StackTrace);
end;
end.
That’s where third party libraries, like the JEDI Code Library come into play. They can register functions into the Exception class, to help it generate this debug information. All you need to do is add JclDebug to the list of used units.
When an exception occurs, these hooks are called, they fetch a list of addresses, using stack frames, and voilá, the stack trace:
To generate a readable stack trace, you’d need debug information. Without it, you’ll just have a list of addresses. The debug information allows you to map those addresses to source code locations.
Including debug information
The debug information can be included in your executable by the compiler, by the JCL IDE expert, or by a post-build event. It can also be distributed as a separate file together with your executable, for example in the form of an external map file.
About the PE file format
Executables, aka images, follow the PE format. PE stands for Portable Executable. Without going into too much detail, these files are built up from sections. These sections are, amongst others:
- An MS DOS stub. Running the program in an old MS DOS environment, will run this stub, just to tell you that it can’t run the actual program.
- The core program
- Resources (strings, icons, version info)
- Imported functions / statically linked dll’s
- Exported functions
- Custom sections, like the embedded debug data
The screenshot above is of PEView, and shows the various section headers and sections, including the custom JCLDEBUG section that is added (in this case), by the JCL Debug IDE expert when building the program. It’s not the textual, big, map data, but a binary representation of the same information. This binary JDBG format is more compact than the map file, although it could be smaller when compressed.
How UPX breaks JCL Debug
UPX packs your executable. It takes the information of the various sections, compresses is, and restores that information in memory when you run the program. In the file on disk that information is no longer obviously there, though. The debug information is still in there, somewhere, but the JCL can’t access it anymore.
JCL Debug uses another JCL feature to read the debug info, PeMapImgFindSectionFromModule in the unit JclPeImage.pas reads a section by name from a loaded module, which doesn’t work in the packed executable.
Possible solutions
I want my executable to be as small as possible, while still having the debug information embedded in it. There are probably various ways to solve that, for example:
- Add the debug info after UPX’ing. Disadvantage: The debug info itself won’t be compressed.
- Make a debug info source that can decypher the UPX sections and extract the debug info from it. Disadvantage: quite complex, I think, since it’d require to reverse engineer a bit of whatever magic UPX is doing
- Add compressed debug info, and implement a debug info source that can read it. Sounds doable…
You can register a debug info source (a TJclDebugInfoSource descendant) that provides debug information to the JclDebug logic that builds the stack trace. By default, a couple of sources are registered that try to read debug info from a map file, a jdbg file, or from the executable itself.
Adding compressed debug info
My solution is to add a compressed version of the jdbg file to the executable after it is compressed with UPX. Adding it can be done using a simple command line utility. Since the executable has to be UPX’ed first anyway, it’s no use making this an IDE expert.
Looking into JclDebug.pas, you’ll find various functions for adding and converting debug information. The one I’m looking for is InsertDebugDataIntoExecutableFile, to see what exactly it’s doing. In fact, this function is invoking a TJclBinDebugGenerator, which searches for, opens and parses a map file on the fly and convert it to the binary format of a Jdbg file. It then opens the given executable, and uses some functions from JclPeImage.pas to append the information to the executable stream.
Even though the logic for converting a map file to the binary Jcl Debug information is in a separate class (TJclBinDebugGenerator), of which an instance can be passed to InsertDebugDataIntoExecutableFile,
I couldn’t find a way to recycle this, apart from copying a fairly big chunk of code and modifying it, so I did. Leveraging more Jcl code, I made a version that generates the binary debug data, compresses it using a TJclZLibCompressStream, and then appends it to the executable.
The executable can then read its own debug data using a TJclDebugInfoCompressedBinary info source.
The full code for writing and reading the compressed debug info can be found in this gist on GitHub.
Steps for using this in practise
The code in the gist can be used freely
DebugInfoTool can be compiled once and used in a post-build event for your project, or separately in your build pipeline. Simply upx your executable first, then pass it to DebugInfoTool, which will add debug info based on the map file.
upx.exe "$(OUTPUTPATH)"
DebugInfoTool.exe" "$(OUTPUTPATH)"
In your project, include JclDebug.pas (from the Jcl) and JclCompressedDebug.pas (from my gist), and add this line to your project’s dpr file, before Application.Run:
TJclDebugInfoList.RegisterDebugInfoSourceFirst(TJclDebugInfoCompressedBinary);
Final notes
At the time of writing my solution, I couldn’t find a way to extend TJclBinDebugGenerator, but by now I think there might be. If so, it means I could get rid of my own InsertCompressedDebugDataIntoExecutableFile, and make my solution a good 200 lines (65%) shorter and a lot simpler.