TFireMonkeyContainer bugfix for instances created at runtime

TFireMonkeyContainer is a small open-source VCL component that can host a FMX form, allowing you to embed FireMonkey forms inside your VCL application.  It works with XE3 and above.

A 3D FireMonkey form embedded in a VCL application,
using TFireMonkeyContainer

This post is to note a bugfix that handles the case where instances of TFireMonkeyContainer are freed in a different order to the order in which they were created.  This is most likely to occur if you create instances dynamically at runtime, such as in a tabbed multidocument application (like a web browser): creating one on a new tab, but freeing them in different order, such as if a user closes a different tab.

If running in debug, you would get an assert in this situation.  If running in release, you would notice focus issues, where the host form may not have drawn its title bar as though it was the active form when the FMX form inside it was focused.  If you only ever had one embedded FMX form per VCL form or if you never created an instance at runtime, ie only used it by placing a TFireMonkeyContainer on a form at designtime, you wouldn't run into this bug, although this update does clean up the code and you should use this latest version anyway.

The bug was due to the control needing to subclass the VCL form on which it sits.  In some slightly silly code the implications of which I overlooked, this was done once per TFireMonkeyContainer instance, but should have been done only once for a given VCL form no matter how many embedded FMX forms were on that host VCL form, and removed only when the last TFireMonkeyContainer instance was freed.

If you haven't used TFireMonkeyContainer before, and you want to make use of FMX in your VCL app, try it out!  You can find the project on Google Code, and check out or update the source from SVN.  Read more posts about it on the Code & Components page.

Mysteries of IDE plugins: Painting in the code editor (Part 1)

Part 1 of a series on how to write an IDE plugin that can paint on the IDE code editor line-by-line along with the code.

Introduction: IDE plugins

IDE plugins are a mysterious topic.  With little official documentation, most material is scattered across a variety of blogs and websites, often out of date and referring to pre-Galileo IDEs.  But the documentation is improving and there are a number of great websites (try David G Hoyle's - the best website on the the 'Open Tools API' I have seen) delving into the interfaces that OTAPI makes available.

If you want to write a normal IDE plugin, such as a dockable window containing your own material, something that interacts with the projects or project groups, adds or removes text to source code in the editor, implements a debug visualizer, etc, then the above collection of links plus some Googling should get you going.

But there are some things that the IDE's API simply does not expose, things that advanced plugins need to do - like painting on the code editor.  There is no painting access or interface in the published API.  So how do open-source (like GExperts or CnPack) or commercial (like Castalia) plugins do it?

I can't answer how commercial plugins work, since I don't know, though I suspect very similarly to the technique presented here.  But I can tell you a technique I figured out with the help of looking at how CnPack achieved it to solve a few issues.  It works well, and seems stable, reliable, and fast.

This is the first of a series of posts on advanced Delphi IDE plugin functionality.  There are several goals: now I've found out how to do some of these trickier things, I want to share the information.  Second, I am writing a family of IDE plugins - things to fix parts of the IDE that after many years of usage I wish worked differently, or to add functionality that doesn't exist.  And while I want to sell these plugins for a (small to trivial!) cost - I hope you, Delphi readers, will find them as useful as I will - I don't want to hide the information I learned about how to build them.  Quite the opposite: I'd love to see a healthy plugin ecosystem develop around Delphi and Appmethod, rather than the small number of open source and commercial plugins that currently exist.

The following assumes you have a rough idea of how to create an IDE plugin - ie how to create a BPL that it loaded by the IDE.  But even if you don't, you can read on anyway :)

Contents

• Introduction, and a few quick notes on IDE plugins (above)
• What is TEditControl?
• Painting unsuccessfully: by hooking windows and messages
• Painting successfully: patching IDE methods at runtime

By the end of this article, if you already know how to write an IDE plugin, you should be able to make your plugin paint on the code editor in an event that occurs each time a line is painted.  Completing the functionality, such as drawing in the right spot for each line, handling the gutter area, handling folded code etc, will be discussed in future articles in this series.

What is TEditControl?

Delphi/C++Builder is written mostly in Delphi, and so uses VCL menus, actions, and controls, an Application object, etc.  You can see this when writing standard plugins, where you use a descendant of TForm (eg a TDockableForm) to make a dockable window in the IDE, add images to the IDE's image list, add menu items, etc.

This extends to the code editor itself, an instance of TEditControl, an internal Embarcadero-only descendant of TWinControl.  Here you can see it in Spy++:

An instance of TEditControl seen inside the IDE.

There is a single instance of the edit control shared by all source code tabs in the main IDE window - the tabs are more like a TTabControl than a TPageControl.  However, you can have multiple edit windows, each with an edit control, even showing the same source code.

The relationship between TEditControls and the documented OTAPI interfaces, such as IOTAEditView, can get complicated and will be addressed in another article.

But how do you find this control and paint on it?

Painting unsuccessfully: by hooking messages

I tried a number of techniques that didn't work and I won't delve into great detail.  My rough approach was to hook the window messages or events related to painting, and also to scrolling or other events in order to update internal information in my own plugins.

For example, I tried setting the control's WindowProc to my own, in order to catch WM_PAINT messages - none were caught.  You could also try finding an OnPaint event or similar.  I also tried catching WM_HSCROLL and WM_VSCROLL messages via WindowProc, installing an application OnMessage handler (note if you try this approach that it uses a multicast event dispatcher, so don't assign your OnMessage event to Application.OnMessage directly), and installing a message hook, and none of those methods caught those messages either.  I don't know why: it's possible that I was hooking the wrong control (despite it being the edit control) or that the messages are handled somewhere else.  I don't know the internal structure of how the IDE interacts with the edit controls.

I would recommend you do not go down this road, since I had no success with this or several variations of it.  (That's one short sentence that describes quite a bit of spent time.)

To do any of these, you also need to find instances of the edit control itself, wherever it is (or they are) in the IDE, another minor complication.  But it turns out that purely for painting, you don't need to know about any specific instance of the edit control.  Instead...

Painting successfully: patching IDE methods at runtime

One method of implementing painting I had read about online was hooking or patching methods of the IDE classes.  This was referred to obliquely, as 'you can patch the methods', with no details about how.  So figuring out how was my next approach.  There are two steps: how to patch any arbitrary method, and then finding the right method(s) to patch.

How to patch an arbitrary method at runtime

A method has an address: a location in memory at which it starts.  Virtual methods or thunked (eg most Windows API) methods jump to this from another location in a variety of ways.  To patch a specific method, you need to find its true location (following the virtual call or the thunk) and then overwrite the first few bytes of the method itself with a call to jump to your replacement.  If your replacement does everything you want, that's all you need to do, but if you want to call the original implementation as well then you need to replace the bytes you overwrote with their original contents and then call the original method by its original address.

Luckily there are several libraries out there for doing this at runtime in Delphi.

The patching code I use is a variant of Chau Chee Yang's code.  (You could probably also use another, such as the Delphi Detours Library.)  Using it, you can patch a method with code similar to:

  FOriginal := TCodeRedirect.GetActualAddr(...address of a method...);
  FHooked := TCodeRedirect.Create(@FOriginal, @ReplacementMethod);

GetActualAddr is a method that checks for an indirect jump.  I have not investigated the details of how well this handles both virtual methods and WinAPI-style thunked methods - in fact while I know the theory of how both are implemented I'm hazy on the exact details since I've never had to examine them closely.  This code works as-is for the purposes of the patching done here.

My version of the above code includes a bugfix for the Disable method: the code on the linked page attempts to write to memory to patch back the method, but it fails because it does not change the protection of the memory first allowing it to be written to. It also does not re-protect it once written, or flush the instruction cache afterwards.  A simple modification based on Enable gives:

procedure TCodeRedirect.Disable;
var
  OldProtect: Cardinal;
  P: pointer;
  n: NativeUInt;
begin
  if FInjectRec.Jump <> 0 then begin
    // David M - based on Enable()
    P := GetActualAddr(FSourceProc);
    if VirtualProtect(P, SizeOf(TInjectRec), PAGE_EXECUTE_READWRITE, OldProtect) then begin
      WriteProcessMemory(GetCurrentProcess, GetActualAddr(FSourceProc), @FInjectRec, SizeOf(FInjectRec), n);
      VirtualProtect(P, SizeOf(TInjectRec), OldProtect, @OldProtect);
      FlushInstructionCache(GetCurrentProcess, P, SizeOf(TInjectRec));
    end;
  end;
end;

Finding the right method to patch

You can now patch a method on a Delphi class live at runtime.  But which method?

At this point, I started searching through the methods in coreideX.bpl looking for appropriate ones to patch.  It looked like being a long search with many false starts and crashes based on incorrect method prototypes.  But I realised that there are open-source projects which implement editor painting already and they may well use this or another method, and so I looked at one to see how they achieved it: CnPack, which I had already downloaded based on David Heffernan's suggestion for how to keep track of code folding - a topic I will return to in a later article.

Note: I have to thank the authors of CnPack for their code, which helped me solve some problems with my own.  It took me quite some time to figure this out, and I gratefully used their code for reference and help. 

CnPack does indeed patch IDE methods - in fact it patches a lot of IDE methods, many more than just this one.  They patch one in particular, though: not a generic paint method for the control, but a method PaintLine - which looked perfect, since the purpose of painting on the edit control is (probably) to paint on specific lines of code, not to paint 'in general'.

Remember, patching the TEditControl.PaintLine method means that whenever any instance of TEditControl tries to paint a line it calls your implementation instead of its own.  You don't need to know about specific instantiations, just the class, because you are writing a replacement method for the class.  Your method will then call into the original patched-out code, in order to draw normally, as well as performing whatever painting it wants to do.

Note: this is completely unsupported by Embarcadero (or me.)  Their internal code can change at any time, even in the one version of the IDE.  Nevertheless, this specific method seems to be stable since at least Delphi 2010.  CnPack has a number of different method prototypes for different versions of the IDE, so this has changed in the past and may in the future.

The method in question is Editorcontrol.TCustomEditControl.PaintLine.  It resides in coreide*.bpl, such as coreide160.bpl, which is always loaded into the IDE.

PaintLine seen in Dependency Viewer

In the above screenshot, you can see PaintLine in Dependency Viewer with a "mangled" name.  Once demangled via tdump, it is (given in C++ format):

__fastcall Editorcontrol::TCustomEditControl::PaintLine(Ek::TPaintContext&, int, int, int)

We don't know the structure of TPaintContext (which would be very useful) nor the meanings of those three integer parameters (yet; actually CnPack has deciphered two of them.)  Also not shown is the implicit Self parameter.  But this is enough to create a stub.

Hooking PaintLine and using it to paint

We already have a prototype for the method, but in the form of an object method not a standalone procedure.  To hook, we need to define a compatible method with the extra Self parameter:

TPaintLineProc = function(Self: TWinControl; A: Pointer; B, C, D: Integer): Integer; register;

A is a pointer / reference to TPaintContext, which we don't have the definition for so cannot use.  B, C and D are the three int params.  However, the first parameter is the normally-hidden Self parameter; that is, the object on which this method is called.  Note the register calling convention, which so far as I can tell is fastcall by another name.  This allows you to implement an object-oriented method - one called on an object, that is, with a Self parameter - in procedural style.

Create a stub method using this prototype:

function HookedIDEPaintLine(Self: TWinControl; A: Pointer; B, C, D: Integer): Integer;

begin

  // ...

end;

Note: 'Self' here is a TWinControl, since we know that the edit control is a TWinControl descendant, and we can assume that the patch works and will only be called with a TEditControl 'Self' parameter.  You could also make it a TObject and check its type at runtime.  I did this in my first implementation.

To be completely clear, this is not an object method but a plain, non-OO procedure.

And to get it to be called, hook the IDE's method:

var

  FOriginalPaintLine: TPaintLineProc;

  FPaintLineHook : TCodeRedirect;

  ...

  FOriginalPaintLine := TCodeRedirect.GetActualAddr(GetProcAddress(FCoreIDEHandle, '@Editorcontrol@TCustomEditControl@PaintLine$qqrr16Ek@TPaintContextiii'));

  // Patch the code editor's PaintLine to use our method

  FPaintLineHook := TCodeRedirect.Create(@FOriginalPaintLine, @HookedIDEPaintLine);

Note: FCoreIDEHandle is the module handle of coreide*.bpl, loaded into the current process.  Finding this is left as an exercise for the reader, and there are variety of ways.  This method may get you started.

Now, fill in the stub method to call the original method:

function HookedIDEPaintLine(Self: TWinControl; A: Pointer; B, C, D: Integer): Integer;

begin

  FPaintLineHook.Disable;

  try

    Result := FOriginalPaintLine(Self, A, B, C, D);

  finally

    FPaintLineHook.Enable;

  end;

end;

You need to disable the hook before calling through the pointer to the method because otherwise you will recursively call back into your own hooked implementation, because you are calling the address of the patched code that jumps to your method (because that is the address of the real original method, the first few bytes of which were overwritten.)  Unpatch it back to what it was before calling it.

Note: Another exercise for the reader: call back into your own painting class to call your own code, rather than remain in procedural code only.

At this point you have a lot of work complete, but no visible result at all because your hooked method still doesn't do any painting and just calls the original.  To paint, you need a canvas, and luckily we are dealing with VCL controls and know that the edit control is a descendant of TWinControl.

Note: Since Delphi is written in Delphi, you can find out quite a lot of information when debugging the IDE in a second instance of the IDE - one example is inspecting the Self object (the edit control) above if you break in your method.

There is a handy VCL class to represent a canvas on a TWinControl: TControlCanvas.  Create and use one:

  Canvas := TControlCanvas.Create;

  Canvas.Control := Self; // The TEditControl

  Canvas.TextOut(0, 0, 'Hello world');

The results

Voila!  You now have text on the code editor:

Hello world!

Next steps

There are a number of important missing elements that need to be addressed in order to turn this proof-of-concept code into useful code:

• It always draws at (0, 0), not at the line that is supposedly being painted.  Since this is called for every line, the above 'Hello world' text is actually drawn over itself in the same palce many times, once for each line the code editor paints.  We need to extract the line information and paint at the correct offset down for each line, and correct offset right to handle the code gutter
• It doesn't have any idea what unit or text it is actually drawing over
• It doesn't have any idea what lines of text are visible
• ...and when it does, that is going to have to include handling code folding

Luckily, these are also solvable and the next article will address these problems too.  Until then, consider we have made great progress: we can now paint on the code editor from an IDE plugin!

Why am I investigating these topics?

I'm glad you asked :)

I've used Delphi since it was Turbo Pascal, first as a teenager/student and then over the past decade in action as my day-to-day IDE - including C++ Builder.  There are things about the IDE I wish worked differently: mistakes I keep making because I expect certain behaviour, even though I should know by now it works subtly differently.  There are also features I wish it had, and feature tweaks that would just make it a bit 'slicker', something it needs in order to compare to IDEs like Visual Studio which have a very polished UI.  Polish matters in an application you use all day, both visually and in behaviour.

I run a small Delphi consulting company called Parnassus, through which I solve problems and write good code.  I'm expanding and am writing a family of small IDE plugins, each of which improves the IDE by changing current behaviour or adding new features.  The first of these, the one through which I have learned about the problems and techniques documented in this blog series, is small - it was my learning project - but useful: a new implementation of bookmarks, one much better than the bookmarks in the IDE or GExperts.

Preview of an alpha, pre-release version of Parnassus Bookmarks

It will be available soon, and if you are interested in beta-testing please contact me by email or by commenting below.

Useful reference about disabling specific compiler warnings

Earlier today while working on an IDE plugin, I got the following compiler warnings:

[dcc32 Warning] W1029 Duplicate constructor 'TLineDifference.CreateEqual' with identical parameters will be inacessible [sic] from C++ 

[dcc32 Warning] W1029 Duplicate constructor 'TLineDifference.CreateAdded' with identical parameters will be inacessible [sic] from C++  

This is caused by a record type having two or more constructors taking the same parameters.  (For interest, the type TLineDifference here represents a single difference in two text files, ie one of many calculated when diff-ing source code, for example.  The type of difference - a line being added, removed or equal - could be a parameter but when using the type in code, it is more readable and clear-in-intent for it to be part of the constructor name.)

In the past I've dealt with C++ compatibility warnings, in code that is not intended for C++ consumption, by making sure the "Project Options > Delphi Compiler > Output - C/C++ > C++ Output file generation" option was set to "DCUs only".  But in this case, it didn't have any effect.  And since the code in question is (a) internal to the BPL and (b) is intended for Delphi use only I wanted to disable the warning.

Enter this very useful blog post on Delphi's Hidden Hints and Warnings.

For each warning code (eg W1029) there is a corresponding name you can use either on the command line or in a {$WARN} directive to enable or disable it, restore it to its default on/off state, or promote it to an error.  For example, using the name for W1029 gives:

{$WARN DUPLICATE_CTOR_DTOR OFF}

The page has a table of code to names - very useful - and some other miscellaneous information.

Some side notes:

• You have to use the name not the warning code.  I would prefer to write something like {$WARN W1029 OFF} because it's then self-documenting what warning it is that is being blocked, but neither this nor variations of this syntax work.  You must use the name not code.
• This specific warning, W1029, has to be turned off in the project file not in the unit that generates the warning (in fact, the warning doesn't specific a file or line number.)
• The blog has a number of other interesting articles on Delphi topics, such as using generics on enumerated types which do not have RTTI.
• In a 'it's a small world' coincidence, while writing this from Estonia, I believe I knew the blog author Marc ten or more years ago as a friend-of-a-friend when I lived in the same town in Australia.  I don't think either of us knew the other used Delphi.  It is a small world indeed.