Suggestion for HxD:User templates for data inspector

Wishlists for new functionality and features.
dukk
Posts: 2
Joined: 24 Feb 2021 14:39

Re: Suggestion for HxD:User templates for data inspector

Post by dukk »

no questions yet.

You clarify all.

(sorry ALL for my possible bad english)
JakeTheDog
Posts: 2
Joined: 15 Feb 2021 10:16

Re: Suggestion for HxD:User templates for data inspector

Post by JakeTheDog »

Woah, that looks awesome you really put a lot of thought into the design.
Eagerly waiting for the release :D
Maël
Site Admin
Posts: 1454
Joined: 12 Mar 2005 14:15

Re: Suggestion for HxD:User templates for data inspector

Post by Maël »

Add option to specify memory layout of arrays:
https://en.wikipedia.org/wiki/Row-_and_ ... ajor_order
Maël
Site Admin
Posts: 1454
Joined: 12 Mar 2005 14:15

Re: Suggestion for HxD:User templates for data inspector

Post by Maël »

Support ROM images (SNES, N64, Amiga, etc.) with lookup tables for sprites, text tables, levels, etc.

Possibly, but not for an 1.0 release, with the ability to resize data structures, while keeping length and size fields up to date, and adapting memory allocation inside the file accordingly.
A simpler version of this would target PNG files or RIFF/WAVE files, that allow for easier resizing/memory allocation, while still needing a couple length and offset fields updated to stay valid.

From a high level view, resizing structured files is adding a kind of linker capability, since you need to section the file into parts, update offsets, possibly consider alignment issues, and memory allocation/section sizes, but also need to add constraints (that express logical dependencies between fields/parts of the file), such that the file remains valid. This includes updating indeces/lookup tables, length fields, range or size constraints but also checksums.

Editing resource sections in PE files is the "light" version of this, editing anything inside a PE file is the more complete/complex version, since program code might be dependent on this as well. Without additional "debugging" or "compiler/linking" information this might be difficult, since pointers in assembly code might need to be updated as well (and disassembling in general is not feasible without human assistance). In PE files there is a notion of relocations, but they can be stripped, or not available for other file types, like ROMs.

So it might be that files can be partially edited, with some sections remaining read-only (and just be moveable as entire sections, but besides changing the starting offset, remain unchanged), while others can be edited in fine grained detail, like resource sections (that were designed to be edited, independent of other parts of a PE file -- besides indices close to the PE header).
Maël
Site Admin
Posts: 1454
Joined: 12 Mar 2005 14:15

Re: Suggestion for HxD:User templates for data inspector

Post by Maël »

Use case example submitted by Denis Collis:
Simple example. Say I have C defs...

Code: Select all

typedef enum
{
  DEV_UART = 0,
  DEV_MODEM = 1,
  DEV_NFC = 2,
  DEV_BLE = 3,
}

typedef struct
{
  uint32_t serial;
  devtype_e devtype;
  char desc[28];
} prod_t;
If I then click on the first byte of the struct in the file, then the Data Inspector shows something like:

Code: Select all

|  prod_t   | serial:1020304,  devtype:'DEV_MODEM', desc:"Telit LE910"   |
And a feature request:
...maybe have capability of reading a .h header file/s with custom definitions?
MarcosG
Posts: 1
Joined: 18 Jan 2022 23:11

Suggestion for HxD: Blocks Tagging

Post by MarcosG »

I think it would be very useful to have the possibility of labeling sectors of the binary with different colors and a text that identifies each one. I think it is a very necessary option for binary analysis.

The idea is basically to be able to select blocks and put a color label on them with a simple text that identifies that block. It would be nice to be able to select the background color and the font color. And of course, being able to save these labels in a file parallel to the one analyzed.

And in the right panel there could be a list with all the tags and quick access by clicking on them.

Something like this:
8f1oP[1].png
8f1oP[1].png (214.24 KiB) Viewed 27407 times
Image taken from Google.

I would love to see this option in new versions. I really think it's very useful and I didn't find a tool with the convenience of HxD and a labeling option.

I greatly appreciate the work you are doing and I congratulate you on the position you have achieved.

Greetings!
Maël
Site Admin
Posts: 1454
Joined: 12 Mar 2005 14:15

Re: Suggestion for HxD:User templates for data inspector

Post by Maël »

The screenshot in the post above comes from the software at synalysis.net, which has a Windows version at hexinator.com/benefits.

A higher resolution screenshot from this software that shows more detail:
Parsing-results-explained[1].png
Parsing-results-explained[1].png (71.85 KiB) Viewed 27176 times

It uses "grammars" defined in XML which is quite verbose, and not easy to read or edit manually, for example for BMPs:

Code: Select all

<?xml version="1.0" encoding="UTF-8"?>
<ufwb version="1.23.4">
    <grammar name="Windows Bitmaps" start="id:2" author="Andreas Pehnack" fileextension="bmp" uti="com.microsoft.bmp">
        <description>Grammar for the Windows Bitmap file format</description>
        <structure name="Defaults" id="3" encoding="ISO_8859-1:1987" endian="little" signed="no"/>
        <structure name="Bitmap File" id="2" extends="id:3">
            <structref name="Header" id="6" fillcolor="00F900" structure="id:5"/>
            <structref name="BITMAPINFOHEADER" id="8" fillcolor="FF9200" structure="id:7"/>
            <structref name="ColorPallete" id="33" structure="id:28"/>
        </structure>
        <structure name="Header" id="5" extends="id:3">
            <number name="bfType" mustmatch="yes" id="10" type="integer" length="2" endian="big" display="hex">
                <fixedvalues>
                    <fixedvalue value="0x424D"/>
                </fixedvalues>
            </number>
            <number name="bfSize" id="11" type="integer" length="4"/>
            <number name="bfReserved" id="12" type="integer" length="4">
                <fixedvalues>
                    <fixedvalue value="0"/>
                </fixedvalues>
            </number>
            <offset name="bfOffBits" id="14" length="4" references="id:13" follownullreference="yes"/>
        </structure>
        <structure name="Info" id="7" length="this.biSize" extends="id:3" fillcolor="929292">
            <number name="biSize" id="16" fillcolor="FF2600" type="integer" length="4">
                <fixedvalues>
                    <fixedvalue value="40"/>
                    <fixedvalue name="&lt;unnamed&gt;" value="108"/>
                </fixedvalues>
            </number>
            <number name="biWidth" id="17" fillcolor="FEFB00" type="integer" length="4"/>
            <number name="biHeight" id="18" fillcolor="FEFB00" type="integer" length="4" signed="yes"/>
            <number name="biPlanes" id="19" fillcolor="797979" type="integer" length="2">
                <fixedvalues>
                    <fixedvalue value="1"/>
                </fixedvalues>
            </number>
            <number name="biBitCount" id="20" fillcolor="FF89D8" type="integer" length="2"/>
            <number name="biCompression" id="21" fillcolor="72FA78" type="integer" length="4"/>
            <number name="biSizeImage" id="22" fillcolor="FFD478" type="integer" length="4"/>
            <number name="biXPelsPerMeter" id="23" type="integer" length="4"/>
            <number name="biYPelsPerMeter" id="24" fillcolor="C0C0C0" type="integer" length="4"/>
            <number name="biClrUsed" id="25" fillcolor="73FDFF" type="integer" length="4"/>
            <number name="biClrImportant" id="26" fillcolor="7980FF" type="integer" length="4"/>
        </structure>
        <structure name="ColorPallete" id="28" extends="id:3" encoding="ISO_8859-1:1987" endian="little" signed="no">
            <binary name="RBGPallet" id="29" fillcolor="D783FF" repeatmin="0" repeatmax="biClrUsed" length="32" lengthunit="bit"/>
        </structure>
        <structure name="ImageData" id="13" extends="id:3">
            <binary name="ImageData" id="31" fillcolor="D5D5D5" length="remaining"/>
        </structure>
    </grammar>
</ufwb>
WhiteNinja
Posts: 1
Joined: 27 Sep 2022 08:45

Re: Suggestion for HxD:User templates for data inspector

Post by WhiteNinja »

Any further progress on this since January? I couldn't find any of this functionality within HxD so I assume it has not been released yet?
Maël
Site Admin
Posts: 1454
Joined: 12 Mar 2005 14:15

Re: Suggestion for HxD:User templates for data inspector

Post by Maël »

The "pattern languages" might contain useful file structures or sytnax ideas: https://github.com/WerWolv/PatternLanguage https://github.com/WerWolv/ImHex-Patterns https://imhex.werwolv.net/docs/
It is however not a purely declarative language, whereas HSD is by design. Computations are mostly performed by annotations (e.g., for pointers) and otherwise lookups can be implemented using expressions/simple functions, but all should remain stateless and non-imperative, if possible.

Good counter example of what not to do:

Code: Select all

u8 string[while(std::mem::read_unsigned($, 1) != 0xFF)] @ 0x00;
This is too much of an imperative style for implementing a string that terminates with a $FF byte. It's preferable to have an encoding neutral (i.e., not byte based) solution that focuses on the intent, not the implementation method. HSD would use a style like this (possibly less verbose, but more explicit on the intent):

Code: Select all

string<encoding=utf8, termination=$FF>;
A example showing more of the syntax's abilities (the slash / denotes options):

Code: Select all

string8<8, encoding = utf8, termination=none/zero, padding=none/space/zero>; Display=String;
If the ImHex example is an actual byte array, as claimed in its documentation (eventhough the code uses "string" as a keyword, i.e., unclean mixing of types) the following style would still be preferable, i.e., a dynamic array (empty brackets) with a type annotation defining termination conditions:

Code: Select all

UInt8[]<termination=$FF>;
Type annotations will also be done with pointy brackets <> (reminiscent of C++ templates or C#/Delphi generics) and not double square brackets [[]] as this would be less readable when appended to arrays (dynamic or fixed size):

Code: Select all

UInt8[][[termination=$FF]];  // counter example, do NOT use
More generally, the difference between imperative and declarative, includes intentionally omitting print or other imperative language functions, such that the structure definition file can be a model that is used to translate into various targets (for example for logical inference, file editing, or automatically generating file readers/writers in various programming languages or even to implement them in hardware, microcontrollers, FPGA, discrete circuits etc.).

Printing should be done if necessary by annotations (like type annotations), since parsing is declarative, and therefore order of parsing cannot be relied on/is up to the implementation.

https://imhex.werwolv.net/docs/core_lan ... _flow.html
Control flow code should not be interspersed with type declarations. Instead use HSD partition, union or case types (see structsyntax.txt). Same idea: not imperative, but declarative.

So no while or for-loop, just arithmetic expressions that can be evaluated in a stateless way, and only recursive functions that are built-in, so they are well controlled and the intent is perfectly clear. They could be replaced since their interface remains static and their purpose is well defined.

Function definitions in HSD should really be limited to a minimum, and be very generic and reusable, i.e., not a byte search function or similar (that would be built-in and more specialized, such as a null-terminating search function), but work as much as possible on a higher level (so it can be lowered to implementations as necessary). Too lowlevel descriptions don't transfer well to other systems/approaches, which is known from compiler development. (Again think of FPGA or circuit targets and how functions need to be really focused on intent, so they can easily be translated to such targets, and not be so low level and C language focused that they don't translate naturally.)

What are declarative functions? They should allow to make simple computations or execute queries (but not change data, just search, list, order), a bit like SQL or .NET LINQ or SPARQL.

It allows also to evolve the implementation and choose a simple one at first, but optimizing it later, since the intent remains clear.

More importantly, it allows to clearly determine potential deadlocks or infinite loops (which are due to endless recursion in the datastructure itself, not the imperative logic of the code, since it does not exist). So we can introduce various max depth/max recursion constraints (that can be set in the HSD file) that will be respected while parsing, similar to stackoverflow errors in recursive function calls in traditional programs.
By providing a fixed set of built in functions (e.g., to search arrays or strings or lists) tests can be made when searching takes too long, progress windows shown, and the option given to interrupt the search all with useful user messages/message boxes/progress windows.

However it might be useful to further compare the available annotations of ImHex or its data types and see how they differ from HSD.
Maël
Site Admin
Posts: 1454
Joined: 12 Mar 2005 14:15

Re: Suggestion for HxD:User templates for data inspector

Post by Maël »

WhiteNinja wrote: 27 Sep 2022 08:47 Any further progress on this since January? I couldn't find any of this functionality within HxD so I assume it has not been released yet?
Correct. But donations are welcome to help me dedicate more time on this.
Maël
Site Admin
Posts: 1454
Joined: 12 Mar 2005 14:15

Re: Suggestion for HxD:User templates for data inspector

Post by Maël »

CSS is a declarative language that can also use functions to calculate property values, much like HSD can have virtual fields (not virtual as in C++/Delphi, but virtual, as in not having any memory location allocated in the stream the structure template is applied to).

https://cmiller.tech/blog/css-is-declarative/

The functions in CSS could be an inspiration for implementing some flexibility while keeping HSD declarative:
https://web.dev/learn/css/functions/#ma ... xpressions
minmax, fit-content, rgb, var, calc, clamp

This post also highlights a similar idea:
https://stackoverflow.com/questions/267 ... age-is-css
CSS uses various selectors to apply properties to elements within HTML, a markup language
With HSD I can also refer to other parts of HSD through qualified fields, and then define properties for those fields (datatypes, colors, formatting).

Structure templates are similar, they highlight parts of a stream, and give it attributes (structural and additionally visual). Style sheets are all about visual layout, animations, colors, fonts etc. and some functions to describe it in a more dynamical way.

HSD is only focused very little on the visual, it will mix in some visual attributes like colors to it, but they could be externalized to stylesheets, and be described in a generic way (like program wide, user setting wide, and not template specific -- much like the <em> emphasize tags in HTML vs. the explicit bold or italic tags).

So HSD is a bit more like a DTD file, that parses a file into a DOM tree (or alternatively a HTML text representation with standardized formatting), and also mixes in a bit of styling functionality (coloring mostly).

But eventhough HSD is not directly comparable to CSS, it is also declarative, and computing values dynamically could be done in similar ways and all the insights/limitations known in implementing and designing CSS could prove valuable.
Post Reply