Done: DataInspector: provide some Varint decoding for Git

Wishlists for new functionality and features.
Post Reply
PhilipOakley
Posts: 1
Joined: 02 Apr 2019 14:51

Done: DataInspector: provide some Varint decoding for Git

Post by PhilipOakley » 02 Apr 2019 18:46

This follow up an email exchange:

Decoding Git objects and packs is complicated by its use of variable length integer formats. Providing an option to show Git varint decoding would be helpful for inspecting these big binary files.

These varints use the byte's msb to indicate of there is more data within the integer. Some use little endian (LE), others big endian (BE) approaches. The little endian format may hide a few type bits in the first byte, and the big endian may also use the 'no leading zero (+1)' effect for further compression.

There are at least three git formats are in use: the object type/length field (LE), the pack OFS_DELTA size (LE), and 'varint' (BE, '+1').

https://github.com/git/git/blob/master/ ... 1047-L1070
https://github.com/git/git/blob/master/ ... 1181-L1190
https://github.com/git/git/blob/master/varint.c#L4-L18

Philip

Maël
Site Admin
Posts: 1138
Joined: 12 Mar 2005 14:15

Re: DataInspector: provide some Varint decoding for Git

Post by Maël » 05 Apr 2019 12:40

From previous email exchange:
It could be added to the data inspector.

The basic unsigned format seems reasonably simple, the more compact ones less.

I found this article
https://en.wikipedia.org/wiki/Variable-length_quantity which also
describes the Git encoding. However it is slightly confusing and the
linked source code is doing things implicitly (insufficient documentation).
https://github.com/git/git/blob/master/varint.c

Do you have more material on this? (The documentation I found suggests the git varint logic is more complex than your description?)
It looks like a pretty specialized data type, with many variations and a bit hard to implement, given the documentation is code (and it's not clear how stable this code "definition" is).
So I was wondering if providing a plugin interface just for implementing additional datatypes for the data inspector wouldn't be better. Something that allows you to write a DLL in C or any language that supports exporting C like functions in DLLs.
What do you think?

Edit: if you don't mind I could add your reply to the quoted email, as well.

Maël
Site Admin
Posts: 1138
Joined: 12 Mar 2005 14:15

Re: DataInspector: provide some Varint decoding for Git

Post by Maël » 08 Apr 2019 17:47

Let me know if you are interested in this plugin approach, as mentioned in my last mail.

Bo98
Posts: 3
Joined: 24 Apr 2019 01:58

Re: DataInspector: provide some Varint decoding for Git

Post by Bo98 » 24 Apr 2019 02:03

Maël wrote:
08 Apr 2019 17:47
Let me know if you are interested in this plugin approach, as mentioned in my last mail.
I was about to create a thread with the same idea before I saw your post. I may not be the original poster you were asking, but I would myself be very much interested in a plugin based approach. There's been custom types I've come across in various binary files that I would love to able to create decoders for in the inspector.

Maël
Site Admin
Posts: 1138
Joined: 12 Mar 2005 14:15

Re: DataInspector: provide some Varint decoding for Git

Post by Maël » 26 Apr 2019 17:54

Since a couple people have voiced interest in extending the data inspector, I will add a plugin interface.

Maël
Site Admin
Posts: 1138
Joined: 12 Mar 2005 14:15

Re: DataInspector: provide some Varint decoding for Git

Post by Maël » 02 May 2019 10:28


Bo98
Posts: 3
Joined: 24 Apr 2019 01:58

Re: DataInspector: provide some Varint decoding for Git

Post by Bo98 » 05 May 2019 20:28

Maël wrote:
02 May 2019 10:28
https://github.com/maelh/hxd-plugin-framework

Let me know how it goes.
Many thanks for this!

Tried it out but I can't launch the development version of HxD. Seems to depend on "kernel.dll" which does not exist (kernel32.dll is also loaded and works fine).

Maël
Site Admin
Posts: 1138
Joined: 12 Mar 2005 14:15

Re: DataInspector: provide some Varint decoding for Git

Post by Maël » 05 May 2019 21:36

There was a missing text file (which I use during development), which I added to the download now (same link).

I am not sure why you got a kernel.dll error, though. Was it with a compiled plugin or just the pure devel version of HxD? If with a plugin, Delphi or VC++?

Bo98
Posts: 3
Joined: 24 Apr 2019 01:58

Re: DataInspector: provide some Varint decoding for Git

Post by Bo98 » 05 May 2019 21:47

Maël wrote:
05 May 2019 21:36
There was a missing text file (which I use during development), which I added to the download now (same link).

I am not sure why you got a kernel.dll error, though. Was it with a compiled plugin or just the pure devel version of HxD? If with a plugin, Delphi or VC++?
Works fine now with the missing text file, thanks! Turns out the missing DLL is normal.

Just put in a plugin I quickly put together and it works excellently. Thanks again for adding this!

Maël
Site Admin
Posts: 1138
Joined: 12 Mar 2005 14:15

Re: DataInspector: provide some Varint decoding for Git

Post by Maël » 06 May 2019 02:08

Great it works. Can you star the repository so more people see it? Thanks.

Maël
Site Admin
Posts: 1138
Joined: 12 Mar 2005 14:15

Re: Done: DataInspector: provide some Varint decoding for Git

Post by Maël » 06 Jul 2019 19:56

Done in HxD 2.3 as a plugin interface is available now, officially.

Post Reply