Done: Out of memory error

Wishlists for new functionality and features.
Post Reply
TorPedoCXC
Posts: 4
Joined: 21 Oct 2010 15:42

Done: Out of memory error

Post by TorPedoCXC » 21 Oct 2010 16:07

Hi.
I am trying to edit a 4 GB large text file (replacing). The speed of your program is just great, but it only runs as a 32bit process and stops working with an out of memory error when reaching half of the file (the Taskmanager shows 2 GB RAM utilization).

Is it possible to compile a 64bit version? As I understand searching and replacing needs to store the whole file in memory. That will be a problem with a 32bit program, because AFAIK they can only obtain a maximum of 2GB RAM.

Daniel

TorPedoCXC
Posts: 4
Joined: 21 Oct 2010 15:42

Re: Out of memory error

Post by TorPedoCXC » 10 Nov 2010 19:49

Will this problem be adressed in the future? It's a bit cumbersome to replace only half of the file, save, and then continue to replace the other half.

Maël
Site Admin
Posts: 1148
Joined: 12 Mar 2005 14:15

Re: Out of memory error

Post by Maël » 13 Nov 2010 04:00

It will be addressed. But it's not merely a compile for a 64 bit version, so it's not something in the near future.

Still I wonder why there is such a huge memory consumption. How many bytes get replaced?

Maël
Site Admin
Posts: 1148
Joined: 12 Mar 2005 14:15

Re: Out of memory error

Post by Maël » 25 Nov 2010 18:33

Can you tell me the approximate number of replaces and the length of the search and replace pattern?

TorPedoCXC
Posts: 4
Joined: 21 Oct 2010 15:42

Re: Out of memory error

Post by TorPedoCXC » 08 Dec 2010 21:37

I am replacing the text-string "MinimumYield" with "MY". The program starts replacing and crashes with an OOM-error when reaching the 2 GB limit. I also tested this with a smaller file (3 GB), which also crashed at 3 quarters of the full progress bar. There are a total of about 15.000.000 occurences of this text string in the file. Could it be that the undo-function takes up a lot of RAM? Would it help if you've got access to the file I was working on?

Maël
Site Admin
Posts: 1148
Joined: 12 Mar 2005 14:15

Re: Out of memory error

Post by Maël » 10 Dec 2010 22:21

Yeah, the undo data + the data structures managing the edits of the file at many places is using up quite some space + memory fragmentation and it's going to be over the limit.

I'll keep that in mind for optimization, but 64-bit support is probably closer. Also if you would have an example file it would help, though it's not strictly necessary.

TorPedoCXC
Posts: 4
Joined: 21 Oct 2010 15:42

Re: Out of memory error

Post by TorPedoCXC » 24 Jan 2011 22:30

I tried to edit the file ProsperDataExport.xml which is included in this Zip-File: https://services.prosper.com/DataExport ... rt_xml.zip

I managed to edit it by replacing only half of it, saving, reloading and continuing the replacing. Luckily this was a one time task.

Maël
Site Admin
Posts: 1148
Joined: 12 Mar 2005 14:15

Re: Out of memory error

Post by Maël » 30 Oct 2017 19:02

Just in case this is still relevant.
HxD is available in 64 bit now.

https://mh-nexus.de/downloads/HxDSetup.zip

Post Reply