Page 1 of 1

Done: Out of memory error

Posted: 21 Oct 2010 16:07
by TorPedoCXC
Hi.
I am trying to edit a 4 GB large text file (replacing). The speed of your program is just great, but it only runs as a 32bit process and stops working with an out of memory error when reaching half of the file (the Taskmanager shows 2 GB RAM utilization).

Is it possible to compile a 64bit version? As I understand searching and replacing needs to store the whole file in memory. That will be a problem with a 32bit program, because AFAIK they can only obtain a maximum of 2GB RAM.

Daniel

Re: Out of memory error

Posted: 10 Nov 2010 19:49
by TorPedoCXC
Will this problem be adressed in the future? It's a bit cumbersome to replace only half of the file, save, and then continue to replace the other half.

Re: Out of memory error

Posted: 13 Nov 2010 04:00
by Maël
It will be addressed. But it's not merely a compile for a 64 bit version, so it's not something in the near future.

Still I wonder why there is such a huge memory consumption. How many bytes get replaced?

Re: Out of memory error

Posted: 25 Nov 2010 18:33
by Maël
Can you tell me the approximate number of replaces and the length of the search and replace pattern?

Re: Out of memory error

Posted: 08 Dec 2010 21:37
by TorPedoCXC
I am replacing the text-string "MinimumYield" with "MY". The program starts replacing and crashes with an OOM-error when reaching the 2 GB limit. I also tested this with a smaller file (3 GB), which also crashed at 3 quarters of the full progress bar. There are a total of about 15.000.000 occurences of this text string in the file. Could it be that the undo-function takes up a lot of RAM? Would it help if you've got access to the file I was working on?

Re: Out of memory error

Posted: 10 Dec 2010 22:21
by Maël
Yeah, the undo data + the data structures managing the edits of the file at many places is using up quite some space + memory fragmentation and it's going to be over the limit.

I'll keep that in mind for optimization, but 64-bit support is probably closer. Also if you would have an example file it would help, though it's not strictly necessary.

Re: Out of memory error

Posted: 24 Jan 2011 22:30
by TorPedoCXC
I tried to edit the file ProsperDataExport.xml which is included in this Zip-File: https://services.prosper.com/DataExport ... rt_xml.zip

I managed to edit it by replacing only half of it, saving, reloading and continuing the replacing. Luckily this was a one time task.

Re: Out of memory error

Posted: 30 Oct 2017 19:02
by Maël
Just in case this is still relevant.
HxD is available in 64 bit now.

https://mh-nexus.de/downloads/HxDSetup.zip