Memory issue using Islands/Archipelagos

[davide-roversi]

Hello everyone, I am posting this here and also on Gitter (link: https://app.gitter.im/#/room/#pagmo2_Lobby:gitter.im)

I am using Pygmo for the single-objective optimization of a problem of size 8 (i.e. 8 decision variables, and 1 fitness value per-evolution). The fitness function takes about 1~2 seconds to run on single-core. To speed things up, I implemented both multi-processing via bfe and multi-core using the island/archipelago setup.

Bfe works like a charm, significantly speeds up the process, and does not use a lot of RAM. The problem is, for single-objective optimization, it only works on GACO, PSO, PSO_GEN amd CMAES, all of which do not perform well with the problem I am trying to solve. SADE is a far better option, but does not support “set_bfe()”.

So I switched to the coarse-grained archipelago setup. I have 20 CPU cores (virtual + physical), and 32GB RAM at my disposal. Using SADE with a population of just 7 individuals per-island, times 20 islands, ram gets full at 99.9% after a few evolutions. If I increase the islands sizes, I sometimes get out-of-memory errors. Looking at the resource monitor as the optimization progresses, I see that the RAM usage continuously rises, and never gets reset throughout the evolutions.

My questions are two:

1. Is the archipelagos memory issue fixable / am I doing something wrong? Or is a RAM upgrade the only way out?
2. Is there absolutely no way to run a bfe on SADE? Or any other algorithm that does not have the set_bfe() function built-in?

Other info:

• OS: Windows 11 Home
• CPU: 13th Gen Intel(R) Core(TM) i7-13700H 2.40 GHz
• Pygmo version: 2.19.5

[bluescarni]

@davide-roversi what is the memory consumption when you are using the BFE on a single population and how many individuals do you have in the population?

[bluescarni]

@davide-roversi The default BFE behaviour is to use multiprocessing as well, just like the archipelago, thus also in the case of BFE your problem is being copied around to remote processes.

The difference I think is that in the archipelago, in addition to the copies being made and sent around for multiprocessing, you also have one problem copy per island, so the overall memory consumption is increased.

Is there any internal problem state that grows over time as you perform more function evaluations?

[bluescarni]

Finally, your cluster suggestion. To run using the ipyparallel_island UDI, do I need to have a physical cluster myself, or would it run on some online resource?

Conceivably you could rent some compute from AWS or some other cloud provider and set up your cluster there, but it may start to be a bit on the expensive side. Having access to an institutional cluster would certainly be cheaper :)

[davide-roversi]

I do have a few arrays that are increased throughout the evolutions, such as an array containing all best fitnesses, and one containing all best individuals. However, let's say I have 250 max iterations and a decision variables vector of length 8, those to arrays I mention would end up being 250-by-1 and 250-by-8. It would surprise me if that was the cause for memory filling up. Moreover, once again, using BFE on GACO or PSO does not lead me to memory issues, despite above mentioned evolutions-increasing arrays being present.

Additionally, I have gone back on pygmo API to see if I was overlooking something. I noticed that, when creating the archipelago, the default udi is set to pygmo.mp_island(use_pool=True) . If I turn it to False, the memory usage stays constant at around 20 GB, even if I run a population of 100 individuals per-island, with 20 cores (note that with the setting turned to True, I get memory issues with as low as 7 individuals per-island). Now, reading the docs, what I understand is that use_pool=False generates a new process for each evolution task, whereas use_pool=True generates one global pool at the start and keeps using those processes all along the evolutions. Is that correct? I still struggle to understand why this would (apparently) solve the memory issue. Also, I am not quite sure I grasp what consequences having the setting True or False has on the island interactions within the archipelago and, therefore, on the algorithm convergence.

[bluescarni]

@davide-roversi

use_pool=True dispatches evolution tasks to a global pool of processes which is automatically created on first use. This pool is a global variable which stays alive for the duration of the program and which is shared by all archipelagos.

use_pool=False instead creates a new process every time an evolution starts in an island, and the process is terminated after the evolution is concluded.

In theory using the pool can be more efficient because you are avoiding creating ex-novo a process every time an island evolves (which can have a large overhead), but it also means that the resources used during evolution are not necessarily cleaned up when the evolution concludes.

It looks like there is some memory leakage, but it is hard to say with certainty what is going on exactly without seeing the code or having at least a reproducer to investigate. It could be just Python holding onto references to your problems in the pool processes instead of garbage collecting them, but again, I would need to take a look at the code in order to investigate.

[davide-roversi]

I understand, thank you for the insights. The code is quite extensive, so I might try to reduce it to a minimum working example before sending it over. For now, having run some simulations in the past week with use_pool = False did not lead to anything crashing, and (at least from the impression I get) did not slow anything down too much.