Massive memory leak in Julia due to tiny memory leak in C #55794
Labels
GC
Garbage collector
Comments
|
Copying the Tested both on an M2 and on a Linux x86_64 machine. Could reproduce it on Linux, but the increase in RSS seems to be much less severe on Mac.
using Printf
using NCDatasets
function create_dummpy_netcdf()
dset = NCDataset("output.nc", "c", format=:netcdf4)
close(dset)
end
function output_dummy_netcdf()
dset = NCDataset("output.nc", "a")
zeros(Float64, 1024, 1024)
close(dset)
end
function main()
create_dummpy_netcdf()
for i in 1:1_000
output_dummy_netcdf()
@info Printf.@sprintf "Live bytes: %9.3f MiB\n" Base.gc_live_bytes()/2^20
@info Printf.@sprintf "Max. RSS: %9.3f MiB\n" Sys.maxrss()/2^20
end
end
main()
|
|
Wondering if this could be related to something specific to (E.g. maybe |
|
You mean something like #42566? |
|
Just to confirm this leak is not coming from the pool allocator... Added this patch to v1.10.4: diff --git a/src/gc-pages.c b/src/gc-pages.c
index 682e76611f..be2ec0462a 100644
--- a/src/gc-pages.c
+++ b/src/gc-pages.c
@@ -9,6 +9,11 @@
extern "C" {
#endif
+JL_DLLEXPORT uint64_t jl_get_pg_size(void)
+{
+ return GC_PAGE_SZ;
+}
+
// Try to allocate memory in chunks to permit faster allocation
// and improve memory locality of the pools
#ifdef _P64
@@ -19,6 +24,12 @@ extern "C" {
#define MIN_BLOCK_PG_ALLOC (1) // 16 KB
static int block_pg_cnt = DEFAULT_BLOCK_PG_ALLOC;
+static _Atomic(size_t) current_pg_count = 0;
+
+JL_DLLEXPORT uint64_t jl_current_pg_count(void)
+{
+ return (uint64_t)jl_atomic_load(¤t_pg_count);
+}
void jl_gc_init_page(void)
{
@@ -148,6 +159,7 @@ exit:
SetLastError(last_error);
#endif
errno = last_errno;
+ jl_atomic_fetch_add(¤t_pg_count, 1);
return meta;
}
@@ -188,6 +200,7 @@ void jl_gc_free_page(jl_gc_pagemeta_t *pg) JL_NOTSAFEPOINT
madvise(p, decommit_size, MADV_DONTNEED);
#endif
msan_unpoison(p, decommit_size);
+ jl_atomic_fetch_add(¤t_pg_count, -1);
}
#ifdef __cplusplusand then ran this MWE: using Printf
using NCDatasets
function create_dummpy_netcdf()
dset = NCDataset("output.nc", "c", format=:netcdf4)
close(dset)
end
function output_dummy_netcdf()
dset = NCDataset("output.nc", "a")
zeros(Float64, 1024, 1024)
close(dset)
end
function main()
create_dummpy_netcdf()
for i in 1:1_000
output_dummy_netcdf()
@info Printf.@sprintf "Live bytes: %9.3f MiB\n" Base.gc_live_bytes()/2^20
@info Printf.@sprintf "Max. RSS: %9.3f MiB\n" Sys.maxrss()/2^20
@info Printf.@sprintf "Current page count: %d\n" @ccall jl_current_pg_count()::Cint
end
end
main()Current page count was fairly stable: |
Possibly. |
|
I get: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
It is copied from my post on Julia discourse.
The latest version (v4.9.2) of
netCDF-C(a C library for outputting data in the network Common Data Form.) is known to have memory leak. The leaked memory is not more than several MBs (see this link). However, when I useNCDatasets.jl, which is a wrapper in Julia fornetcdf-C, I find that the memory leak reaches several GBs in the demo below:The total memory usage reaches 8.2 GiB in the end, which is roughly 1000 times the allocation of
zeros(Float64, 1024, 1024). Removingzeros(Float64, 1024, 1024)inoutput_dummy_netcdfmakes the total memory usage limited to 400 MiB. It seems to me that the memory leak in the C code is "amplified" by allocations in Julia.This issue has been posted as Alexander-Barth/NCDatasets.jl#266 (with version info). @Alexander-Barth finds that this bug can be produced simply with
ccall-ing functions inlibnetcdf.so. I am bringing the discussion here because I think it is related to how Julia manages memory.I have also tried using
Valgrindto profile the heap memory usage. However, when running withValgrind, this bug cannot be reproduced. If you know a better tool for profiling memory in Julia, please let me know.The text was updated successfully, but these errors were encountered: