Export Results / Save hangs upon errors in serialization

[purnimab]

When exporting results, an error in saving out the .json model file (in this case, because I had parameterized using functions, for which serialization has not yet been implemented) prevents the exporting of any subsequent files, such as mcmc or plot files. A simple try catch along with a warning to the user about which files were not properly saved should be a sufficient fix.

Traceback (most recent call last):
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\refl1d\experiment.py", line 228, in save_json
    experiment = to_dict(self)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 192, in to_dict
    return Serializer(use_refs=use_refs).to_dict(obj)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 164, in to_dict
    output = self.dataclass_to_dict(obj, include=include, exclude=exclude)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 152, in dataclass_to_dict
    output = dict([(f.name, self.to_dict(getattr(dclass, f.name))) for f in all_fields])
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 152, in <listcomp>
    output = dict([(f.name, self.to_dict(getattr(dclass, f.name))) for f in all_fields])
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 171, in to_dict
    return type(obj)(self.to_dict(v) for v in obj)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 171, in <genexpr>
    return type(obj)(self.to_dict(v) for v in obj)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 164, in to_dict
    output = self.dataclass_to_dict(obj, include=include, exclude=exclude)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 152, in dataclass_to_dict
    output = dict([(f.name, self.to_dict(getattr(dclass, f.name))) for f in all_fields])
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 152, in <listcomp>
    output = dict([(f.name, self.to_dict(getattr(dclass, f.name))) for f in all_fields])
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 171, in to_dict
    return type(obj)(self.to_dict(v) for v in obj)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 171, in <genexpr>
    return type(obj)(self.to_dict(v) for v in obj)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 164, in to_dict
    output = self.dataclass_to_dict(obj, include=include, exclude=exclude)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 152, in dataclass_to_dict
    output = dict([(f.name, self.to_dict(getattr(dclass, f.name))) for f in all_fields])
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 152, in <listcomp>
    output = dict([(f.name, self.to_dict(getattr(dclass, f.name))) for f in all_fields])
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 164, in to_dict
    output = self.dataclass_to_dict(obj, include=include, exclude=exclude)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 152, in dataclass_to_dict
    output = dict([(f.name, self.to_dict(getattr(dclass, f.name))) for f in all_fields])
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 152, in <listcomp>
    output = dict([(f.name, self.to_dict(getattr(dclass, f.name))) for f in all_fields])
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 164, in to_dict
    output = self.dataclass_to_dict(obj, include=include, exclude=exclude)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 152, in dataclass_to_dict
    output = dict([(f.name, self.to_dict(getattr(dclass, f.name))) for f in all_fields])
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 152, in <listcomp>
    output = dict([(f.name, self.to_dict(getattr(dclass, f.name))) for f in all_fields])
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 188, in to_dict
    raise ValueError("obj %s is not serializable" % str(obj))
ValueError: obj <bumps.parameter.UserFunction object at 0x000001C21A939060> is not serializable
C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\refl1d\experiment.py:235: UserWarning:

failed to create json structure for model

[bmaranville]

Not sure if this is specific to webview - if not, will apply fix to master branch too.

[bmaranville]

Huh... There is already a try/catch (and a helpful warning!) in that save_json function. I wonder why it's not working as expected? I'll try making a simple model with a Function to see if I can reproduce.

[purnimab]

Previous error was when I said "Export Results" after a DREAM fit. This traceback is after "Export Results" on a DE fit (no uncertainty state)

future: <Task finished name='Task-74881' coro=<AsyncServer._handle_event_internal() done, defined at C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\socketio\async_server.py:595> exception=ValueError('obj <bumps.parameter.UserFunction object at 0x000001C21B2AE170> is not serializable')>
Traceback (most recent call last):
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\socketio\async_server.py", line 597, in _handle_event_internal
    r = await server._trigger_event(data[0], namespace, sid, *data[1:])
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\socketio\async_server.py", line 631, in _trigger_event
    ret = await handler(*args)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\webview\server\webserver.py", line 134, in with_sid
    return await function(*args, **kwargs)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\webview\server\api.py", line 202, in export_results
    await asyncio.wrap_future(future)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\concurrent\futures\thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\webview\server\api.py", line 223, in _export_results
    serialized = serialize_problem(problem, serializer)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\webview\server\state_hdf5_backed.py", line 42, in serialize_problem
    return json.dumps(to_dict(problem)).encode()
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 192, in to_dict
    return Serializer(use_refs=use_refs).to_dict(obj)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 164, in to_dict
    output = self.dataclass_to_dict(obj, include=include, exclude=exclude)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 152, in dataclass_to_dict
    output = dict([(f.name, self.to_dict(getattr(dclass, f.name))) for f in all_fields])
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 152, in <listcomp>
    output = dict([(f.name, self.to_dict(getattr(dclass, f.name))) for f in all_fields])
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 173, in to_dict
    return list(self.to_dict(v) for v in obj)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 173, in <genexpr>
    return list(self.to_dict(v) for v in obj)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 164, in to_dict
    output = self.dataclass_to_dict(obj, include=include, exclude=exclude)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 152, in dataclass_to_dict
    output = dict([(f.name, self.to_dict(getattr(dclass, f.name))) for f in all_fields])
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 152, in <listcomp>
    output = dict([(f.name, self.to_dict(getattr(dclass, f.name))) for f in all_fields])
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 171, in to_dict
    return type(obj)(self.to_dict(v) for v in obj)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 171, in <genexpr>
    return type(obj)(self.to_dict(v) for v in obj)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 164, in to_dict
    output = self.dataclass_to_dict(obj, include=include, exclude=exclude)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 152, in dataclass_to_dict
    output = dict([(f.name, self.to_dict(getattr(dclass, f.name))) for f in all_fields])
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 152, in <listcomp>
    output = dict([(f.name, self.to_dict(getattr(dclass, f.name))) for f in all_fields])
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 171, in to_dict
    return type(obj)(self.to_dict(v) for v in obj)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 171, in <genexpr>
    return type(obj)(self.to_dict(v) for v in obj)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 164, in to_dict
    output = self.dataclass_to_dict(obj, include=include, exclude=exclude)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 152, in dataclass_to_dict
    output = dict([(f.name, self.to_dict(getattr(dclass, f.name))) for f in all_fields])
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 152, in <listcomp>
    output = dict([(f.name, self.to_dict(getattr(dclass, f.name))) for f in all_fields])
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 164, in to_dict
    output = self.dataclass_to_dict(obj, include=include, exclude=exclude)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 152, in dataclass_to_dict
    output = dict([(f.name, self.to_dict(getattr(dclass, f.name))) for f in all_fields])
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 152, in <listcomp>
    output = dict([(f.name, self.to_dict(getattr(dclass, f.name))) for f in all_fields])
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 164, in to_dict
    output = self.dataclass_to_dict(obj, include=include, exclude=exclude)
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 152, in dataclass_to_dict
    output = dict([(f.name, self.to_dict(getattr(dclass, f.name))) for f in all_fields])
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 152, in <listcomp>
    output = dict([(f.name, self.to_dict(getattr(dclass, f.name))) for f in all_fields])
  File "C:\Users\ppb1\Documents\Refl1d-0.8.15-Webview-Jupyter\env\lib\site-packages\bumps\serialize.py", line 188, in to_dict
    raise ValueError("obj %s is not serializable" % str(obj))
ValueError: obj <bumps.parameter.UserFunction object at 0x000001C21B2AE170> is not serializable

[purnimab]

Version used:
Refl1D-windows-exe-webview-jupyter.tar.gz
201 MB
Dec 5, 2023

Not sure which branch this is compiling from.

[bmaranville]

Ah yes - I see there's no try/catch around the serialization of the fitProblem in bumps.webview.server.api.export_results. I just added one: see bumps/bumps@e942a12

I just rebuilt Refl1D-windows-exe-webview-jupyter.tar.gz, if you want to try it out.