Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

put wandb setup first in _setup_dashboards and arrange cast data pass… #184

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

barikata1984
Copy link
Contributor

…ed to wandb.Table into a double-nested list

@barikata1984
Copy link
Contributor Author

confirmed

@barikata1984
Copy link
Contributor Author

barikata1984 commented Nov 7, 2023

Why:

  1. training curves, such as total_loss and rgb_loss, are not plotted properly on a synced wandb dashboard: only a single point was plotted on it
  2. log_table() for _WandB did not work with the following error:
  File "/home/atsushi/workspace/kaolin-wisp/app/nerf/main_nerf.py", line 128, in <module>
    trainer.train()  # Run in headless mode
  File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/base_trainer.py", line 371, in train
    self.iterate()
  File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/base_trainer.py", line 330, in iterate
    self.end_epoch()    # determines if optimization keeps running
  File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/base_trainer.py", line 308, in end_epoch
    self.validate()
  File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/multiview_trainer.py", line 294, in validate
    self.tracker.log_table("args", record_dict, self.epoch)
  File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/tracker/tracker.py", line 227, in log_table
    dashboard.log_table(caption, data, step)
  File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/tracker/tracker.py", line 453, in log_table
    table = wandb.Table(columns=data.keys(), data=data.values())
  File "/home/atsushi/miniconda3/envs/wisp/lib/python3.9/site-packages/wandb/data_types.py", line 290, in __init__
    self._init_from_list(data, columns, optional, dtype)
  File "/home/atsushi/miniconda3/envs/wisp/lib/python3.9/site-packages/wandb/data_types.py", line 309, in _init_from_list
    assert isinstance(data, list), "data argument expects a `list` object"
AssertionError: data argument expects a `list` object

What:

  1. Configure a wandb dashboard first if it is enabled because wandb.init() should be called earlier than instantiating SummaryWrite according to the official description of wandb (link)
  2. Cast columns argument into a list and data argument into a 2D list because the columns argument should be iterable and, both the data argument and its element should be iterable as well based on the source (link)

P.S. PR #183 was my fault. Sorry about that. Please ignore it

@barikata1984 barikata1984 reopened this Nov 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant