bug: Schema not found the second time a spark table is accessed

[christophediprima]

What happened?

I am trying to insert into two different iceberg tables using spark connect. But as soon as I access a table using get_schema,table or insert I won't be able to do it a second time.

My code looks like this:

import ibis
from   pyspark.sql import SparkSession

session = SparkSession.builder.getOrCreate()
con = ibis.pyspark.connect(session)

new_dataframe = Dataframe creation here

con.insert(
  "my_table",
  new_dataframe,
  database="my_catalog.my_database",
)

new_dataframe_2 = Dataframe creation here

con.insert(
  "my_other_table",
  new_dataframe_2,
  database="my_catalog.my_database",
)

First insert will work but not the second one. If first insert is commented, second one will work. Same thing will happen regardless of the get_schema,table or insert method.

What version of ibis are you using?

ibis-framework[pyspark]==10.0.0.dev490

What backend(s) are you using, if any?

PySpark

Relevant log output

pyspark.errors.exceptions.connect.AnalysisException: [SCHEMA_NOT_FOUND] The schema `my_catalog`. cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a catalog, verify the current_schema() output, or qualify the name with the correct catalog.
To tolerate the error on drop use DROP SCHEMA IF EXISTS.

Code of Conduct

• I agree to follow this project's Code of Conduct