Preserving ordering in sol2 tables and converting to ordered JSON

[gsisinna]

First of all, thank you for this library and the effort that was put into it!

Description
I'm using the sol2 library along with the nlohmann JSON library in my project. I need to preserve the ordering of data in sol tables and then convert them to ordered JSON using the nlohmann JSON library. However, it seems that the default behavior of sol2 doesn't preserve ordering when converting sol tables to JSON.

Expected Behavior:
When converting sol tables to JSON, the ordering of elements should be preserved, ensuring that the JSON output maintains the same order as the original sol table.

Steps to Reproduce:

1. Create a sol table and populate it with data, ensuring that the data has a specific order:

sol::state lua;
sol::table myTable = lua.create_table();
myTable["first"] = 1;
myTable["second"] = 2;
myTable["third"] = 3;

2. Convert the sol table to JSON using sol2:

nlohmann::ordered_json jsonOutput = tableToJson(myTable);

nlohmann::ordered_json tableToJson(sol::table table) {
    nlohmann::ordered_json json;
    table.for_each([&json](sol::object const& key, sol::object const& value) {
        if (key.is<std::string>()) {
            std::string keyStr = key.as<std::string>();
            if (value.is<int>()) {
                json[keyStr] = value.as<int>();
            } else if (value.is<double>()) {
                json[keyStr] = value.as<double>();
            } else if (value.is<std::string>()) {
                json[keyStr] = value.as<std::string>();
            } else if (value.is<bool>()) {
                json[keyStr] = value.as<bool>();
            } else if (value.is<sol::table>()) {
                json[keyStr] = tableToJson(value.as<sol::table>());
            } else {
                // Handle other types if needed
            }
        }
    });
    return json;
}

3. Compare the ordering of elements in the original sol table with the generated JSON output:

std::cout << jsonOutput.dump() << std::endl;

Output:

JSON Output:
{
    "first": 1,
    "third": 3,
    "second": 2
}

Additional Information:
I'm currently using the nlohmann ordered JSON library (nlohmann::ordered_json) to handle JSON objects. It would be helpful to have guidance on how to correctly store data in sol tables while preserving ordering and then convert them to ordered JSON using sol2.

Environment:
Operating System: WSL2 (Debian)
Compiler: Clang 14.0.6 x86_64
sol2 Version: 3.3.0
nlohmann JSON Version: 3.11.2

Full Code:

#include <iostream>
#include <sol/sol.hpp>
#include <nlohmann/json.hpp>

using Json = nlohmann::ordered_json;

// Function to convert a sol table to JSON
Json tableToJson(sol::table table) {
  Json json;
  table.for_each([&json](sol::object const & key, sol::object const & value) {
    if (key.is<std::string>()) {
      std::string keyStr = key.as<std::string>();
      if (value.is<int>()) {
        json[keyStr] = value.as<int>();
      } else if (value.is<double>()) {
        json[keyStr] = value.as<double>();
      } else if (value.is<std::string>()) {
        json[keyStr] = value.as<std::string>();
      } else if (value.is<bool>()) {
        json[keyStr] = value.as<bool>();
      } else if (value.is<sol::table>()) {
        json[keyStr] = tableToJson(value.as<sol::table>());
      } else {
        // Handle other types if needed
      }
    }
  });
  return json;
}

int main() {
  // Create a Lua state
  sol::state lua;

  // Create a Lua table
  sol::table myTable = lua.create_table();
  myTable["first"] = 1;
  myTable["second"] = 2;
  myTable["third"] = 3;

  // Convert the Lua table to JSON
  Json jsonOutput = tableToJson(myTable);

  // Inspect the JSON output
  std::cout << "JSON Output:\n" << jsonOutput.dump(4) << std::endl;

  return 0;
}

Thanks a lot for the support!

[Rochet2]

Sol tables are just lua tables. And lua tables are not guaranteed to be ordered, so the assumption in step 1 that the code creates the table with specific order is incorrect. As I understand, tables are basically hash maps.

From table constructor docs

The order of the assignments in a constructor is undefined.

The for_each documentation tells that it does not guarantee order.

The iterators you use to walk through a sol::table are NOT guaranteed to iterate in numeric order

Furthermore, in lua you can only A) iterate through a table by specifying the keys to iterate yourself and accessing that specific key in a loop, B) by iterating from 1 up to the first absent index (ipairs), C) by looping through all keys in the table with pairs which does not guarantee order (pairs calls next). for_each in sol is the same as pairs so it will never have guaranteed order even if the data would be ordered, and from the documentation of next which pairs uses:

The order in which the indices are enumerated is not specified, even for numeric indices.

To collect all of this together, what you can do is create an ordered list containing the data instead and iterate through it using ipairs (or in sol, just get length and iterate from 1 to length):

// myTable = {
// { first = 1 },
// { second = 2 },
// { third = 3 },
// }
sol::table myTable = lua.create_table();
myTable[1] = lua.create_table_with(1, "first", 2, 1);
myTable[2] = lua.create_table_with(1, "second", 2, 2);
myTable[3] = lua.create_table_with(1, "third", 2, 3);

for (int i = 1, count = myTable.size(); i <= count; ++i) {
    lua["print"](i, myTable[i], myTable[i][1], myTable[i][2]);
}

This is a lot less efficient and inconvenient overall but it guarantees order. There are of course other options, like creating your own data structure, or exposing some kind of C++ data structure to lua depending on what you need. On that vein, if you want an easy way for ordering while having a regular map, you could track the insertions and deletions from the table in a separate list. However, seems that for_each in sol does not check the metamethod __pairs 🤔. We could expose the order table and iterate through it instead.

-- Define the table with metatable
function createTrackedTable()
    local data = {}
    local order = {}

    local mt = {
        __newindex = function(t, k, v)
            if v == nil then
                -- Remove key from data and order if the value is nil
                rawset(data, k, nil)
                for i, key in ipairs(order) do
                    if key == k then
                        table.remove(order, i)
                        break
                    end
                end
            else
                -- Add new key-value pair to data and key to order
                if rawget(data, k)  == nil then
                    table.insert(order, k)
                end
                rawset(data, k, v)
            end
        end,

        __index = data,

        __pairs = function(t)
            local i = 0
            local function iter(data, oldk)
                i = i + 1
                local k = order[i]
                if k then
                    return k, data[k]
                end
            end
            return iter, data, nil
        end,
    
        order = order,
    }

    return setmetatable({}, mt)
end

-- Usage
local trackedTable = createTrackedTable()

-- Insert elements
trackedTable["first"] = 1
trackedTable["second"] = 2
trackedTable["third"] = 3

-- Print elements in their insertion order
for k, v in pairs(trackedTable) do
    print(k, v)
end

-- Remove element
trackedTable["second"] = nil
-- add it back in
trackedTable["second"] = 2

-- Print elements in their insertion order
for k, v in pairs(trackedTable) do
    print(k, v)
end

    // Create a tracked table
    sol::function createTrackedTable = lua["createTrackedTable"];
    sol::table trackedTable = createTrackedTable();

    // Insert elements into the tracked table
    trackedTable["first"] = 1;
    trackedTable["second"] = 2;
    trackedTable["third"] = 3;

    // Access the order table
    sol::table mt = lua["getmetatable"](trackedTable);
    sol::table order = mt["order"];

    // Iterate over the order table to print elements
    for (int i = 1, count = order.size(); i <= count; ++i) {
        sol::object key = order[i];
        lua["print"](key, trackedTable[key]);
    }

In general only lists, stacks, queues and similar data structures have insertion order preserved. Maps and sets order the data on its own, based on the keys or values usually, so they may not suit your needs if you require insertion order being preserved. I would say that JavaScript is quite unique in its implementation of a map style structure with insertion order when iterated (ref). Not sure I have seen anything similar elsewhere yet. 👀

[gsisinna]

@Rochet2 thank you very much for the technical explanation and support. I have continued to read the documentation of both Lua and Sol2 and will try to implement one of your proposed methods to achieve separate sorting. Basically, even if I do not respect the insertion order I need the keys to always be in a certain order before conversion to Json.
In general, I am well aware of the pattern to be followed for the final result in terms of key sorting, but I am trying to get something that scales well for future software modifications and customization of results 👍🏻