Es module usage

[nickchomey]

Sobek seems extremely promising, but I have one main question about the es module support.

The Readme says

Also of note is that due to the nature of ESM you need to have an event loop implementation to use it. That is still not provided by Sobek and likely never will be.

Does this mean we can't just swap Sobek for Goja in order to use ES modules in a Go app?

Similarly, does it mean that k6 has implemented a particular event loop to allow for this? Could k6 be used within a Go app, rather than just Sobek, to allow for es module support?

Thanks!

[mstoykov]

Hi @nickchomey,

Does this mean we can't just swap Sobek for Goja in order to use ES modules in a Go app?

Even if you swapped it you will still need an event loop. ECMAScript modules get executed quite differently to how scripts do (original PR to goja for scale). This requires that:

1. You use the API for running modules instead of scripts
2. which in itself is based on promises and will require you to implement some stuff in order to use it

both of those + the fact that ECMAScript modules are asynchronous by nature make it requirement that you an event loop. It can be a bare minimum one, but you need one.

While in the future we might make a project simialr to goja_nodejs, the current purpose of Sobek is to compliment k6 development.

I would like to pull out some code from k6 and make it generally available for Sobek users to just use it, but this is unfortunately not in the current priority list and likely will require some changes. Some of those changes are already kind of underway for different reasons, so 🤷

Could k6 be used within a Go app, rather than just Sobek, to allow for es module support?

You definitely can run js code using k6, and you might be able to reuse parts of it in another go app, but we are unlikely to make changes to support anything and it is possible we will make breaking changes to the parts you use. Also you might want to check the license as it is AGPL 3.0, which depending on your use case might a problem.

You are probably better off looking into the tests files such as

sobek/modules_test.go

Lines 295 to 377 in 2dc9daf

	func runModules(t testing.TB, files map[string]string) func(*Runtime) *Promise {
	type cacheElement struct {
	m ModuleRecord
	err error
	}
	mu := sync.Mutex{}
	cache := make(map[string]cacheElement)
	var hostResolveImportedModule func(referencingScriptOrModule interface{}, specifier string) (ModuleRecord, error)
	hostResolveImportedModule = func(_ interface{}, specifier string) (ModuleRecord, error) {
	mu.Lock()
	defer mu.Unlock()
	k, ok := cache[specifier]
	if ok {
	return k.m, k.err
	}
	src, ok := files[specifier]
	if !ok {
	return nil, fmt.Errorf("can't find %q from files", specifier)
	}
	p, err := ParseModule(specifier, src, hostResolveImportedModule)
	if err != nil {
	cache[specifier] = cacheElement{err: err}
	return nil, err
	}
	cache[specifier] = cacheElement{m: p}
	return p, nil
	}
	linked := make(map[ModuleRecord]error)
	linkMu := new(sync.Mutex)
	link := func(m ModuleRecord) error {
	linkMu.Lock()
	defer linkMu.Unlock()
	if err, ok := linked[m]; ok {
	return err
	}
	err := m.Link()
	linked[m] = err
	return err
	}
	m, err := hostResolveImportedModule(nil, "a.js")
	if err != nil {
	t.Fatalf("got error %s", err)
	}
	p := m.(*SourceTextModuleRecord)
	err = link(p)
	if err != nil {
	t.Fatalf("got error %s", err)
	}
	return func(vm *Runtime) *Promise {
	eventLoopQueue := make(chan func(), 2) // the most basic and likely buggy event loop
	vm.SetImportModuleDynamically(func(referencingScriptOrModule interface{}, specifierValue Value, pcap interface{}) {
	specifier := specifierValue.String()
	eventLoopQueue <- func() {
	ex := vm.runWrapped(func() {
	m, err := hostResolveImportedModule(referencingScriptOrModule, specifier)
	vm.FinishLoadingImportModule(referencingScriptOrModule, specifierValue, pcap, m, err)
	})
	if ex != nil {
	vm.FinishLoadingImportModule(referencingScriptOrModule, specifierValue, pcap, nil, ex)
	}
	}
	})
	var promise *Promise
	eventLoopQueue <- func() { promise = m.Evaluate(vm) }
	outer:
	for {
	select {
	case fn := <-eventLoopQueue:
	fn()
	default:
	break outer
	}
	}
	return promise
	}
	}

and

https://github.com/grafana/sobek/blob/main/modules_integration_test.go

While both of those are fairly crude they actually do implement more or less what is in k6, but without a bunch of k6 specific stuff, or just in a very simplistic way that handles errors badly.

p.s. as noted in the readme the current ESM API evolved while being developed and hte specificaiton changes half way through. The api isn't particularly ergonomic and likely will be redone in the future.

[nickchomey]

Thanks very much!

I'll explore the tests you linked to to see if it could be implemented in another tool (the open source Conduit CDC/ETL tool is what I have in mind, but surely sobek would be of interest to many others who want to get ESM support in goja).

As for running it via k6, thanks for pointing out the agpl license. That's, of course, completely fair for you to use. In this particular context, I'd just be using it internally rather than offering it to others, so shouldn't be affected by the need to open source everything. An alternative is, of course, to just run the k6 binary on its own and interact via its cli commands.

Anyway, thanks again for the great response and for all of your work on sobek and k6! I'll be following the progress of both projects, and hopefully even find a way to contribute someday!