_id) alias to improve the developer experience ([#1609](https://github.com/sourcenetwork/defradb/issues/1609)).
-To get a full outline of the changes, we invite you to review the official changelog below. This release does include a Breaking Change to existing v0.5.x databases. If you need help migrating an existing deployment, reach out at [hello@source.network](mailto:hello@source.network) or join our Discord at https://discord.source.network/.
+To get a full outline of the changes, we invite you to review the official changelog below. This release does include a Breaking Change to existing v0.5.x databases. If you need help migrating an existing deployment, reach out at [hello@source.network](mailto:hello@source.network) or join our Discord at https://discord.gg/w7jYQVJ/.
### Features
@@ -329,7 +406,7 @@ There many new features in this release, but most importantly, this is the first
To get a full outline of the changes, we invite you to review the official changelog below. Some highlights are the first iteration of our schema update system, allowing developers to add new fields to schemas using our JSON Patch based DDL, a new DAG based delete system which will persist "soft-delete" ops into the CRDT Merkle DAG, and a early prototype for our collection level peer-to-peer synchronization.
-This release does include a Breaking Change to existing v0.4.x databases. If you need help migrating an existing deployment, reach out at [hello@source.network](mailto:hello@source.network) or join our Discord at https://discord.source.network/.
+This release does include a Breaking Change to existing v0.4.x databases. If you need help migrating an existing deployment, reach out at [hello@source.network](mailto:hello@source.network) or join our Discord at https://discord.gg/w7jYQVJ/.
### Features
@@ -469,7 +546,7 @@ DefraDB v0.4 is a major pre-production release. Until the stable version 1.0 is
There are various new features in this release - some of which are breaking - and we invite you to review the official changelog below. Some highlights are persistence of replicators, DateTime scalars, TLS support, and GQL subscriptions.
-This release does include a Breaking Change to existing v0.3.x databases. If you need help migrating an existing deployment, reach out at [hello@source.network](mailto:hello@source.network) or join our Discord at https://discord.source.network/.
+This release does include a Breaking Change to existing v0.3.x databases. If you need help migrating an existing deployment, reach out at [hello@source.network](mailto:hello@source.network) or join our Discord at https://discord.gg/w7jYQVJ/.
### Features
@@ -642,7 +719,7 @@ DefraDB v0.3 is a major pre-production release. Until the stable version 1.0 is
There are *several* new features in this release, and we invite you to review the official changelog below. Some highlights are various new features for Grouping & Aggregation for the query system, like top-level aggregation and group filtering. Moreover, a brand new Query Explain system was added to introspect the execution plans created by DefraDB. Lastly we introduced a revamped CLI configuration system.
-This release does include a Breaking Change to existing v0.2.x databases. If you need help migrating an existing deployment, reach out at [hello@source.network](mailto:hello@source.network) or join our Discord at https://discord.source.network/.
+This release does include a Breaking Change to existing v0.2.x databases. If you need help migrating an existing deployment, reach out at [hello@source.network](mailto:hello@source.network) or join our Discord at https://discord.gg/w7jYQVJ/.
### Features
@@ -877,7 +954,7 @@ This release is jam-packed with new features and a small number of breaking chan
Much more than just that has been added to ensure we're building reliable software expected of any database, such as expanded test & benchmark suites, automated bug detection, performance gains, and more.
-This release does include a Breaking Change to existing v0.1 databases regarding the internal data model, which affects the "Content Identifiers" we use to generate DocKeys and VersionIDs. If you need help migrating an existing deployment, reach out at hello@source.network or join our Discord at https://discord.source.network.
+This release does include a Breaking Change to existing v0.1 databases regarding the internal data model, which affects the "Content Identifiers" we use to generate DocKeys and VersionIDs. If you need help migrating an existing deployment, reach out at hello@source.network or join our Discord at https://discord.gg/w7jYQVJ.
### Features
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index c7cfb9b590..c590fa54f4 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -5,7 +5,7 @@ This document will guide you through the process of contributing to the project.
All contributions are appreciated, whether it's identifying problems, highlighting missing features, or contributing to the codebase in simple or complex ways.
-You are encouraged to join the [Source Network Discord](discord.source.network) to discuss ideas, ask questions, and find inspiration for future developments.
+You are encouraged to join the [Source Network Discord](discord.gg/w7jYQVJ) to discuss ideas, ask questions, and find inspiration for future developments.
## Getting started
To get started, clone the repository, build, and run it:
diff --git a/Makefile b/Makefile
index 5dddc2872e..0ddde9790f 100644
--- a/Makefile
+++ b/Makefile
@@ -217,11 +217,26 @@ test\:cli:
test\:names:
gotestsum --format testname -- $(DEFAULT_TEST_DIRECTORIES) $(TEST_FLAGS)
+.PHONY: test\:lens
+test\:lens:
+ @$(MAKE) deps:lens
+ gotestsum --format testname -- ./$(LENS_TEST_DIRECTORY)/... $(TEST_FLAGS)
+
+.PHONY: test\:lens-quick
+test\:lens-quick:
+ @$(MAKE) deps:lens
+ gotestsum --format testname -- ./$(LENS_TEST_DIRECTORY)/...
+
.PHONY: test\:all
test\:all:
@$(MAKE) test:names
@$(MAKE) test:lens
+.PHONY: test\:all-quick
+test\:all-quick:
+ @$(MAKE) test:quick
+ @$(MAKE) test:lens-quick
+
.PHONY: test\:verbose
test\:verbose:
gotestsum --format standard-verbose -- $(DEFAULT_TEST_DIRECTORIES) $(TEST_FLAGS)
@@ -246,11 +261,6 @@ test\:bench-short:
test\:scripts:
@$(MAKE) -C ./tools/scripts/ test
-.PHONY: test\:lens
-test\:lens:
- @$(MAKE) deps:lens
- gotestsum --format testname -- ./$(LENS_TEST_DIRECTORY)/... $(TEST_FLAGS)
-
.PHONY: test\:coverage
test\:coverage:
@$(MAKE) deps:lens
diff --git a/README.md b/README.md
index d77dd18f48..0e5997b902 100644
--- a/README.md
+++ b/README.md
@@ -1,7 +1,7 @@
[](https://goreportcard.com/report/github.com/sourcenetwork/defradb)
[](https://codecov.io/gh/sourcenetwork/defradb)
-[](https://discord.source.network/)
+[](https://discord.gg/w7jYQVJ)
[](https://twitter.com/sourcenetwrk)
@@ -100,8 +100,8 @@ Submit a `mutation` request to create a document of the `User` type:
```shell
defradb client query '
mutation {
- create_User(data: "{\"age\": 31, \"verified\": true, \"points\": 90, \"name\": \"Bob\"}") {
- _key
+ create_User(input: {age: 31, verified: true, points: 90, name: "Bob"}) {
+ _docID
}
}
'
@@ -113,13 +113,13 @@ Expected response:
{
"data": [
{
- "_key": "bae-91171025-ed21-50e3-b0dc-e31bccdfa1ab",
+ "_docID": "bae-91171025-ed21-50e3-b0dc-e31bccdfa1ab",
}
]
}
```
-`_key` is the document's key, a unique identifier of the document, determined by its schema and initial data.
+`_docID` is the document's unique identifier determined by its schema and initial data.
## Query documents
@@ -129,7 +129,7 @@ Once you have populated your node with data, you can query it:
defradb client query '
query {
User {
- _key
+ _docID
age
name
points
@@ -138,7 +138,7 @@ defradb client query '
'
```
-This query obtains *all* users and returns their fields `_key, age, name, points`. GraphQL queries only return the exact fields requested.
+This query obtains *all* users and returns their fields `_docID, age, name, points`. GraphQL queries only return the exact fields requested.
You can further filter results with the `filter` argument.
@@ -146,7 +146,7 @@ You can further filter results with the `filter` argument.
defradb client query '
query {
User(filter: {points: {_ge: 50}}) {
- _key
+ _docID
age
name
points
@@ -166,7 +166,7 @@ To get the most recent commit in the MerkleDAG for the document identified as `b
```shell
defradb client query '
query {
- latestCommits(dockey: "bae-91171025-ed21-50e3-b0dc-e31bccdfa1ab") {
+ latestCommits(docID: "bae-91171025-ed21-50e3-b0dc-e31bccdfa1ab") {
cid
delta
height
@@ -334,7 +334,7 @@ defradb client schema add '
Start (or continue running from above) *nodeB*, that will be receiving updates:
```shell
-defradb start --rootdir ~/.defradb-nodeB --url localhost:9182 --p2paddr /ip4/0.0.0.0/tcp/9172 --tcpaddr /ip4/0.0.0.0/tcp/9162
+defradb start --rootdir ~/.defradb-nodeB --url localhost:9182 --p2paddr /ip4/0.0.0.0/tcp/9172
```
Here we *do not* specify `--peers` as we will manually define a replicator after startup via the `rpc` client command.
@@ -432,7 +432,7 @@ defradb client backup import path/to/backup.json
## Community
-Discuss on [Discord](https://discord.source.network/) or [Github Discussions](https://github.com/sourcenetwork/defradb/discussions). The Source project is on [Twitter](https://twitter.com/sourcenetwrk).
+Discuss on [Discord](https://discord.gg/w7jYQVJ) or [Github Discussions](https://github.com/sourcenetwork/defradb/discussions). The Source project is on [Twitter](https://twitter.com/sourcenetwrk).
## Licensing
diff --git a/cli/cli.go b/cli/cli.go
index 0f93b69633..2ee882afce 100644
--- a/cli/cli.go
+++ b/cli/cli.go
@@ -63,6 +63,11 @@ func NewDefraCommand(cfg *config.Config) *cobra.Command {
schema_migrate,
)
+ view := MakeViewCommand()
+ view.AddCommand(
+ MakeViewAddCommand(),
+ )
+
index := MakeIndexCommand()
index.AddCommand(
MakeIndexCreateCommand(),
@@ -86,7 +91,7 @@ func NewDefraCommand(cfg *config.Config) *cobra.Command {
collection := MakeCollectionCommand(cfg)
collection.AddCommand(
MakeCollectionGetCommand(),
- MakeCollectionKeysCommand(),
+ MakeCollectionListDocIDsCommand(),
MakeCollectionDeleteCommand(),
MakeCollectionUpdateCommand(),
MakeCollectionCreateCommand(),
@@ -98,6 +103,7 @@ func NewDefraCommand(cfg *config.Config) *cobra.Command {
MakeDumpCommand(),
MakeRequestCommand(),
schema,
+ view,
index,
p2p,
backup,
diff --git a/cli/collection_create.go b/cli/collection_create.go
index 4dca9be33a..82e1e5db09 100644
--- a/cli/collection_create.go
+++ b/cli/collection_create.go
@@ -11,7 +11,6 @@
package cli
import (
- "encoding/json"
"io"
"os"
@@ -66,35 +65,19 @@ Example: create from stdin
return ErrNoDocOrFile
}
- var docMap any
- if err := json.Unmarshal(docData, &docMap); err != nil {
- return err
- }
-
- switch t := docMap.(type) {
- case map[string]any:
- doc, err := client.NewDocFromMap(t)
+ if client.IsJSONArray(docData) {
+ docs, err := client.NewDocsFromJSON(docData, col.Schema())
if err != nil {
return err
}
- return col.Create(cmd.Context(), doc)
- case []any:
- docs := make([]*client.Document, len(t))
- for i, v := range t {
- docMap, ok := v.(map[string]any)
- if !ok {
- return ErrInvalidDocument
- }
- doc, err := client.NewDocFromMap(docMap)
- if err != nil {
- return err
- }
- docs[i] = doc
- }
return col.CreateMany(cmd.Context(), docs)
- default:
- return ErrInvalidDocument
}
+
+ doc, err := client.NewDocFromJSON(docData, col.Schema())
+ if err != nil {
+ return err
+ }
+ return col.Create(cmd.Context(), doc)
},
}
cmd.Flags().StringVarP(&file, "file", "f", "", "File containing document(s)")
diff --git a/cli/collection_delete.go b/cli/collection_delete.go
index 85539d5eb3..dcd7c9d872 100644
--- a/cli/collection_delete.go
+++ b/cli/collection_delete.go
@@ -17,15 +17,15 @@ import (
)
func MakeCollectionDeleteCommand() *cobra.Command {
- var keys []string
+ var argDocIDs []string
var filter string
var cmd = &cobra.Command{
- Use: "delete [--filter --key ]",
- Short: "Delete documents by key or filter.",
- Long: `Delete documents by key or filter and lists the number of documents deleted.
+ Use: "delete [--filter --docID ]",
+ Short: "Delete documents by docID or filter.",
+ Long: `Delete documents by docID or filter and lists the number of documents deleted.
-Example: delete by key(s)
- defradb client collection delete --name User --key bae-123,bae-456
+Example: delete by docID(s)
+ defradb client collection delete --name User --docID bae-123,bae-456
Example: delete by filter
defradb client collection delete --name User --filter '{ "_gte": { "points": 100 } }'
@@ -37,26 +37,26 @@ Example: delete by filter
}
switch {
- case len(keys) == 1:
- docKey, err := client.NewDocKeyFromString(keys[0])
+ case len(argDocIDs) == 1:
+ docID, err := client.NewDocIDFromString(argDocIDs[0])
if err != nil {
return err
}
- res, err := col.DeleteWithKey(cmd.Context(), docKey)
+ res, err := col.DeleteWithDocID(cmd.Context(), docID)
if err != nil {
return err
}
return writeJSON(cmd, res)
- case len(keys) > 1:
- docKeys := make([]client.DocKey, len(keys))
- for i, v := range keys {
- docKey, err := client.NewDocKeyFromString(v)
+ case len(argDocIDs) > 1:
+ docIDs := make([]client.DocID, len(argDocIDs))
+ for i, v := range argDocIDs {
+ docID, err := client.NewDocIDFromString(v)
if err != nil {
return err
}
- docKeys[i] = docKey
+ docIDs[i] = docID
}
- res, err := col.DeleteWithKeys(cmd.Context(), docKeys)
+ res, err := col.DeleteWithDocIDs(cmd.Context(), docIDs)
if err != nil {
return err
}
@@ -68,11 +68,11 @@ Example: delete by filter
}
return writeJSON(cmd, res)
default:
- return ErrNoDocKeyOrFilter
+ return ErrNoDocIDOrFilter
}
},
}
- cmd.Flags().StringSliceVar(&keys, "key", nil, "Document key")
+ cmd.Flags().StringSliceVar(&argDocIDs, "docID", nil, "Document ID")
cmd.Flags().StringVar(&filter, "filter", "", "Document filter")
return cmd
}
diff --git a/cli/collection_get.go b/cli/collection_get.go
index d908bbdb7a..d753e0a8db 100644
--- a/cli/collection_get.go
+++ b/cli/collection_get.go
@@ -19,7 +19,7 @@ import (
func MakeCollectionGetCommand() *cobra.Command {
var showDeleted bool
var cmd = &cobra.Command{
- Use: "get [--show-deleted]",
+ Use: "get [--show-deleted]",
Short: "View document fields.",
Long: `View document fields.
@@ -33,11 +33,11 @@ Example:
return cmd.Usage()
}
- docKey, err := client.NewDocKeyFromString(args[0])
+ docID, err := client.NewDocIDFromString(args[0])
if err != nil {
return err
}
- doc, err := col.Get(cmd.Context(), docKey, showDeleted)
+ doc, err := col.Get(cmd.Context(), docID, showDeleted)
if err != nil {
return err
}
diff --git a/cli/collection_keys.go b/cli/collection_list_doc_ids.go
similarity index 65%
rename from cli/collection_keys.go
rename to cli/collection_list_doc_ids.go
index a453c16a86..d7009cb300 100644
--- a/cli/collection_keys.go
+++ b/cli/collection_list_doc_ids.go
@@ -16,14 +16,14 @@ import (
"github.com/sourcenetwork/defradb/http"
)
-func MakeCollectionKeysCommand() *cobra.Command {
+func MakeCollectionListDocIDsCommand() *cobra.Command {
var cmd = &cobra.Command{
- Use: "keys",
- Short: "List all document keys.",
- Long: `List all document keys.
+ Use: "docIDs",
+ Short: "List all document IDs (docIDs).",
+ Long: `List all document IDs (docIDs).
Example:
- defradb client collection keys --name User
+ defradb client collection docIDs --name User
`,
RunE: func(cmd *cobra.Command, args []string) error {
col, ok := tryGetCollectionContext(cmd)
@@ -31,16 +31,16 @@ Example:
return cmd.Usage()
}
- docCh, err := col.GetAllDocKeys(cmd.Context())
+ docCh, err := col.GetAllDocIDs(cmd.Context())
if err != nil {
return err
}
- for docKey := range docCh {
- results := &http.DocKeyResult{
- Key: docKey.Key.String(),
+ for docIDResult := range docCh {
+ results := &http.DocIDResult{
+ DocID: docIDResult.ID.String(),
}
- if docKey.Err != nil {
- results.Error = docKey.Err.Error()
+ if docIDResult.Err != nil {
+ results.Error = docIDResult.Err.Error()
}
if err := writeJSON(cmd, results); err != nil {
return err
diff --git a/cli/collection_update.go b/cli/collection_update.go
index 317a2e8119..9fd2deed3f 100644
--- a/cli/collection_update.go
+++ b/cli/collection_update.go
@@ -17,24 +17,24 @@ import (
)
func MakeCollectionUpdateCommand() *cobra.Command {
- var keys []string
+ var argDocIDs []string
var filter string
var updater string
var cmd = &cobra.Command{
- Use: "update [--filter --key --updater ] ",
- Short: "Update documents by key or filter.",
- Long: `Update documents by key or filter.
+ Use: "update [--filter --docID --updater ] ",
+ Short: "Update documents by docID or filter.",
+ Long: `Update documents by docID or filter.
Example: update from string
- defradb client collection update --name User --key bae-123 '{ "name": "Bob" }'
+ defradb client collection update --name User --docID bae-123 '{ "name": "Bob" }'
Example: update by filter
defradb client collection update --name User \
--filter '{ "_gte": { "points": 100 } }' --updater '{ "verified": true }'
-Example: update by keys
+Example: update by docIDs
defradb client collection update --name User \
- --key bae-123,bae-456 --updater '{ "verified": true }'
+ --docID bae-123,bae-456 --updater '{ "verified": true }'
`,
Args: cobra.RangeArgs(0, 1),
RunE: func(cmd *cobra.Command, args []string) error {
@@ -44,26 +44,26 @@ Example: update by keys
}
switch {
- case len(keys) == 1 && updater != "":
- docKey, err := client.NewDocKeyFromString(keys[0])
+ case len(argDocIDs) == 1 && updater != "":
+ docID, err := client.NewDocIDFromString(argDocIDs[0])
if err != nil {
return err
}
- res, err := col.UpdateWithKey(cmd.Context(), docKey, updater)
+ res, err := col.UpdateWithDocID(cmd.Context(), docID, updater)
if err != nil {
return err
}
return writeJSON(cmd, res)
- case len(keys) > 1 && updater != "":
- docKeys := make([]client.DocKey, len(keys))
- for i, v := range keys {
- docKey, err := client.NewDocKeyFromString(v)
+ case len(argDocIDs) > 1 && updater != "":
+ docIDs := make([]client.DocID, len(argDocIDs))
+ for i, v := range argDocIDs {
+ docID, err := client.NewDocIDFromString(v)
if err != nil {
return err
}
- docKeys[i] = docKey
+ docIDs[i] = docID
}
- res, err := col.UpdateWithKeys(cmd.Context(), docKeys, updater)
+ res, err := col.UpdateWithDocIDs(cmd.Context(), docIDs, updater)
if err != nil {
return err
}
@@ -74,12 +74,12 @@ Example: update by keys
return err
}
return writeJSON(cmd, res)
- case len(keys) == 1 && len(args) == 1:
- docKey, err := client.NewDocKeyFromString(keys[0])
+ case len(argDocIDs) == 1 && len(args) == 1:
+ docID, err := client.NewDocIDFromString(argDocIDs[0])
if err != nil {
return err
}
- doc, err := col.Get(cmd.Context(), docKey, true)
+ doc, err := col.Get(cmd.Context(), docID, true)
if err != nil {
return err
}
@@ -88,11 +88,11 @@ Example: update by keys
}
return col.Update(cmd.Context(), doc)
default:
- return ErrNoDocKeyOrFilter
+ return ErrNoDocIDOrFilter
}
},
}
- cmd.Flags().StringSliceVar(&keys, "key", nil, "Document key")
+ cmd.Flags().StringSliceVar(&argDocIDs, "docID", nil, "Document ID")
cmd.Flags().StringVar(&filter, "filter", "", "Document filter")
cmd.Flags().StringVar(&updater, "updater", "", "Document updater")
return cmd
diff --git a/cli/errors.go b/cli/errors.go
index 937bdd2c9d..bb124bc7f9 100644
--- a/cli/errors.go
+++ b/cli/errors.go
@@ -22,11 +22,12 @@ const (
var (
ErrNoDocOrFile = errors.New("document or file must be defined")
ErrInvalidDocument = errors.New("invalid document")
- ErrNoDocKeyOrFilter = errors.New("document key or filter must be defined")
+ ErrNoDocIDOrFilter = errors.New("docID or filter must be defined")
ErrInvalidExportFormat = errors.New("invalid export format")
ErrNoLensConfig = errors.New("lens config cannot be empty")
ErrInvalidLensConfig = errors.New("invalid lens configuration")
ErrSchemaVersionNotOfSchema = errors.New(errSchemaVersionNotOfSchema)
+ ErrViewAddMissingArgs = errors.New("please provide a base query and output SDL for this view")
)
func NewErrInvalidLensConfig(inner error) error {
diff --git a/cli/index_create.go b/cli/index_create.go
index 42866267fc..099eb7e7a6 100644
--- a/cli/index_create.go
+++ b/cli/index_create.go
@@ -21,12 +21,14 @@ func MakeIndexCreateCommand() *cobra.Command {
var collectionArg string
var nameArg string
var fieldsArg []string
+ var uniqueArg bool
var cmd = &cobra.Command{
- Use: "create -c --collection --fields [-n --name ]",
+ Use: "create -c --collection --fields [-n --name ] [--unique]",
Short: "Creates a secondary index on a collection's field(s)",
Long: `Creates a secondary index on a collection's field(s).
The --name flag is optional. If not provided, a name will be generated automatically.
+The --unique flag is optional. If provided, the index will be unique.
Example: create an index for 'Users' collection on 'name' field:
defradb client index create --collection Users --fields name
@@ -44,6 +46,7 @@ Example: create a named index for 'Users' collection on 'name' field:
desc := client.IndexDescription{
Name: nameArg,
Fields: fields,
+ Unique: uniqueArg,
}
col, err := store.GetCollectionByName(cmd.Context(), collectionArg)
if err != nil {
@@ -62,6 +65,7 @@ Example: create a named index for 'Users' collection on 'name' field:
cmd.Flags().StringVarP(&collectionArg, "collection", "c", "", "Collection name")
cmd.Flags().StringVarP(&nameArg, "name", "n", "", "Index name")
cmd.Flags().StringSliceVar(&fieldsArg, "fields", []string{}, "Fields to index")
+ cmd.Flags().BoolVarP(&uniqueArg, "unique", "u", false, "Make the index unique")
return cmd
}
diff --git a/cli/utils.go b/cli/utils.go
index 8c1a40dc1f..0f3fa0e565 100644
--- a/cli/utils.go
+++ b/cli/utils.go
@@ -99,9 +99,6 @@ func setStoreContext(cmd *cobra.Command, cfg *config.Config) error {
// loadConfig loads the rootDir containing the configuration file,
// otherwise warn about it and load a default configuration.
func loadConfig(cfg *config.Config) error {
- if err := cfg.LoadRootDirFromFlagOrDefault(); err != nil {
- return err
- }
return cfg.LoadWithRootdir(cfg.ConfigFileExists())
}
diff --git a/cli/version_test.go b/cli/version_test.go
index 4f62f3659b..fdc6aba6e6 100644
--- a/cli/version_test.go
+++ b/cli/version_test.go
@@ -42,7 +42,7 @@ func TestVersionFull(t *testing.T) {
assert.NoError(t, err)
t.Log(buf.String())
assert.Contains(t, buf.String(), "* HTTP API")
- assert.Contains(t, buf.String(), "* DocKey versions")
+ assert.Contains(t, buf.String(), "* DocID versions")
assert.Contains(t, buf.String(), "* P2P multicodec")
}
@@ -59,11 +59,11 @@ func TestVersionJSON(t *testing.T) {
{
"release": "",
"commit": "",
- "commitdate": "",
+ "commitDate": "",
"go": "",
- "httpapi": "v0",
- "dockeyversions": "1",
- "netprotocol": "/defra/0.0.1"
+ "httpAPI": "v0",
+ "docIDVersions": "1",
+ "netProtocol": "/defra/0.0.1"
}`)
}
@@ -80,10 +80,10 @@ func TestVersionJSONFull(t *testing.T) {
{
"release": "",
"commit": "",
- "commitdate": "",
+ "commitDate": "",
"go": "",
- "httpapi": "v0",
- "dockeyversions": "1",
- "netprotocol": "/defra/0.0.1"
+ "httpAPI": "v0",
+ "docIDVersions": "1",
+ "netProtocol": "/defra/0.0.1"
}`)
}
diff --git a/cli/view.go b/cli/view.go
new file mode 100644
index 0000000000..aaf9b58a4d
--- /dev/null
+++ b/cli/view.go
@@ -0,0 +1,25 @@
+// Copyright 2023 Democratized Data Foundation
+//
+// Use of this software is governed by the Business Source License
+// included in the file licenses/BSL.txt.
+//
+// As of the Change Date specified in that file, in accordance with
+// the Business Source License, use of this software will be governed
+// by the Apache License, Version 2.0, included in the file
+// licenses/APL.txt.
+
+package cli
+
+import (
+ "github.com/spf13/cobra"
+)
+
+func MakeViewCommand() *cobra.Command {
+ var cmd = &cobra.Command{
+ Use: "view",
+ Short: "Manage views within a running DefraDB instance",
+ Long: "Manage (add) views withing a running DefraDB instance",
+ }
+
+ return cmd
+}
diff --git a/cli/view_add.go b/cli/view_add.go
new file mode 100644
index 0000000000..46779fb784
--- /dev/null
+++ b/cli/view_add.go
@@ -0,0 +1,43 @@
+// Copyright 2023 Democratized Data Foundation
+//
+// Use of this software is governed by the Business Source License
+// included in the file licenses/BSL.txt.
+//
+// As of the Change Date specified in that file, in accordance with
+// the Business Source License, use of this software will be governed
+// by the Apache License, Version 2.0, included in the file
+// licenses/APL.txt.
+
+package cli
+
+import "github.com/spf13/cobra"
+
+func MakeViewAddCommand() *cobra.Command {
+ var cmd = &cobra.Command{
+ Use: "add [query] [sdl]",
+ Short: "Add new view",
+ Long: `Add new database view.
+
+Example: add from an argument string:
+ defradb client view add 'Foo { name, ...}' 'type Foo { ... }'
+
+Learn more about the DefraDB GraphQL Schema Language on https://docs.source.network.`,
+ RunE: func(cmd *cobra.Command, args []string) error {
+ store := mustGetStoreContext(cmd)
+
+ if len(args) != 2 {
+ return ErrViewAddMissingArgs
+ }
+
+ query := args[0]
+ sdl := args[1]
+
+ defs, err := store.AddView(cmd.Context(), query, sdl)
+ if err != nil {
+ return err
+ }
+ return writeJSON(cmd, defs)
+ },
+ }
+ return cmd
+}
diff --git a/client/collection.go b/client/collection.go
index 3e6bb64cc4..3a42871c62 100644
--- a/client/collection.go
+++ b/client/collection.go
@@ -33,26 +33,32 @@ type CollectionDefinition struct {
type Collection interface {
// Name returns the name of this collection.
Name() string
+
// ID returns the ID of this Collection.
ID() uint32
+
// SchemaRoot returns the Root of the Schema used to define this Collection.
SchemaRoot() string
// Definition contains the metadata defining what a Collection is.
Definition() CollectionDefinition
+
// Schema returns the SchemaDescription used to define this Collection.
Schema() SchemaDescription
+
// Description returns the CollectionDescription of this Collection.
Description() CollectionDescription
// Create a new document.
//
- // Will verify the DocKey/CID to ensure that the new document is correctly formatted.
+ // Will verify the DocID/CID to ensure that the new document is correctly formatted.
Create(context.Context, *Document) error
+
// CreateMany new documents.
//
- // Will verify the DocKeys/CIDs to ensure that the new documents are correctly formatted.
+ // Will verify the DocIDs/CIDs to ensure that the new documents are correctly formatted.
CreateMany(context.Context, []*Document) error
+
// Update an existing document with the new values.
//
// Any field that needs to be removed or cleared should call doc.Clear(field) before.
@@ -60,93 +66,102 @@ type Collection interface {
//
// Will return a ErrDocumentNotFound error if the given document is not found.
Update(context.Context, *Document) error
+
// Save the given document in the database.
//
- // If a document exists with the given DocKey it will update it. Otherwise a new document
+ // If a document exists with the given DocID it will update it. Otherwise a new document
// will be created.
Save(context.Context, *Document) error
- // Delete will attempt to delete a document by key.
+
+ // Delete will attempt to delete a document by DocID.
//
// Will return true if a deletion is successful, and return false along with an error
// if it cannot. If the document doesn't exist, then it will return false and a ErrDocumentNotFound error.
- // This operation will hard-delete all state relating to the given DocKey. This includes data, block, and head storage.
- Delete(context.Context, DocKey) (bool, error)
- // Exists checks if a given document exists with supplied DocKey.
+ // This operation will hard-delete all state relating to the given DocID. This includes data, block, and head storage.
+ Delete(context.Context, DocID) (bool, error)
+
+ // Exists checks if a given document exists with supplied DocID.
//
// Will return true if a matching document exists, otherwise will return false.
- Exists(context.Context, DocKey) (bool, error)
+ Exists(context.Context, DocID) (bool, error)
// UpdateWith updates a target document using the given updater type.
//
- // Target can be a Filter statement, a single docKey, a single document,
- // an array of docKeys, or an array of documents.
+ // Target can be a Filter statement, a single DocID, a single document,
+ // an array of DocIDs, or an array of documents.
// It is recommended to use the respective typed versions of Update
- // (e.g. UpdateWithFilter or UpdateWithKey) over this function if you can.
+ // (e.g. UpdateWithFilter or UpdateWithDocID) over this function if you can.
//
// Returns an ErrInvalidUpdateTarget error if the target type is not supported.
// Returns an ErrInvalidUpdater error if the updater type is not supported.
UpdateWith(ctx context.Context, target any, updater string) (*UpdateResult, error)
+
// UpdateWithFilter updates using a filter to target documents for update.
//
// The provided updater must be a string Patch, string Merge Patch, a parsed Patch, or parsed Merge Patch
// else an ErrInvalidUpdater will be returned.
UpdateWithFilter(ctx context.Context, filter any, updater string) (*UpdateResult, error)
- // UpdateWithKey updates using a DocKey to target a single document for update.
+
+ // UpdateWithDocID updates using a DocID to target a single document for update.
//
// The provided updater must be a string Patch, string Merge Patch, a parsed Patch, or parsed Merge Patch
// else an ErrInvalidUpdater will be returned.
//
- // Returns an ErrDocumentNotFound if a document matching the given DocKey is not found.
- UpdateWithKey(ctx context.Context, key DocKey, updater string) (*UpdateResult, error)
- // UpdateWithKeys updates documents matching the given DocKeys.
+ // Returns an ErrDocumentNotFound if a document matching the given DocID is not found.
+ UpdateWithDocID(ctx context.Context, docID DocID, updater string) (*UpdateResult, error)
+
+ // UpdateWithDocIDs updates documents matching the given DocIDs.
//
// The provided updater must be a string Patch, string Merge Patch, a parsed Patch, or parsed Merge Patch
// else an ErrInvalidUpdater will be returned.
//
- // Returns an ErrDocumentNotFound if a document is not found for any given DocKey.
- UpdateWithKeys(context.Context, []DocKey, string) (*UpdateResult, error)
+ // Returns an ErrDocumentNotFound if a document is not found for any given DocID.
+ UpdateWithDocIDs(context.Context, []DocID, string) (*UpdateResult, error)
// DeleteWith deletes a target document.
//
- // Target can be a Filter statement, a single docKey, a single document, an array of docKeys,
+ // Target can be a Filter statement, a single DocID, a single document, an array of DocIDs,
// or an array of documents. It is recommended to use the respective typed versions of Delete
- // (e.g. DeleteWithFilter or DeleteWithKey) over this function if you can.
- // This operation will soft-delete documents related to the given DocKey and update the composite block
+ // (e.g. DeleteWithFilter or DeleteWithDocID) over this function if you can.
+ // This operation will soft-delete documents related to the given DocID and update the composite block
// with a status of `Deleted`.
//
// Returns an ErrInvalidDeleteTarget if the target type is not supported.
DeleteWith(ctx context.Context, target any) (*DeleteResult, error)
+
// DeleteWithFilter deletes documents matching the given filter.
//
// This operation will soft-delete documents related to the given filter and update the composite block
// with a status of `Deleted`.
DeleteWithFilter(ctx context.Context, filter any) (*DeleteResult, error)
- // DeleteWithKey deletes using a DocKey to target a single document for delete.
+
+ // DeleteWithDocID deletes using a DocID to target a single document for delete.
//
- // This operation will soft-delete documents related to the given DocKey and update the composite block
+ // This operation will soft-delete documents related to the given DocID and update the composite block
// with a status of `Deleted`.
//
- // Returns an ErrDocumentNotFound if a document matching the given DocKey is not found.
- DeleteWithKey(context.Context, DocKey) (*DeleteResult, error)
- // DeleteWithKeys deletes documents matching the given DocKeys.
+ // Returns an ErrDocumentNotFound if a document matching the given DocID is not found.
+ DeleteWithDocID(context.Context, DocID) (*DeleteResult, error)
+
+ // DeleteWithDocIDs deletes documents matching the given DocIDs.
//
- // This operation will soft-delete documents related to the given DocKeys and update the composite block
+ // This operation will soft-delete documents related to the given DocIDs and update the composite block
// with a status of `Deleted`.
//
- // Returns an ErrDocumentNotFound if a document is not found for any given DocKey.
- DeleteWithKeys(context.Context, []DocKey) (*DeleteResult, error)
+ // Returns an ErrDocumentNotFound if a document is not found for any given DocID.
+ DeleteWithDocIDs(context.Context, []DocID) (*DeleteResult, error)
- // Get returns the document with the given DocKey.
+ // Get returns the document with the given DocID.
//
- // Returns an ErrDocumentNotFound if a document matching the given DocKey is not found.
- Get(ctx context.Context, key DocKey, showDeleted bool) (*Document, error)
+ // Returns an ErrDocumentNotFound if a document matching the given DocID is not found.
+ Get(ctx context.Context, docID DocID, showDeleted bool) (*Document, error)
// WithTxn returns a new instance of the collection, with a transaction
// handle instead of a raw DB handle.
WithTxn(datastore.Txn) Collection
- // GetAllDocKeys returns all the document keys that exist in the collection.
- GetAllDocKeys(ctx context.Context) (<-chan DocKeysResult, error)
+ // GetAllDocIDs returns all the document IDs that exist in the collection.
+ GetAllDocIDs(ctx context.Context) (<-chan DocIDResult, error)
// CreateIndex creates a new index on the collection.
// `IndexDescription` contains the description of the index to be created.
@@ -162,11 +177,11 @@ type Collection interface {
GetIndexes(ctx context.Context) ([]IndexDescription, error)
}
-// DocKeysResult wraps the result of an attempt at a DocKey retrieval operation.
-type DocKeysResult struct {
- // If a DocKey was successfully retrieved, this will be that key.
- Key DocKey
- // If an error was generated whilst attempting to retrieve the DocKey, this will be the error.
+// DocIDResult wraps the result of an attempt at a DocID retrieval operation.
+type DocIDResult struct {
+ // If a DocID was successfully retrieved, this will be that DocID.
+ ID DocID
+ // If an error was generated whilst attempting to retrieve the DocID, this will be the error.
Err error
}
@@ -174,16 +189,16 @@ type DocKeysResult struct {
type UpdateResult struct {
// Count contains the number of documents updated by the update call.
Count int64
- // DocKeys contains the DocKeys of all the documents updated by the update call.
- DocKeys []string
+ // DocIDs contains the DocIDs of all the documents updated by the update call.
+ DocIDs []string
}
// DeleteResult wraps the result of an delete call.
type DeleteResult struct {
// Count contains the number of documents deleted by the delete call.
Count int64
- // DocKeys contains the DocKeys of all the documents deleted by the delete call.
- DocKeys []string
+ // DocIDs contains the DocIDs of all the documents deleted by the delete call.
+ DocIDs []string
}
// P2PCollection is the gRPC response representation of a P2P collection topic
diff --git a/client/ctype.go b/client/ctype.go
index 96ec19eaf2..7c194c73bf 100644
--- a/client/ctype.go
+++ b/client/ctype.go
@@ -22,4 +22,46 @@ const (
LWW_REGISTER
OBJECT
COMPOSITE
+ PN_COUNTER
)
+
+// IsSupportedFieldCType returns true if the type is supported as a document field type.
+func (t CType) IsSupportedFieldCType() bool {
+ switch t {
+ case NONE_CRDT, LWW_REGISTER, PN_COUNTER:
+ return true
+ default:
+ return false
+ }
+}
+
+// IsCompatibleWith returns true if the CRDT is compatible with the field kind
+func (t CType) IsCompatibleWith(kind FieldKind) bool {
+ switch t {
+ case PN_COUNTER:
+ if kind == FieldKind_INT || kind == FieldKind_FLOAT {
+ return true
+ }
+ return false
+ default:
+ return true
+ }
+}
+
+// String returns the string representation of the CRDT.
+func (t CType) String() string {
+ switch t {
+ case NONE_CRDT:
+ return "none"
+ case LWW_REGISTER:
+ return "lww"
+ case OBJECT:
+ return "object"
+ case COMPOSITE:
+ return "composite"
+ case PN_COUNTER:
+ return "pncounter"
+ default:
+ return "unknown"
+ }
+}
diff --git a/client/db.go b/client/db.go
index b1b63f29d6..240d2d5dfc 100644
--- a/client/db.go
+++ b/client/db.go
@@ -36,7 +36,7 @@ type DB interface {
// NewConcurrentTxn returns a new transaction on the root store that may be managed externally.
//
- // It may be used with other functions in the client package. It is threadsafe and mutliple threads/Go routines
+ // It may be used with other functions in the client package. It is threadsafe and multiple threads/Go routines
// can safely operate on it concurrently.
NewConcurrentTxn(context.Context, bool) (datastore.Txn, error)
@@ -123,14 +123,42 @@ type Store interface {
// It will return an error if the provided schema version ID does not exist.
SetDefaultSchemaVersion(context.Context, string) error
- // SetMigration sets the migration for the given source-destination schema version IDs. Is equivilent to
+ // AddView creates a new Defra View.
+ //
+ // It takes a GQL query string, for example:
+ //
+ // Author {
+ // name
+ // books {
+ // name
+ // }
+ // }
+ //
+ //
+ // A GQL SDL that matches its output type must also be provided. There can only be one `type` declaration,
+ // any nested objects must be declared as embedded/schema-only types using the `interface` keyword.
+ // Relations must only be specified on the parent side of the relationship. For example:
+ //
+ // type AuthorView {
+ // name: String
+ // books: [BookView]
+ // }
+ // interface BookView {
+ // name: String
+ // }
+ //
+ // It will return the collection definitions of the types defined in the SDL if successful, otherwise an error
+ // will be returned. This function does not execute the given query.
+ AddView(ctx context.Context, gqlQuery string, sdl string) ([]CollectionDefinition, error)
+
+ // SetMigration sets the migration for the given source-destination schema version IDs. Is equivalent to
// calling `LensRegistry().SetMigration(ctx, cfg)`.
//
// There may only be one migration per schema version id. If another migration was registered it will be
// overwritten by this migration.
//
// Neither of the schema version IDs specified in the configuration need to exist at the time of calling.
- // This is to allow the migration of documents of schema versions unknown to the local node recieved by the
+ // This is to allow the migration of documents of schema versions unknown to the local node received by the
// P2P system.
//
// Migrations will only run if there is a complete path from the document schema version to the latest local
diff --git a/client/descriptions.go b/client/descriptions.go
index efe470114d..7a4ec0ba7e 100644
--- a/client/descriptions.go
+++ b/client/descriptions.go
@@ -12,6 +12,8 @@ package client
import (
"fmt"
+
+ "github.com/sourcenetwork/defradb/client/request"
)
// CollectionDescription describes a Collection and all its associated metadata.
@@ -30,6 +32,13 @@ type CollectionDescription struct {
// The ID of the schema version that this collection is at.
SchemaVersionID string
+ // BaseQuery contains the base query of this view, if this collection is a view.
+ //
+ // The query will be saved, and then may be accessed by other actors on demand. Actor defined
+ // aggregates, filters and other logic (such as LensVM transforms) will execute on top of this
+ // base query before the result is returned to the actor.
+ BaseQuery *request.Select
+
// Indexes contains the secondary indexes that this Collection has.
Indexes []IndexDescription
}
@@ -118,7 +127,7 @@ type FieldKind uint8
func (f FieldKind) String() string {
switch f {
- case FieldKind_DocKey:
+ case FieldKind_DocID:
return "ID"
case FieldKind_BOOL:
return "Boolean"
@@ -146,6 +155,8 @@ func (f FieldKind) String() string {
return "[String]"
case FieldKind_STRING_ARRAY:
return "[String!]"
+ case FieldKind_BLOB:
+ return "Blob"
default:
return fmt.Sprint(uint8(f))
}
@@ -154,7 +165,7 @@ func (f FieldKind) String() string {
// Note: These values are serialized and persisted in the database, avoid modifying existing values.
const (
FieldKind_None FieldKind = 0
- FieldKind_DocKey FieldKind = 1
+ FieldKind_DocID FieldKind = 1
FieldKind_BOOL FieldKind = 2
FieldKind_BOOL_ARRAY FieldKind = 3
FieldKind_INT FieldKind = 4
@@ -162,11 +173,11 @@ const (
FieldKind_FLOAT FieldKind = 6
FieldKind_FLOAT_ARRAY FieldKind = 7
_ FieldKind = 8 // safe to repurpose (was never used)
- _ FieldKind = 9 // safe to repurpose (previoulsy old field)
+ _ FieldKind = 9 // safe to repurpose (previously old field)
FieldKind_DATETIME FieldKind = 10
FieldKind_STRING FieldKind = 11
FieldKind_STRING_ARRAY FieldKind = 12
- _ FieldKind = 13 // safe to repurpose (was never used)
+ FieldKind_BLOB FieldKind = 13
_ FieldKind = 14 // safe to repurpose (was never used)
_ FieldKind = 15 // safe to repurpose (was never used)
@@ -186,11 +197,11 @@ const (
// their enum values.
//
// It is currently used to by [db.PatchSchema] to allow string representations of
-// [FieldKind] to be provided instead of their raw int values. This useage may expand
+// [FieldKind] to be provided instead of their raw int values. This usage may expand
// in the future. They currently roughly correspond to the GQL field types, but this
-// equality is not guarenteed.
+// equality is not guaranteed.
var FieldKindStringToEnumMapping = map[string]FieldKind{
- "ID": FieldKind_DocKey,
+ "ID": FieldKind_DocID,
"Boolean": FieldKind_BOOL,
"[Boolean]": FieldKind_NILLABLE_BOOL_ARRAY,
"[Boolean!]": FieldKind_BOOL_ARRAY,
@@ -204,6 +215,7 @@ var FieldKindStringToEnumMapping = map[string]FieldKind{
"String": FieldKind_STRING,
"[String]": FieldKind_NILLABLE_STRING_ARRAY,
"[String!]": FieldKind_STRING_ARRAY,
+ "Blob": FieldKind_BLOB,
}
// RelationType describes the type of relation between two types.
@@ -238,7 +250,7 @@ type FieldDescription struct {
// ID contains the internal ID of this field.
//
// Whilst this ID will typically match the field's index within the Schema's Fields
- // slice, there is no guarentee that they will be the same.
+ // slice, there is no guarantee that they will be the same.
//
// It is immutable.
ID FieldID
@@ -268,7 +280,7 @@ type FieldDescription struct {
// IsInternal returns true if this field is internally generated.
func (f FieldDescription) IsInternal() bool {
- return (f.Name == "_key") || f.RelationType&Relation_Type_INTERNAL_ID != 0
+ return (f.Name == request.DocIDFieldName) || f.RelationType&Relation_Type_INTERNAL_ID != 0
}
// IsObject returns true if this field is an object type.
@@ -287,6 +299,25 @@ func (f FieldDescription) IsPrimaryRelation() bool {
return f.RelationType > 0 && f.RelationType&Relation_Type_Primary != 0
}
+// IsRelation returns true if this field is a relation.
+func (f FieldDescription) IsRelation() bool {
+ return f.RelationType > 0
+}
+
+// IsArray returns true if this field is an array type which includes inline arrays as well
+// as relation arrays.
+func (f FieldDescription) IsArray() bool {
+ return f.Kind == FieldKind_BOOL_ARRAY ||
+ f.Kind == FieldKind_INT_ARRAY ||
+ f.Kind == FieldKind_FLOAT_ARRAY ||
+ f.Kind == FieldKind_STRING_ARRAY ||
+ f.Kind == FieldKind_FOREIGN_OBJECT_ARRAY ||
+ f.Kind == FieldKind_NILLABLE_BOOL_ARRAY ||
+ f.Kind == FieldKind_NILLABLE_INT_ARRAY ||
+ f.Kind == FieldKind_NILLABLE_FLOAT_ARRAY ||
+ f.Kind == FieldKind_NILLABLE_STRING_ARRAY
+}
+
// IsSet returns true if the target relation type is set.
func (m RelationType) IsSet(target RelationType) bool {
return m&target > 0
diff --git a/client/dockey.go b/client/doc_id.go
similarity index 50%
rename from client/dockey.go
rename to client/doc_id.go
index 421820d341..601a6ed791 100644
--- a/client/dockey.go
+++ b/client/doc_id.go
@@ -20,14 +20,14 @@ import (
mbase "github.com/multiformats/go-multibase"
)
-// DocKey versions.
+// DocID versions.
const (
- DocKeyV0 = 0x01
+ DocIDV0 = 0x01
)
-// ValidDocKeyVersions is a map of DocKey versions and their current validity.
-var ValidDocKeyVersions = map[uint16]bool{
- DocKeyV0: true,
+// ValidDocIDVersions is a map of DocID versions and their current validity.
+var ValidDocIDVersions = map[uint16]bool{
+ DocIDV0: true,
}
var (
@@ -35,69 +35,69 @@ var (
SDNNamespaceV0 = uuid.Must(uuid.FromString("c94acbfa-dd53-40d0-97f3-29ce16c333fc"))
)
-// DocKey is the root key identifier for documents in DefraDB.
-type DocKey struct {
+// DocID is the root identifier for documents in DefraDB.
+type DocID struct {
version uint16
uuid uuid.UUID
cid cid.Cid
}
-// NewDocKeyV0 creates a new dockey identified by the root data CID,peerID, and namespaced by the versionNS.
-func NewDocKeyV0(dataCID cid.Cid) DocKey {
- return DocKey{
- version: DocKeyV0,
+// NewDocIDV0 creates a new DocID identified by the root data CID, peerID, and namespaced by the versionNS.
+func NewDocIDV0(dataCID cid.Cid) DocID {
+ return DocID{
+ version: DocIDV0,
uuid: uuid.NewV5(SDNNamespaceV0, dataCID.String()),
cid: dataCID,
}
}
-// NewDocKeyFromString creates a new DocKey from a string.
-func NewDocKeyFromString(key string) (DocKey, error) {
- parts := strings.SplitN(key, "-", 2)
+// NewDocIDFromString creates a new DocID from a string.
+func NewDocIDFromString(docID string) (DocID, error) {
+ parts := strings.SplitN(docID, "-", 2)
if len(parts) != 2 {
- return DocKey{}, ErrMalformedDocKey
+ return DocID{}, ErrMalformedDocID
}
versionStr := parts[0]
_, data, err := mbase.Decode(versionStr)
if err != nil {
- return DocKey{}, err
+ return DocID{}, err
}
buf := bytes.NewBuffer(data)
version, err := binary.ReadUvarint(buf)
if err != nil {
- return DocKey{}, err
+ return DocID{}, err
}
- if _, ok := ValidDocKeyVersions[uint16(version)]; !ok {
- return DocKey{}, ErrInvalidDocKeyVersion
+ if _, ok := ValidDocIDVersions[uint16(version)]; !ok {
+ return DocID{}, ErrInvalidDocIDVersion
}
uuid, err := uuid.FromString(parts[1])
if err != nil {
- return DocKey{}, err
+ return DocID{}, err
}
- return DocKey{
+ return DocID{
version: uint16(version),
uuid: uuid,
}, nil
}
-// UUID returns the doc key in UUID form.
-func (key DocKey) UUID() uuid.UUID {
- return key.uuid
+// UUID returns the underlying document identifier in UUID form.
+func (docID DocID) UUID() uuid.UUID {
+ return docID.uuid
}
-// String returns the doc key in string form.
-func (key DocKey) String() string {
+// String returns the underlying document identifier in string form.
+func (docID DocID) String() string {
buf := make([]byte, 1)
- binary.PutUvarint(buf, uint64(key.version))
+ binary.PutUvarint(buf, uint64(docID.version))
versionStr, _ := mbase.Encode(mbase.Base32, buf)
- return versionStr + "-" + key.uuid.String()
+ return versionStr + "-" + docID.uuid.String()
}
-// Bytes returns the DocKey in Byte format.
-func (key DocKey) Bytes() []byte {
+// Bytes returns the underlying document identifier in Byte format.
+func (docID DocID) Bytes() []byte {
buf := make([]byte, binary.MaxVarintLen16)
- binary.PutUvarint(buf, uint64(key.version))
- return append(buf, key.uuid.Bytes()...)
+ binary.PutUvarint(buf, uint64(docID.version))
+ return append(buf, docID.uuid.Bytes()...)
}
diff --git a/client/document.go b/client/document.go
index bcb8ae6070..93e06df27e 100644
--- a/client/document.go
+++ b/client/document.go
@@ -1,4 +1,4 @@
-// Copyright 2022 Democratized Data Foundation
+// Copyright 2023 Democratized Data Foundation
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt.
@@ -12,11 +12,15 @@ package client
import (
"encoding/json"
+ "regexp"
"strings"
"sync"
+ "time"
"github.com/fxamacker/cbor/v2"
"github.com/ipfs/go-cid"
+ "github.com/sourcenetwork/immutable"
+ "github.com/valyala/fastjson"
"github.com/sourcenetwork/defradb/client/request"
ccid "github.com/sourcenetwork/defradb/core/cid"
@@ -53,52 +57,46 @@ import (
// @body: A document interface can be implemented by both a TypedDocument and a
// UnTypedDocument, which use a schema and schemaless approach respectively.
type Document struct {
- key DocKey
- // SchemaVersionID holds the id of the schema version that this document is
- // currently at.
- //
- // Migrating the document will update this value to the output version of the
- // migration.
- SchemaVersionID string
- fields map[string]Field
- values map[Field]Value
- head cid.Cid
- mu sync.RWMutex
+ id DocID
+ fields map[string]Field
+ values map[Field]*FieldValue
+ head cid.Cid
+ mu sync.RWMutex
// marks if document has unsaved changes
isDirty bool
-}
-// NewDocWithKey creates a new Document with a specified key.
-func NewDocWithKey(key DocKey) *Document {
- doc := newEmptyDoc()
- doc.key = key
- return doc
+ schemaDescription SchemaDescription
}
-func newEmptyDoc() *Document {
+func newEmptyDoc(sd SchemaDescription) *Document {
return &Document{
- fields: make(map[string]Field),
- values: make(map[Field]Value),
+ fields: make(map[string]Field),
+ values: make(map[Field]*FieldValue),
+ schemaDescription: sd,
}
}
+// NewDocWithID creates a new Document with a specified key.
+func NewDocWithID(docID DocID, sd SchemaDescription) *Document {
+ doc := newEmptyDoc(sd)
+ doc.id = docID
+ return doc
+}
+
// NewDocFromMap creates a new Document from a data map.
-func NewDocFromMap(data map[string]any) (*Document, error) {
+func NewDocFromMap(data map[string]any, sd SchemaDescription) (*Document, error) {
var err error
- doc := &Document{
- fields: make(map[string]Field),
- values: make(map[Field]Value),
- }
+ doc := newEmptyDoc(sd)
- // check if document contains special _key field
- k, hasKey := data["_key"]
- if hasKey {
- delete(data, "_key") // remove the key so it isn't parsed further
+ // check if document contains special _docID field
+ k, hasDocID := data[request.DocIDFieldName]
+ if hasDocID {
+ delete(data, request.DocIDFieldName) // remove the DocID so it isn't parsed further
kstr, ok := k.(string)
if !ok {
- return nil, NewErrUnexpectedType[string]("data[_key]", k)
+ return nil, NewErrUnexpectedType[string]("data["+request.DocIDFieldName+"]", k)
}
- if doc.key, err = NewDocKeyFromString(kstr); err != nil {
+ if doc.id, err = NewDocIDFromString(kstr); err != nil {
return nil, err
}
}
@@ -108,9 +106,9 @@ func NewDocFromMap(data map[string]any) (*Document, error) {
return nil, err
}
- // if no key was specified, then we assume it doesn't exist and we generate, and set it.
- if !hasKey {
- err = doc.generateAndSetDocKey()
+ // if no DocID was specified, then we assume it doesn't exist and we generate, and set it.
+ if !hasDocID {
+ err = doc.generateAndSetDocID()
if err != nil {
return nil, err
}
@@ -119,15 +117,279 @@ func NewDocFromMap(data map[string]any) (*Document, error) {
return doc, nil
}
+var jsonArrayPattern = regexp.MustCompile(`^\s*\[.*\]\s*$`)
+
+// IsJSONArray returns true if the given byte array is a JSON Array.
+func IsJSONArray(obj []byte) bool {
+ return jsonArrayPattern.Match(obj)
+}
+
// NewFromJSON creates a new instance of a Document from a raw JSON object byte array.
-func NewDocFromJSON(obj []byte) (*Document, error) {
- data := make(map[string]any)
- err := json.Unmarshal(obj, &data)
+func NewDocFromJSON(obj []byte, sd SchemaDescription) (*Document, error) {
+ doc := newEmptyDoc(sd)
+ err := doc.SetWithJSON(obj)
+ if err != nil {
+ return nil, err
+ }
+ err = doc.generateAndSetDocID()
if err != nil {
return nil, err
}
+ return doc, nil
+}
+
+// ManyFromJSON creates a new slice of Documents from a raw JSON array byte array.
+// It will return an error if the given byte array is not a valid JSON array.
+func NewDocsFromJSON(obj []byte, sd SchemaDescription) ([]*Document, error) {
+ v, err := fastjson.ParseBytes(obj)
+ if err != nil {
+ return nil, err
+ }
+ a, err := v.Array()
+ if err != nil {
+ return nil, err
+ }
+
+ docs := make([]*Document, len(a))
+ for i, v := range a {
+ o, err := v.Object()
+ if err != nil {
+ return nil, err
+ }
+ doc := newEmptyDoc(sd)
+ err = doc.setWithFastJSONObject(o)
+ if err != nil {
+ return nil, err
+ }
+ err = doc.generateAndSetDocID()
+ if err != nil {
+ return nil, err
+ }
+ docs[i] = doc
+ }
+
+ return docs, nil
+}
+
+// validateFieldSchema takes a given value as an interface,
+// and ensures it matches the supplied field description.
+// It will do any minor parsing, like dates, and return
+// the typed value again as an interface.
+func validateFieldSchema(val any, field FieldDescription) (any, error) {
+ switch field.Kind {
+ case FieldKind_DocID, FieldKind_STRING, FieldKind_BLOB:
+ return getString(val)
+
+ case FieldKind_STRING_ARRAY:
+ return getArray(val, getString)
+
+ case FieldKind_NILLABLE_STRING_ARRAY:
+ return getNillableArray(val, getString)
+
+ case FieldKind_BOOL:
+ return getBool(val)
+
+ case FieldKind_BOOL_ARRAY:
+ return getArray(val, getBool)
+
+ case FieldKind_NILLABLE_BOOL_ARRAY:
+ return getNillableArray(val, getBool)
+
+ case FieldKind_FLOAT:
+ return getFloat64(val)
+
+ case FieldKind_FLOAT_ARRAY:
+ return getArray(val, getFloat64)
+
+ case FieldKind_NILLABLE_FLOAT_ARRAY:
+ return getNillableArray(val, getFloat64)
+
+ case FieldKind_DATETIME:
+ return getDateTime(val)
+
+ case FieldKind_INT:
+ return getInt64(val)
+
+ case FieldKind_INT_ARRAY:
+ return getArray(val, getInt64)
+
+ case FieldKind_NILLABLE_INT_ARRAY:
+ return getNillableArray(val, getInt64)
+
+ case FieldKind_FOREIGN_OBJECT:
+ return getString(val)
+
+ case FieldKind_FOREIGN_OBJECT_ARRAY:
+ return nil, NewErrFieldOrAliasToFieldNotExist(field.Name)
+ }
- return NewDocFromMap(data)
+ return nil, NewErrUnhandledType("FieldKind", field.Kind)
+}
+
+func getString(v any) (string, error) {
+ switch val := v.(type) {
+ case *fastjson.Value:
+ b, err := val.StringBytes()
+ return string(b), err
+ default:
+ return val.(string), nil
+ }
+}
+
+func getBool(v any) (bool, error) {
+ switch val := v.(type) {
+ case *fastjson.Value:
+ return val.Bool()
+ default:
+ return val.(bool), nil
+ }
+}
+
+func getFloat64(v any) (float64, error) {
+ switch val := v.(type) {
+ case *fastjson.Value:
+ return val.Float64()
+ case int:
+ return float64(val), nil
+ case int32:
+ return float64(val), nil
+ case int64:
+ return float64(val), nil
+ case float64:
+ return val, nil
+ default:
+ return 0, NewErrUnexpectedType[float64]("field", v)
+ }
+}
+
+func getInt64(v any) (int64, error) {
+ switch val := v.(type) {
+ case *fastjson.Value:
+ return val.Int64()
+ case int:
+ return int64(val), nil
+ case int32:
+ return int64(val), nil
+ case int64:
+ return val, nil
+ case float64:
+ return int64(val), nil
+ default:
+ return 0, NewErrUnexpectedType[int64]("field", v)
+ }
+}
+
+func getDateTime(v any) (time.Time, error) {
+ var s string
+ switch val := v.(type) {
+ case *fastjson.Value:
+ b, err := val.StringBytes()
+ if err != nil {
+ return time.Time{}, err
+ }
+ s = string(b)
+ case time.Time:
+ return val, nil
+ default:
+ s = val.(string)
+ }
+ return time.Parse(time.RFC3339, s)
+}
+
+func getArray[T any](
+ v any,
+ typeGetter func(any) (T, error),
+) ([]T, error) {
+ switch val := v.(type) {
+ case *fastjson.Value:
+ if val.Type() == fastjson.TypeNull {
+ return nil, nil
+ }
+
+ valArray, err := val.Array()
+ if err != nil {
+ return nil, err
+ }
+
+ arr := make([]T, len(valArray))
+ for i, arrItem := range valArray {
+ if arrItem.Type() == fastjson.TypeNull {
+ continue
+ }
+ arr[i], err = typeGetter(arrItem)
+ if err != nil {
+ return nil, err
+ }
+ }
+
+ return arr, nil
+ case []any:
+ arr := make([]T, len(val))
+ for i, arrItem := range val {
+ var err error
+ arr[i], err = typeGetter(arrItem)
+ if err != nil {
+ return nil, err
+ }
+ }
+
+ return arr, nil
+ case []T:
+ return val, nil
+ default:
+ return []T{}, nil
+ }
+}
+
+func getNillableArray[T any](
+ v any,
+ typeGetter func(any) (T, error),
+) ([]immutable.Option[T], error) {
+ switch val := v.(type) {
+ case *fastjson.Value:
+ if val.Type() == fastjson.TypeNull {
+ return nil, nil
+ }
+
+ valArray, err := val.Array()
+ if err != nil {
+ return nil, err
+ }
+
+ arr := make([]immutable.Option[T], len(valArray))
+ for i, arrItem := range valArray {
+ if arrItem.Type() == fastjson.TypeNull {
+ arr[i] = immutable.None[T]()
+ continue
+ }
+ v, err := typeGetter(arrItem)
+ if err != nil {
+ return nil, err
+ }
+ arr[i] = immutable.Some(v)
+ }
+
+ return arr, nil
+ case []any:
+ arr := make([]immutable.Option[T], len(val))
+ for i, arrItem := range val {
+ if arrItem == nil {
+ arr[i] = immutable.None[T]()
+ continue
+ }
+ v, err := typeGetter(arrItem)
+ if err != nil {
+ return nil, err
+ }
+ arr[i] = immutable.Some(v)
+ }
+
+ return arr, nil
+ case []immutable.Option[T]:
+ return val, nil
+ default:
+ return []immutable.Option[T]{}, nil
+ }
}
// Head returns the current head CID of the document.
@@ -144,10 +406,10 @@ func (doc *Document) SetHead(head cid.Cid) {
doc.head = head
}
-// Key returns the generated DocKey for this document.
-func (doc *Document) Key() DocKey {
- // Reading without a read-lock as we assume the DocKey is immutable
- return doc.key
+// ID returns the generated DocID for this document.
+func (doc *Document) ID() DocID {
+ // Reading without a read-lock as we assume the DocID is immutable
+ return doc.id
}
// Get returns the raw value for a given field.
@@ -163,7 +425,7 @@ func (doc *Document) Get(field string) (any, error) {
}
// GetValue given a field as a string, return the Value type.
-func (doc *Document) GetValue(field string) (Value, error) {
+func (doc *Document) GetValue(field string) (*FieldValue, error) {
doc.mu.RLock()
defer doc.mu.RUnlock()
path, subPaths, hasSubPaths := parseFieldPath(field)
@@ -187,7 +449,7 @@ func (doc *Document) GetValue(field string) (Value, error) {
}
// GetValueWithField gets the Value type from a given Field type
-func (doc *Document) GetValueWithField(f Field) (Value, error) {
+func (doc *Document) GetValueWithField(f Field) (*FieldValue, error) {
doc.mu.RLock()
defer doc.mu.RUnlock()
v, exists := doc.values[f]
@@ -201,30 +463,52 @@ func (doc *Document) GetValueWithField(f Field) (Value, error) {
// JSON Merge Patch object. Note: fields indicated as nil in the Merge
// Patch are to be deleted
// @todo: Handle sub documents for SetWithJSON
-func (doc *Document) SetWithJSON(patch []byte) error {
- var patchObj map[string]any
- err := json.Unmarshal(patch, &patchObj)
+func (doc *Document) SetWithJSON(obj []byte) error {
+ v, err := fastjson.ParseBytes(obj)
+ if err != nil {
+ return err
+ }
+ o, err := v.Object()
if err != nil {
return err
}
- for k, v := range patchObj {
- err = doc.Set(k, v)
+ return doc.setWithFastJSONObject(o)
+}
+
+func (doc *Document) setWithFastJSONObject(obj *fastjson.Object) error {
+ var visitErr error
+ obj.Visit(func(k []byte, v *fastjson.Value) {
+ fieldName := string(k)
+ err := doc.Set(fieldName, v)
if err != nil {
- return err
+ visitErr = err
+ return
}
- }
- return nil
+ })
+ return visitErr
}
// Set the value of a field.
func (doc *Document) Set(field string, value any) error {
- return doc.setAndParseType(field, value)
-}
-
-// SetAs is the same as set, but you can manually set the CRDT type.
-func (doc *Document) SetAs(field string, value any, t CType) error {
- return doc.setCBOR(t, field, value)
+ fd, exists := doc.schemaDescription.GetField(field)
+ if !exists {
+ return NewErrFieldNotExist(field)
+ }
+ if fd.IsRelation() && !fd.IsObjectArray() {
+ if !strings.HasSuffix(field, request.RelatedObjectID) {
+ field = field + request.RelatedObjectID
+ }
+ fd, exists = doc.schemaDescription.GetField(field)
+ if !exists {
+ return NewErrFieldNotExist(field)
+ }
+ }
+ val, err := validateFieldSchema(value, fd)
+ if err != nil {
+ return err
+ }
+ return doc.setCBOR(fd.Typ, field, val)
}
// Delete removes a field, and marks it to be deleted on the following db.Update() call.
@@ -241,7 +525,7 @@ func (doc *Document) Delete(fields ...string) error {
return nil
}
-func (doc *Document) set(t CType, field string, value Value) error {
+func (doc *Document) set(t CType, field string, value *FieldValue) error {
doc.mu.Lock()
defer doc.mu.Unlock()
var f Field
@@ -257,87 +541,16 @@ func (doc *Document) set(t CType, field string, value Value) error {
}
func (doc *Document) setCBOR(t CType, field string, val any) error {
- value := newCBORValue(t, val)
+ value := NewFieldValue(t, val)
return doc.set(t, field, value)
}
-func (doc *Document) setObject(t CType, field string, val *Document) error {
- value := newValue(t, val)
- return doc.set(t, field, &value)
-}
-
-// @todo: Update with document schemas
-func (doc *Document) setAndParseType(field string, value any) error {
- if value == nil {
- return doc.setCBOR(LWW_REGISTER, field, value)
- }
-
- switch val := value.(type) {
- // int (any number)
- case int:
- err := doc.setCBOR(LWW_REGISTER, field, int64(val))
- if err != nil {
- return err
- }
- case uint64:
- err := doc.setCBOR(LWW_REGISTER, field, int64(val))
- if err != nil {
- return err
- }
-
- case float64:
- // case int64:
-
- // Check if its actually a float or just an int
- if float64(int64(val)) == val { //int
- err := doc.setCBOR(LWW_REGISTER, field, int64(val))
- if err != nil {
- return err
- }
- } else { //float
- err := doc.setCBOR(LWW_REGISTER, field, val)
- if err != nil {
- return err
- }
- }
-
- // string, bool, and more
- case string, bool, int64, []any, []bool, []*bool, []int64, []*int64, []float64, []*float64, []string, []*string:
- err := doc.setCBOR(LWW_REGISTER, field, val)
- if err != nil {
- return err
- }
-
- // sub object, recurse down.
- // @TODO: Object Definitions
- // You can use an object as a way to override defaults
- // and types for JSON literals.
- // Eg.
- // Instead of { "Timestamp": 123 }
- // - which is parsed as an int
- // Use { "Timestamp" : { "_Type": "uint64", "_Value": 123 } }
- // - Which is parsed as an uint64
- case map[string]any:
- subDoc := newEmptyDoc()
- err := subDoc.setAndParseObjectType(val)
- if err != nil {
- return err
- }
-
- err = doc.setObject(OBJECT, field, subDoc)
- if err != nil {
- return err
- }
-
- default:
- return NewErrUnhandledType(field, val)
- }
- return nil
-}
-
func (doc *Document) setAndParseObjectType(value map[string]any) error {
for k, v := range value {
- err := doc.setAndParseType(k, v)
+ if v == nil {
+ continue
+ }
+ err := doc.Set(k, v)
if err != nil {
return err
}
@@ -353,7 +566,7 @@ func (doc *Document) Fields() map[string]Field {
}
// Values gets the document values as a map.
-func (doc *Document) Values() map[Field]Value {
+func (doc *Document) Values() map[Field]*FieldValue {
doc.mu.RLock()
defer doc.mu.RUnlock()
return doc.values
@@ -392,8 +605,7 @@ func (doc *Document) String() (string, error) {
return string(j), nil
}
-// ToMap returns the document as a map[string]any
-// object.
+// ToMap returns the document as a map[string]any object.
func (doc *Document) ToMap() (map[string]any, error) {
return doc.toMapWithKey()
}
@@ -424,7 +636,7 @@ func (doc *Document) Clean() {
val, _ := doc.GetValueWithField(v)
if val.IsDirty() {
if val.IsDelete() {
- doc.SetAs(v.Name(), nil, v.Type()) //nolint:errcheck
+ doc.Set(v.Name(), nil) //nolint:errcheck
}
val.Clean()
}
@@ -479,79 +691,45 @@ func (doc *Document) toMapWithKey() (map[string]any, error) {
docMap[k] = value.Value()
}
- docMap["_key"] = doc.Key().String()
+ docMap[request.DocIDFieldName] = doc.ID().String()
return docMap, nil
}
-// GenerateDocKey generates docKey/docID corresponding to the document.
-func (doc *Document) GenerateDocKey() (DocKey, error) {
+// GenerateDocID generates the DocID corresponding to the document.
+func (doc *Document) GenerateDocID() (DocID, error) {
bytes, err := doc.Bytes()
if err != nil {
- return DocKey{}, err
+ return DocID{}, err
}
cid, err := ccid.NewSHA256CidV1(bytes)
if err != nil {
- return DocKey{}, err
+ return DocID{}, err
}
- return NewDocKeyV0(cid), nil
+ return NewDocIDV0(cid), nil
}
-// setDocKey sets the `doc.key` (should NOT be public).
-func (doc *Document) setDocKey(docID DocKey) {
+// setDocID sets the `doc.id` (should NOT be public).
+func (doc *Document) setDocID(docID DocID) {
doc.mu.Lock()
defer doc.mu.Unlock()
- doc.key = docID
+ doc.id = docID
}
-// generateAndSetDocKey generates the docKey/docID and then (re)sets `doc.key`.
-func (doc *Document) generateAndSetDocKey() error {
- docKey, err := doc.GenerateDocKey()
+// generateAndSetDocID generates the DocID and then (re)sets `doc.id`.
+func (doc *Document) generateAndSetDocID() error {
+ docID, err := doc.GenerateDocID()
if err != nil {
return err
}
- doc.setDocKey(docKey)
+ doc.setDocID(docID)
return nil
}
-func (doc *Document) remapAliasFields(fieldDescriptions []FieldDescription) (bool, error) {
- doc.mu.Lock()
- defer doc.mu.Unlock()
-
- foundAlias := false
- for docField, docFieldValue := range doc.fields {
- for _, fieldDescription := range fieldDescriptions {
- maybeAliasField := docField + request.RelatedObjectID
- if fieldDescription.Name == maybeAliasField {
- foundAlias = true
- doc.fields[maybeAliasField] = docFieldValue
- delete(doc.fields, docField)
- }
- }
- }
-
- return foundAlias, nil
-}
-
-// RemapAliasFieldsAndDockey remaps the alias fields and fixes (overwrites) the dockey.
-func (doc *Document) RemapAliasFieldsAndDockey(fieldDescriptions []FieldDescription) error {
- foundAlias, err := doc.remapAliasFields(fieldDescriptions)
- if err != nil {
- return err
- }
-
- if !foundAlias {
- return nil
- }
-
- // Update the dockey so dockey isn't based on an aliased name of a field.
- return doc.generateAndSetDocKey()
-}
-
// DocumentStatus represent the state of the document in the DAG store.
// It can either be `Active“ or `Deleted`.
type DocumentStatus uint8
@@ -578,65 +756,6 @@ func (dStatus DocumentStatus) IsDeleted() bool {
return dStatus > 1
}
-// loops through an object of the form map[string]any
-// and fills in the Document with each field it finds in the object.
-// Automatically handles sub objects and arrays.
-// Does not allow anonymous fields, error is thrown in this case
-// Eg. The JSON value [1,2,3,4] by itself is a valid JSON Object, but has no
-// field name.
-// func parseJSONObject(doc *Document, data map[string]any) error {
-// for k, v := range data {
-// switch v.(type) {
-
-// // int (any number)
-// case float64:
-// // case int64:
-
-// // Check if its actually a float or just an int
-// val := v.(float64)
-// if float64(int64(val)) == val { //int
-// doc.setCBOR(crdt.LWW_REGISTER, k, int64(val))
-// } else { //float
-// panic("todo")
-// }
-// break
-
-// // string
-// case string:
-// doc.setCBOR(crdt.LWW_REGISTER, k, v)
-// break
-
-// // array
-// case []any:
-// break
-
-// // sub object, recurse down.
-// // @TODO: Object Definitions
-// // You can use an object as a way to override defaults
-// // and types for JSON literals.
-// // Eg.
-// // Instead of { "Timestamp": 123 }
-// // - which is parsed as an int
-// // Use { "Timestamp" : { "_Type": "uint64", "_Value": 123 } }
-// // - Which is parsed as an uint64
-// case map[string]any:
-// subDoc := newEmptyDoc()
-// err := parseJSONObject(subDoc, v.(map[string]any))
-// if err != nil {
-// return err
-// }
-
-// doc.setObject(crdt.OBJECT, k, subDoc)
-// break
-
-// default:
-// return errors.Wrap("Unhandled type in raw JSON: %v => %T", k, v)
-
-// }
-// }
-// return nil
-// }
-
// parses a document field path, can have sub elements if we have embedded objects.
// Returns the first path, the remaining split paths, and a bool indicating if there are sub paths
func parseFieldPath(path string) (string, string, bool) {
diff --git a/client/document_test.go b/client/document_test.go
index 9073373cd3..dc5867b562 100644
--- a/client/document_test.go
+++ b/client/document_test.go
@@ -1,4 +1,4 @@
-// Copyright 2022 Democratized Data Foundation
+// Copyright 2023 Democratized Data Foundation
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt.
@@ -14,6 +14,7 @@ import (
"testing"
"github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
ccid "github.com/sourcenetwork/defradb/core/cid"
)
@@ -21,18 +22,32 @@ import (
var (
testJSONObj = []byte(`{
"Name": "John",
- "Age": 26,
- "Address": {
- "Street": "Main",
- "City": "Toronto"
- }
+ "Age": 26
}`)
pref = ccid.NewDefaultSHA256PrefixV1()
+
+ schemaDescriptions = []SchemaDescription{
+ {
+ Name: "User",
+ Fields: []FieldDescription{
+ {
+ Name: "Name",
+ Typ: LWW_REGISTER,
+ Kind: FieldKind_STRING,
+ },
+ {
+ Name: "Age",
+ Typ: LWW_REGISTER,
+ Kind: FieldKind_INT,
+ },
+ },
+ },
+ }
)
func TestNewFromJSON(t *testing.T) {
- doc, err := NewDocFromJSON(testJSONObj)
+ doc, err := NewDocFromJSON(testJSONObj, schemaDescriptions[0])
if err != nil {
t.Error("Error creating new doc from JSON:", err)
return
@@ -48,10 +63,10 @@ func TestNewFromJSON(t *testing.T) {
if err != nil {
t.Error(err)
}
- objKey := NewDocKeyV0(c)
+ objKey := NewDocIDV0(c)
- if objKey.String() != doc.Key().String() {
- t.Errorf("Incorrect doc key. Want %v, have %v", objKey.String(), doc.Key().String())
+ if objKey.String() != doc.ID().String() {
+ t.Errorf("Incorrect document ID. Want %v, have %v", objKey.String(), doc.ID().String())
return
}
@@ -61,31 +76,16 @@ func TestNewFromJSON(t *testing.T) {
assert.Equal(t, doc.fields["Name"].Type(), LWW_REGISTER)
assert.Equal(t, doc.fields["Age"].Name(), "Age")
assert.Equal(t, doc.fields["Age"].Type(), LWW_REGISTER)
- assert.Equal(t, doc.fields["Address"].Name(), "Address")
- assert.Equal(t, doc.fields["Address"].Type(), OBJECT)
//values
assert.Equal(t, doc.values[doc.fields["Name"]].Value(), "John")
assert.Equal(t, doc.values[doc.fields["Name"]].IsDocument(), false)
assert.Equal(t, doc.values[doc.fields["Age"]].Value(), int64(26))
assert.Equal(t, doc.values[doc.fields["Age"]].IsDocument(), false)
- assert.Equal(t, doc.values[doc.fields["Address"]].IsDocument(), true)
-
- //subdoc fields
- subDoc := doc.values[doc.fields["Address"]].Value().(*Document)
- assert.Equal(t, subDoc.fields["Street"].Name(), "Street")
- assert.Equal(t, subDoc.fields["Street"].Type(), LWW_REGISTER)
- assert.Equal(t, subDoc.fields["City"].Name(), "City")
- assert.Equal(t, subDoc.fields["City"].Type(), LWW_REGISTER)
-
- //subdoc values
- assert.Equal(t, subDoc.values[subDoc.fields["Street"]].Value(), "Main")
- assert.Equal(t, subDoc.values[subDoc.fields["Street"]].IsDocument(), false)
- assert.Equal(t, subDoc.values[subDoc.fields["City"]].Value(), "Toronto")
}
func TestSetWithJSON(t *testing.T) {
- doc, err := NewDocFromJSON(testJSONObj)
+ doc, err := NewDocFromJSON(testJSONObj, schemaDescriptions[0])
if err != nil {
t.Error("Error creating new doc from JSON:", err)
return
@@ -101,17 +101,16 @@ func TestSetWithJSON(t *testing.T) {
if err != nil {
t.Error(err)
}
- objKey := NewDocKeyV0(c)
+ objKey := NewDocIDV0(c)
- if objKey.String() != doc.Key().String() {
- t.Errorf("Incorrect doc key. Want %v, have %v", objKey.String(), doc.Key().String())
+ if objKey.String() != doc.ID().String() {
+ t.Errorf("Incorrect document ID. Want %v, have %v", objKey.String(), doc.ID().String())
return
}
updatePatch := []byte(`{
"Name": "Alice",
- "Age": 27,
- "Address": null
+ "Age": 27
}`)
err = doc.SetWithJSON(updatePatch)
if err != nil {
@@ -124,26 +123,15 @@ func TestSetWithJSON(t *testing.T) {
assert.Equal(t, doc.fields["Name"].Type(), LWW_REGISTER)
assert.Equal(t, doc.fields["Age"].Name(), "Age")
assert.Equal(t, doc.fields["Age"].Type(), LWW_REGISTER)
- assert.Equal(t, doc.fields["Address"].Name(), "Address")
- assert.Equal(t, doc.fields["Address"].Type(), OBJECT)
//values
assert.Equal(t, doc.values[doc.fields["Name"]].Value(), "Alice")
assert.Equal(t, doc.values[doc.fields["Name"]].IsDocument(), false)
assert.Equal(t, doc.values[doc.fields["Age"]].Value(), int64(27))
assert.Equal(t, doc.values[doc.fields["Age"]].IsDocument(), false)
- assert.Equal(t, doc.values[doc.fields["Address"]].Value(), nil)
- assert.Equal(t, doc.values[doc.fields["Address"]].IsDocument(), false)
-
- //subdoc fields
- // subDoc := doc.values[doc.fields["Address"]].Value().(*Document)
- // assert.Equal(t, subDoc.fields["Street"].Name(), "Street")
- // assert.Equal(t, subDoc.fields["Street"].Type(), client.LWW_REGISTER)
- // assert.Equal(t, subDoc.fields["City"].Name(), "City")
- // assert.Equal(t, subDoc.fields["City"].Type(), client.LWW_REGISTER)
-
- // //subdoc values
- // assert.Equal(t, subDoc.values[subDoc.fields["Street"]].Value(), "Main")
- // assert.Equal(t, subDoc.values[subDoc.fields["Street"]].IsDocument(), false)
- // assert.Equal(t, subDoc.values[subDoc.fields["City"]].Value(), "Toronto")
+}
+
+func TestNewDocsFromJSON_WithObjectInsteadOfArray_Error(t *testing.T) {
+ _, err := NewDocsFromJSON(testJSONObj, schemaDescriptions[0])
+ require.ErrorContains(t, err, "value doesn't contain array; it contains object")
}
diff --git a/client/errors.go b/client/errors.go
index 048d96c00d..78daf3531b 100644
--- a/client/errors.go
+++ b/client/errors.go
@@ -17,13 +17,17 @@ import (
)
const (
- errFieldNotExist string = "The given field does not exist"
- errUnexpectedType string = "unexpected type"
- errParsingFailed string = "failed to parse argument"
- errUninitializeProperty string = "invalid state, required property is uninitialized"
- errMaxTxnRetries string = "reached maximum transaction reties"
- errRelationOneSided string = "relation must be defined on both schemas"
- errCollectionNotFound string = "collection not found"
+ errFieldNotExist string = "The given field does not exist"
+ errUnexpectedType string = "unexpected type"
+ errParsingFailed string = "failed to parse argument"
+ errUninitializeProperty string = "invalid state, required property is uninitialized"
+ errMaxTxnRetries string = "reached maximum transaction reties"
+ errRelationOneSided string = "relation must be defined on both schemas"
+ errCollectionNotFound string = "collection not found"
+ errFieldOrAliasToFieldNotExist string = "The given field or alias to field does not exist"
+ errUnknownCRDT string = "unknown crdt"
+ errCRDTKindMismatch string = "CRDT type %s can't be assigned to field kind %s"
+ errInvalidCRDTType string = "CRDT type not supported"
)
// Errors returnable from this package.
@@ -31,22 +35,16 @@ const (
// This list is incomplete and undefined errors may also be returned.
// Errors returned from this package may be tested against these errors with errors.Is.
var (
- ErrFieldNotExist = errors.New(errFieldNotExist)
- ErrUnexpectedType = errors.New(errUnexpectedType)
- ErrParsingFailed = errors.New(errParsingFailed)
- ErrUninitializeProperty = errors.New(errUninitializeProperty)
- ErrFieldNotObject = errors.New("trying to access field on a non object type")
- ErrValueTypeMismatch = errors.New("value does not match indicated type")
- ErrIndexNotFound = errors.New("no index found for given ID")
- ErrDocumentNotFound = errors.New("no document for the given key exists")
- ErrInvalidUpdateTarget = errors.New("the target document to update is of invalid type")
- ErrInvalidUpdater = errors.New("the updater of a document is of invalid type")
- ErrInvalidDeleteTarget = errors.New("the target document to delete is of invalid type")
- ErrMalformedDocKey = errors.New("malformed DocKey, missing either version or cid")
- ErrInvalidDocKeyVersion = errors.New("invalid DocKey version")
- ErrMaxTxnRetries = errors.New(errMaxTxnRetries)
- ErrRelationOneSided = errors.New(errRelationOneSided)
- ErrCollectionNotFound = errors.New(errCollectionNotFound)
+ ErrFieldNotExist = errors.New(errFieldNotExist)
+ ErrUnexpectedType = errors.New(errUnexpectedType)
+ ErrFieldNotObject = errors.New("trying to access field on a non object type")
+ ErrValueTypeMismatch = errors.New("value does not match indicated type")
+ ErrDocumentNotFound = errors.New("no document for the given ID exists")
+ ErrInvalidUpdateTarget = errors.New("the target document to update is of invalid type")
+ ErrInvalidUpdater = errors.New("the updater of a document is of invalid type")
+ ErrInvalidDeleteTarget = errors.New("the target document to delete is of invalid type")
+ ErrMalformedDocID = errors.New("malformed document ID, missing either version or cid")
+ ErrInvalidDocIDVersion = errors.New("invalid document ID version")
)
// NewErrFieldNotExist returns an error indicating that the given field does not exist.
@@ -123,3 +121,27 @@ func NewErrCollectionNotFoundForSchema(schemaRoot string) error {
errors.NewKV("SchemaRoot", schemaRoot),
)
}
+
+func NewErrUnknownCRDT(cType CType) error {
+ return errors.New(
+ errUnknownCRDT,
+ errors.NewKV("Type", cType),
+ )
+}
+
+// NewErrFieldOrAliasToFieldNotExist returns an error indicating that the given field or an alias field does not exist.
+func NewErrFieldOrAliasToFieldNotExist(name string) error {
+ return errors.New(errFieldOrAliasToFieldNotExist, errors.NewKV("Name", name))
+}
+
+func NewErrInvalidCRDTType(name, crdtType string) error {
+ return errors.New(
+ errInvalidCRDTType,
+ errors.NewKV("Name", name),
+ errors.NewKV("CRDTType", crdtType),
+ )
+}
+
+func NewErrCRDTKindMismatch(cType, kind string) error {
+ return errors.New(fmt.Sprintf(errCRDTKindMismatch, cType, kind))
+}
diff --git a/client/index.go b/client/index.go
index 69f0362017..5e2d397394 100644
--- a/client/index.go
+++ b/client/index.go
@@ -36,6 +36,8 @@ type IndexDescription struct {
ID uint32
// Fields contains the fields that are being indexed.
Fields []IndexedFieldDescription
+ // Unique indicates whether the index is unique.
+ Unique bool
}
// CollectIndexedFields returns all fields that are indexed by all collection indexes.
diff --git a/client/lens.go b/client/lens.go
index 7b1264275f..35ef9f1ee3 100644
--- a/client/lens.go
+++ b/client/lens.go
@@ -45,18 +45,18 @@ type LensConfig struct {
type LensRegistry interface {
// WithTxn returns a new LensRegistry scoped to the given transaction.
//
- // WARNING: Currently this does not provide snapshot isolation, if other transactions are commited
+ // WARNING: Currently this does not provide snapshot isolation, if other transactions are committed
// after this has been created, the results of those commits will be visible within this scope.
WithTxn(datastore.Txn) LensRegistry
- // SetMigration sets the migration for the given source-destination schema version IDs. Is equivilent to
+ // SetMigration sets the migration for the given source-destination schema version IDs. Is equivalent to
// calling `Store.SetMigration(ctx, cfg)`.
//
// There may only be one migration per schema version id. If another migration was registered it will be
// overwritten by this migration.
//
// Neither of the schema version IDs specified in the configuration need to exist at the time of calling.
- // This is to allow the migration of documents of schema versions unknown to the local node recieved by the
+ // This is to allow the migration of documents of schema versions unknown to the local node received by the
// P2P system.
//
// Migrations will only run if there is a complete path from the document schema version to the latest local
diff --git a/client/mocks/collection.go b/client/mocks/collection.go
index f3d7f58354..b1fac9c243 100644
--- a/client/mocks/collection.go
+++ b/client/mocks/collection.go
@@ -206,21 +206,21 @@ func (_c *Collection_Definition_Call) RunAndReturn(run func() client.CollectionD
}
// Delete provides a mock function with given fields: _a0, _a1
-func (_m *Collection) Delete(_a0 context.Context, _a1 client.DocKey) (bool, error) {
+func (_m *Collection) Delete(_a0 context.Context, _a1 client.DocID) (bool, error) {
ret := _m.Called(_a0, _a1)
var r0 bool
var r1 error
- if rf, ok := ret.Get(0).(func(context.Context, client.DocKey) (bool, error)); ok {
+ if rf, ok := ret.Get(0).(func(context.Context, client.DocID) (bool, error)); ok {
return rf(_a0, _a1)
}
- if rf, ok := ret.Get(0).(func(context.Context, client.DocKey) bool); ok {
+ if rf, ok := ret.Get(0).(func(context.Context, client.DocID) bool); ok {
r0 = rf(_a0, _a1)
} else {
r0 = ret.Get(0).(bool)
}
- if rf, ok := ret.Get(1).(func(context.Context, client.DocKey) error); ok {
+ if rf, ok := ret.Get(1).(func(context.Context, client.DocID) error); ok {
r1 = rf(_a0, _a1)
} else {
r1 = ret.Error(1)
@@ -236,14 +236,14 @@ type Collection_Delete_Call struct {
// Delete is a helper method to define mock.On call
// - _a0 context.Context
-// - _a1 client.DocKey
+// - _a1 client.DocID
func (_e *Collection_Expecter) Delete(_a0 interface{}, _a1 interface{}) *Collection_Delete_Call {
return &Collection_Delete_Call{Call: _e.mock.On("Delete", _a0, _a1)}
}
-func (_c *Collection_Delete_Call) Run(run func(_a0 context.Context, _a1 client.DocKey)) *Collection_Delete_Call {
+func (_c *Collection_Delete_Call) Run(run func(_a0 context.Context, _a1 client.DocID)) *Collection_Delete_Call {
_c.Call.Run(func(args mock.Arguments) {
- run(args[0].(context.Context), args[1].(client.DocKey))
+ run(args[0].(context.Context), args[1].(client.DocID))
})
return _c
}
@@ -253,7 +253,7 @@ func (_c *Collection_Delete_Call) Return(_a0 bool, _a1 error) *Collection_Delete
return _c
}
-func (_c *Collection_Delete_Call) RunAndReturn(run func(context.Context, client.DocKey) (bool, error)) *Collection_Delete_Call {
+func (_c *Collection_Delete_Call) RunAndReturn(run func(context.Context, client.DocID) (bool, error)) *Collection_Delete_Call {
_c.Call.Return(run)
return _c
}
@@ -313,25 +313,25 @@ func (_c *Collection_DeleteWith_Call) RunAndReturn(run func(context.Context, int
return _c
}
-// DeleteWithFilter provides a mock function with given fields: ctx, filter
-func (_m *Collection) DeleteWithFilter(ctx context.Context, filter interface{}) (*client.DeleteResult, error) {
- ret := _m.Called(ctx, filter)
+// DeleteWithDocID provides a mock function with given fields: _a0, _a1
+func (_m *Collection) DeleteWithDocID(_a0 context.Context, _a1 client.DocID) (*client.DeleteResult, error) {
+ ret := _m.Called(_a0, _a1)
var r0 *client.DeleteResult
var r1 error
- if rf, ok := ret.Get(0).(func(context.Context, interface{}) (*client.DeleteResult, error)); ok {
- return rf(ctx, filter)
+ if rf, ok := ret.Get(0).(func(context.Context, client.DocID) (*client.DeleteResult, error)); ok {
+ return rf(_a0, _a1)
}
- if rf, ok := ret.Get(0).(func(context.Context, interface{}) *client.DeleteResult); ok {
- r0 = rf(ctx, filter)
+ if rf, ok := ret.Get(0).(func(context.Context, client.DocID) *client.DeleteResult); ok {
+ r0 = rf(_a0, _a1)
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).(*client.DeleteResult)
}
}
- if rf, ok := ret.Get(1).(func(context.Context, interface{}) error); ok {
- r1 = rf(ctx, filter)
+ if rf, ok := ret.Get(1).(func(context.Context, client.DocID) error); ok {
+ r1 = rf(_a0, _a1)
} else {
r1 = ret.Error(1)
}
@@ -339,45 +339,45 @@ func (_m *Collection) DeleteWithFilter(ctx context.Context, filter interface{})
return r0, r1
}
-// Collection_DeleteWithFilter_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'DeleteWithFilter'
-type Collection_DeleteWithFilter_Call struct {
+// Collection_DeleteWithDocID_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'DeleteWithDocID'
+type Collection_DeleteWithDocID_Call struct {
*mock.Call
}
-// DeleteWithFilter is a helper method to define mock.On call
-// - ctx context.Context
-// - filter interface{}
-func (_e *Collection_Expecter) DeleteWithFilter(ctx interface{}, filter interface{}) *Collection_DeleteWithFilter_Call {
- return &Collection_DeleteWithFilter_Call{Call: _e.mock.On("DeleteWithFilter", ctx, filter)}
+// DeleteWithDocID is a helper method to define mock.On call
+// - _a0 context.Context
+// - _a1 client.DocID
+func (_e *Collection_Expecter) DeleteWithDocID(_a0 interface{}, _a1 interface{}) *Collection_DeleteWithDocID_Call {
+ return &Collection_DeleteWithDocID_Call{Call: _e.mock.On("DeleteWithDocID", _a0, _a1)}
}
-func (_c *Collection_DeleteWithFilter_Call) Run(run func(ctx context.Context, filter interface{})) *Collection_DeleteWithFilter_Call {
+func (_c *Collection_DeleteWithDocID_Call) Run(run func(_a0 context.Context, _a1 client.DocID)) *Collection_DeleteWithDocID_Call {
_c.Call.Run(func(args mock.Arguments) {
- run(args[0].(context.Context), args[1].(interface{}))
+ run(args[0].(context.Context), args[1].(client.DocID))
})
return _c
}
-func (_c *Collection_DeleteWithFilter_Call) Return(_a0 *client.DeleteResult, _a1 error) *Collection_DeleteWithFilter_Call {
+func (_c *Collection_DeleteWithDocID_Call) Return(_a0 *client.DeleteResult, _a1 error) *Collection_DeleteWithDocID_Call {
_c.Call.Return(_a0, _a1)
return _c
}
-func (_c *Collection_DeleteWithFilter_Call) RunAndReturn(run func(context.Context, interface{}) (*client.DeleteResult, error)) *Collection_DeleteWithFilter_Call {
+func (_c *Collection_DeleteWithDocID_Call) RunAndReturn(run func(context.Context, client.DocID) (*client.DeleteResult, error)) *Collection_DeleteWithDocID_Call {
_c.Call.Return(run)
return _c
}
-// DeleteWithKey provides a mock function with given fields: _a0, _a1
-func (_m *Collection) DeleteWithKey(_a0 context.Context, _a1 client.DocKey) (*client.DeleteResult, error) {
+// DeleteWithDocIDs provides a mock function with given fields: _a0, _a1
+func (_m *Collection) DeleteWithDocIDs(_a0 context.Context, _a1 []client.DocID) (*client.DeleteResult, error) {
ret := _m.Called(_a0, _a1)
var r0 *client.DeleteResult
var r1 error
- if rf, ok := ret.Get(0).(func(context.Context, client.DocKey) (*client.DeleteResult, error)); ok {
+ if rf, ok := ret.Get(0).(func(context.Context, []client.DocID) (*client.DeleteResult, error)); ok {
return rf(_a0, _a1)
}
- if rf, ok := ret.Get(0).(func(context.Context, client.DocKey) *client.DeleteResult); ok {
+ if rf, ok := ret.Get(0).(func(context.Context, []client.DocID) *client.DeleteResult); ok {
r0 = rf(_a0, _a1)
} else {
if ret.Get(0) != nil {
@@ -385,7 +385,7 @@ func (_m *Collection) DeleteWithKey(_a0 context.Context, _a1 client.DocKey) (*cl
}
}
- if rf, ok := ret.Get(1).(func(context.Context, client.DocKey) error); ok {
+ if rf, ok := ret.Get(1).(func(context.Context, []client.DocID) error); ok {
r1 = rf(_a0, _a1)
} else {
r1 = ret.Error(1)
@@ -394,54 +394,54 @@ func (_m *Collection) DeleteWithKey(_a0 context.Context, _a1 client.DocKey) (*cl
return r0, r1
}
-// Collection_DeleteWithKey_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'DeleteWithKey'
-type Collection_DeleteWithKey_Call struct {
+// Collection_DeleteWithDocIDs_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'DeleteWithDocIDs'
+type Collection_DeleteWithDocIDs_Call struct {
*mock.Call
}
-// DeleteWithKey is a helper method to define mock.On call
+// DeleteWithDocIDs is a helper method to define mock.On call
// - _a0 context.Context
-// - _a1 client.DocKey
-func (_e *Collection_Expecter) DeleteWithKey(_a0 interface{}, _a1 interface{}) *Collection_DeleteWithKey_Call {
- return &Collection_DeleteWithKey_Call{Call: _e.mock.On("DeleteWithKey", _a0, _a1)}
+// - _a1 []client.DocID
+func (_e *Collection_Expecter) DeleteWithDocIDs(_a0 interface{}, _a1 interface{}) *Collection_DeleteWithDocIDs_Call {
+ return &Collection_DeleteWithDocIDs_Call{Call: _e.mock.On("DeleteWithDocIDs", _a0, _a1)}
}
-func (_c *Collection_DeleteWithKey_Call) Run(run func(_a0 context.Context, _a1 client.DocKey)) *Collection_DeleteWithKey_Call {
+func (_c *Collection_DeleteWithDocIDs_Call) Run(run func(_a0 context.Context, _a1 []client.DocID)) *Collection_DeleteWithDocIDs_Call {
_c.Call.Run(func(args mock.Arguments) {
- run(args[0].(context.Context), args[1].(client.DocKey))
+ run(args[0].(context.Context), args[1].([]client.DocID))
})
return _c
}
-func (_c *Collection_DeleteWithKey_Call) Return(_a0 *client.DeleteResult, _a1 error) *Collection_DeleteWithKey_Call {
+func (_c *Collection_DeleteWithDocIDs_Call) Return(_a0 *client.DeleteResult, _a1 error) *Collection_DeleteWithDocIDs_Call {
_c.Call.Return(_a0, _a1)
return _c
}
-func (_c *Collection_DeleteWithKey_Call) RunAndReturn(run func(context.Context, client.DocKey) (*client.DeleteResult, error)) *Collection_DeleteWithKey_Call {
+func (_c *Collection_DeleteWithDocIDs_Call) RunAndReturn(run func(context.Context, []client.DocID) (*client.DeleteResult, error)) *Collection_DeleteWithDocIDs_Call {
_c.Call.Return(run)
return _c
}
-// DeleteWithKeys provides a mock function with given fields: _a0, _a1
-func (_m *Collection) DeleteWithKeys(_a0 context.Context, _a1 []client.DocKey) (*client.DeleteResult, error) {
- ret := _m.Called(_a0, _a1)
+// DeleteWithFilter provides a mock function with given fields: ctx, filter
+func (_m *Collection) DeleteWithFilter(ctx context.Context, filter interface{}) (*client.DeleteResult, error) {
+ ret := _m.Called(ctx, filter)
var r0 *client.DeleteResult
var r1 error
- if rf, ok := ret.Get(0).(func(context.Context, []client.DocKey) (*client.DeleteResult, error)); ok {
- return rf(_a0, _a1)
+ if rf, ok := ret.Get(0).(func(context.Context, interface{}) (*client.DeleteResult, error)); ok {
+ return rf(ctx, filter)
}
- if rf, ok := ret.Get(0).(func(context.Context, []client.DocKey) *client.DeleteResult); ok {
- r0 = rf(_a0, _a1)
+ if rf, ok := ret.Get(0).(func(context.Context, interface{}) *client.DeleteResult); ok {
+ r0 = rf(ctx, filter)
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).(*client.DeleteResult)
}
}
- if rf, ok := ret.Get(1).(func(context.Context, []client.DocKey) error); ok {
- r1 = rf(_a0, _a1)
+ if rf, ok := ret.Get(1).(func(context.Context, interface{}) error); ok {
+ r1 = rf(ctx, filter)
} else {
r1 = ret.Error(1)
}
@@ -449,31 +449,31 @@ func (_m *Collection) DeleteWithKeys(_a0 context.Context, _a1 []client.DocKey) (
return r0, r1
}
-// Collection_DeleteWithKeys_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'DeleteWithKeys'
-type Collection_DeleteWithKeys_Call struct {
+// Collection_DeleteWithFilter_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'DeleteWithFilter'
+type Collection_DeleteWithFilter_Call struct {
*mock.Call
}
-// DeleteWithKeys is a helper method to define mock.On call
-// - _a0 context.Context
-// - _a1 []client.DocKey
-func (_e *Collection_Expecter) DeleteWithKeys(_a0 interface{}, _a1 interface{}) *Collection_DeleteWithKeys_Call {
- return &Collection_DeleteWithKeys_Call{Call: _e.mock.On("DeleteWithKeys", _a0, _a1)}
+// DeleteWithFilter is a helper method to define mock.On call
+// - ctx context.Context
+// - filter interface{}
+func (_e *Collection_Expecter) DeleteWithFilter(ctx interface{}, filter interface{}) *Collection_DeleteWithFilter_Call {
+ return &Collection_DeleteWithFilter_Call{Call: _e.mock.On("DeleteWithFilter", ctx, filter)}
}
-func (_c *Collection_DeleteWithKeys_Call) Run(run func(_a0 context.Context, _a1 []client.DocKey)) *Collection_DeleteWithKeys_Call {
+func (_c *Collection_DeleteWithFilter_Call) Run(run func(ctx context.Context, filter interface{})) *Collection_DeleteWithFilter_Call {
_c.Call.Run(func(args mock.Arguments) {
- run(args[0].(context.Context), args[1].([]client.DocKey))
+ run(args[0].(context.Context), args[1].(interface{}))
})
return _c
}
-func (_c *Collection_DeleteWithKeys_Call) Return(_a0 *client.DeleteResult, _a1 error) *Collection_DeleteWithKeys_Call {
+func (_c *Collection_DeleteWithFilter_Call) Return(_a0 *client.DeleteResult, _a1 error) *Collection_DeleteWithFilter_Call {
_c.Call.Return(_a0, _a1)
return _c
}
-func (_c *Collection_DeleteWithKeys_Call) RunAndReturn(run func(context.Context, []client.DocKey) (*client.DeleteResult, error)) *Collection_DeleteWithKeys_Call {
+func (_c *Collection_DeleteWithFilter_Call) RunAndReturn(run func(context.Context, interface{}) (*client.DeleteResult, error)) *Collection_DeleteWithFilter_Call {
_c.Call.Return(run)
return _c
}
@@ -563,21 +563,21 @@ func (_c *Collection_DropIndex_Call) RunAndReturn(run func(context.Context, stri
}
// Exists provides a mock function with given fields: _a0, _a1
-func (_m *Collection) Exists(_a0 context.Context, _a1 client.DocKey) (bool, error) {
+func (_m *Collection) Exists(_a0 context.Context, _a1 client.DocID) (bool, error) {
ret := _m.Called(_a0, _a1)
var r0 bool
var r1 error
- if rf, ok := ret.Get(0).(func(context.Context, client.DocKey) (bool, error)); ok {
+ if rf, ok := ret.Get(0).(func(context.Context, client.DocID) (bool, error)); ok {
return rf(_a0, _a1)
}
- if rf, ok := ret.Get(0).(func(context.Context, client.DocKey) bool); ok {
+ if rf, ok := ret.Get(0).(func(context.Context, client.DocID) bool); ok {
r0 = rf(_a0, _a1)
} else {
r0 = ret.Get(0).(bool)
}
- if rf, ok := ret.Get(1).(func(context.Context, client.DocKey) error); ok {
+ if rf, ok := ret.Get(1).(func(context.Context, client.DocID) error); ok {
r1 = rf(_a0, _a1)
} else {
r1 = ret.Error(1)
@@ -593,14 +593,14 @@ type Collection_Exists_Call struct {
// Exists is a helper method to define mock.On call
// - _a0 context.Context
-// - _a1 client.DocKey
+// - _a1 client.DocID
func (_e *Collection_Expecter) Exists(_a0 interface{}, _a1 interface{}) *Collection_Exists_Call {
return &Collection_Exists_Call{Call: _e.mock.On("Exists", _a0, _a1)}
}
-func (_c *Collection_Exists_Call) Run(run func(_a0 context.Context, _a1 client.DocKey)) *Collection_Exists_Call {
+func (_c *Collection_Exists_Call) Run(run func(_a0 context.Context, _a1 client.DocID)) *Collection_Exists_Call {
_c.Call.Run(func(args mock.Arguments) {
- run(args[0].(context.Context), args[1].(client.DocKey))
+ run(args[0].(context.Context), args[1].(client.DocID))
})
return _c
}
@@ -610,30 +610,30 @@ func (_c *Collection_Exists_Call) Return(_a0 bool, _a1 error) *Collection_Exists
return _c
}
-func (_c *Collection_Exists_Call) RunAndReturn(run func(context.Context, client.DocKey) (bool, error)) *Collection_Exists_Call {
+func (_c *Collection_Exists_Call) RunAndReturn(run func(context.Context, client.DocID) (bool, error)) *Collection_Exists_Call {
_c.Call.Return(run)
return _c
}
-// Get provides a mock function with given fields: ctx, key, showDeleted
-func (_m *Collection) Get(ctx context.Context, key client.DocKey, showDeleted bool) (*client.Document, error) {
- ret := _m.Called(ctx, key, showDeleted)
+// Get provides a mock function with given fields: ctx, docID, showDeleted
+func (_m *Collection) Get(ctx context.Context, docID client.DocID, showDeleted bool) (*client.Document, error) {
+ ret := _m.Called(ctx, docID, showDeleted)
var r0 *client.Document
var r1 error
- if rf, ok := ret.Get(0).(func(context.Context, client.DocKey, bool) (*client.Document, error)); ok {
- return rf(ctx, key, showDeleted)
+ if rf, ok := ret.Get(0).(func(context.Context, client.DocID, bool) (*client.Document, error)); ok {
+ return rf(ctx, docID, showDeleted)
}
- if rf, ok := ret.Get(0).(func(context.Context, client.DocKey, bool) *client.Document); ok {
- r0 = rf(ctx, key, showDeleted)
+ if rf, ok := ret.Get(0).(func(context.Context, client.DocID, bool) *client.Document); ok {
+ r0 = rf(ctx, docID, showDeleted)
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).(*client.Document)
}
}
- if rf, ok := ret.Get(1).(func(context.Context, client.DocKey, bool) error); ok {
- r1 = rf(ctx, key, showDeleted)
+ if rf, ok := ret.Get(1).(func(context.Context, client.DocID, bool) error); ok {
+ r1 = rf(ctx, docID, showDeleted)
} else {
r1 = ret.Error(1)
}
@@ -648,15 +648,15 @@ type Collection_Get_Call struct {
// Get is a helper method to define mock.On call
// - ctx context.Context
-// - key client.DocKey
+// - docID client.DocID
// - showDeleted bool
-func (_e *Collection_Expecter) Get(ctx interface{}, key interface{}, showDeleted interface{}) *Collection_Get_Call {
- return &Collection_Get_Call{Call: _e.mock.On("Get", ctx, key, showDeleted)}
+func (_e *Collection_Expecter) Get(ctx interface{}, docID interface{}, showDeleted interface{}) *Collection_Get_Call {
+ return &Collection_Get_Call{Call: _e.mock.On("Get", ctx, docID, showDeleted)}
}
-func (_c *Collection_Get_Call) Run(run func(ctx context.Context, key client.DocKey, showDeleted bool)) *Collection_Get_Call {
+func (_c *Collection_Get_Call) Run(run func(ctx context.Context, docID client.DocID, showDeleted bool)) *Collection_Get_Call {
_c.Call.Run(func(args mock.Arguments) {
- run(args[0].(context.Context), args[1].(client.DocKey), args[2].(bool))
+ run(args[0].(context.Context), args[1].(client.DocID), args[2].(bool))
})
return _c
}
@@ -666,25 +666,25 @@ func (_c *Collection_Get_Call) Return(_a0 *client.Document, _a1 error) *Collecti
return _c
}
-func (_c *Collection_Get_Call) RunAndReturn(run func(context.Context, client.DocKey, bool) (*client.Document, error)) *Collection_Get_Call {
+func (_c *Collection_Get_Call) RunAndReturn(run func(context.Context, client.DocID, bool) (*client.Document, error)) *Collection_Get_Call {
_c.Call.Return(run)
return _c
}
-// GetAllDocKeys provides a mock function with given fields: ctx
-func (_m *Collection) GetAllDocKeys(ctx context.Context) (<-chan client.DocKeysResult, error) {
+// GetAllDocIDs provides a mock function with given fields: ctx
+func (_m *Collection) GetAllDocIDs(ctx context.Context) (<-chan client.DocIDResult, error) {
ret := _m.Called(ctx)
- var r0 <-chan client.DocKeysResult
+ var r0 <-chan client.DocIDResult
var r1 error
- if rf, ok := ret.Get(0).(func(context.Context) (<-chan client.DocKeysResult, error)); ok {
+ if rf, ok := ret.Get(0).(func(context.Context) (<-chan client.DocIDResult, error)); ok {
return rf(ctx)
}
- if rf, ok := ret.Get(0).(func(context.Context) <-chan client.DocKeysResult); ok {
+ if rf, ok := ret.Get(0).(func(context.Context) <-chan client.DocIDResult); ok {
r0 = rf(ctx)
} else {
if ret.Get(0) != nil {
- r0 = ret.Get(0).(<-chan client.DocKeysResult)
+ r0 = ret.Get(0).(<-chan client.DocIDResult)
}
}
@@ -697,30 +697,30 @@ func (_m *Collection) GetAllDocKeys(ctx context.Context) (<-chan client.DocKeysR
return r0, r1
}
-// Collection_GetAllDocKeys_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'GetAllDocKeys'
-type Collection_GetAllDocKeys_Call struct {
+// Collection_GetAllDocIDs_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'GetAllDocIDs'
+type Collection_GetAllDocIDs_Call struct {
*mock.Call
}
-// GetAllDocKeys is a helper method to define mock.On call
+// GetAllDocIDs is a helper method to define mock.On call
// - ctx context.Context
-func (_e *Collection_Expecter) GetAllDocKeys(ctx interface{}) *Collection_GetAllDocKeys_Call {
- return &Collection_GetAllDocKeys_Call{Call: _e.mock.On("GetAllDocKeys", ctx)}
+func (_e *Collection_Expecter) GetAllDocIDs(ctx interface{}) *Collection_GetAllDocIDs_Call {
+ return &Collection_GetAllDocIDs_Call{Call: _e.mock.On("GetAllDocIDs", ctx)}
}
-func (_c *Collection_GetAllDocKeys_Call) Run(run func(ctx context.Context)) *Collection_GetAllDocKeys_Call {
+func (_c *Collection_GetAllDocIDs_Call) Run(run func(ctx context.Context)) *Collection_GetAllDocIDs_Call {
_c.Call.Run(func(args mock.Arguments) {
run(args[0].(context.Context))
})
return _c
}
-func (_c *Collection_GetAllDocKeys_Call) Return(_a0 <-chan client.DocKeysResult, _a1 error) *Collection_GetAllDocKeys_Call {
+func (_c *Collection_GetAllDocIDs_Call) Return(_a0 <-chan client.DocIDResult, _a1 error) *Collection_GetAllDocIDs_Call {
_c.Call.Return(_a0, _a1)
return _c
}
-func (_c *Collection_GetAllDocKeys_Call) RunAndReturn(run func(context.Context) (<-chan client.DocKeysResult, error)) *Collection_GetAllDocKeys_Call {
+func (_c *Collection_GetAllDocIDs_Call) RunAndReturn(run func(context.Context) (<-chan client.DocIDResult, error)) *Collection_GetAllDocIDs_Call {
_c.Call.Return(run)
return _c
}
@@ -1085,25 +1085,25 @@ func (_c *Collection_UpdateWith_Call) RunAndReturn(run func(context.Context, int
return _c
}
-// UpdateWithFilter provides a mock function with given fields: ctx, filter, updater
-func (_m *Collection) UpdateWithFilter(ctx context.Context, filter interface{}, updater string) (*client.UpdateResult, error) {
- ret := _m.Called(ctx, filter, updater)
+// UpdateWithDocID provides a mock function with given fields: ctx, docID, updater
+func (_m *Collection) UpdateWithDocID(ctx context.Context, docID client.DocID, updater string) (*client.UpdateResult, error) {
+ ret := _m.Called(ctx, docID, updater)
var r0 *client.UpdateResult
var r1 error
- if rf, ok := ret.Get(0).(func(context.Context, interface{}, string) (*client.UpdateResult, error)); ok {
- return rf(ctx, filter, updater)
+ if rf, ok := ret.Get(0).(func(context.Context, client.DocID, string) (*client.UpdateResult, error)); ok {
+ return rf(ctx, docID, updater)
}
- if rf, ok := ret.Get(0).(func(context.Context, interface{}, string) *client.UpdateResult); ok {
- r0 = rf(ctx, filter, updater)
+ if rf, ok := ret.Get(0).(func(context.Context, client.DocID, string) *client.UpdateResult); ok {
+ r0 = rf(ctx, docID, updater)
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).(*client.UpdateResult)
}
}
- if rf, ok := ret.Get(1).(func(context.Context, interface{}, string) error); ok {
- r1 = rf(ctx, filter, updater)
+ if rf, ok := ret.Get(1).(func(context.Context, client.DocID, string) error); ok {
+ r1 = rf(ctx, docID, updater)
} else {
r1 = ret.Error(1)
}
@@ -1111,55 +1111,55 @@ func (_m *Collection) UpdateWithFilter(ctx context.Context, filter interface{},
return r0, r1
}
-// Collection_UpdateWithFilter_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'UpdateWithFilter'
-type Collection_UpdateWithFilter_Call struct {
+// Collection_UpdateWithDocID_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'UpdateWithDocID'
+type Collection_UpdateWithDocID_Call struct {
*mock.Call
}
-// UpdateWithFilter is a helper method to define mock.On call
+// UpdateWithDocID is a helper method to define mock.On call
// - ctx context.Context
-// - filter interface{}
+// - docID client.DocID
// - updater string
-func (_e *Collection_Expecter) UpdateWithFilter(ctx interface{}, filter interface{}, updater interface{}) *Collection_UpdateWithFilter_Call {
- return &Collection_UpdateWithFilter_Call{Call: _e.mock.On("UpdateWithFilter", ctx, filter, updater)}
+func (_e *Collection_Expecter) UpdateWithDocID(ctx interface{}, docID interface{}, updater interface{}) *Collection_UpdateWithDocID_Call {
+ return &Collection_UpdateWithDocID_Call{Call: _e.mock.On("UpdateWithDocID", ctx, docID, updater)}
}
-func (_c *Collection_UpdateWithFilter_Call) Run(run func(ctx context.Context, filter interface{}, updater string)) *Collection_UpdateWithFilter_Call {
+func (_c *Collection_UpdateWithDocID_Call) Run(run func(ctx context.Context, docID client.DocID, updater string)) *Collection_UpdateWithDocID_Call {
_c.Call.Run(func(args mock.Arguments) {
- run(args[0].(context.Context), args[1].(interface{}), args[2].(string))
+ run(args[0].(context.Context), args[1].(client.DocID), args[2].(string))
})
return _c
}
-func (_c *Collection_UpdateWithFilter_Call) Return(_a0 *client.UpdateResult, _a1 error) *Collection_UpdateWithFilter_Call {
+func (_c *Collection_UpdateWithDocID_Call) Return(_a0 *client.UpdateResult, _a1 error) *Collection_UpdateWithDocID_Call {
_c.Call.Return(_a0, _a1)
return _c
}
-func (_c *Collection_UpdateWithFilter_Call) RunAndReturn(run func(context.Context, interface{}, string) (*client.UpdateResult, error)) *Collection_UpdateWithFilter_Call {
+func (_c *Collection_UpdateWithDocID_Call) RunAndReturn(run func(context.Context, client.DocID, string) (*client.UpdateResult, error)) *Collection_UpdateWithDocID_Call {
_c.Call.Return(run)
return _c
}
-// UpdateWithKey provides a mock function with given fields: ctx, key, updater
-func (_m *Collection) UpdateWithKey(ctx context.Context, key client.DocKey, updater string) (*client.UpdateResult, error) {
- ret := _m.Called(ctx, key, updater)
+// UpdateWithDocIDs provides a mock function with given fields: _a0, _a1, _a2
+func (_m *Collection) UpdateWithDocIDs(_a0 context.Context, _a1 []client.DocID, _a2 string) (*client.UpdateResult, error) {
+ ret := _m.Called(_a0, _a1, _a2)
var r0 *client.UpdateResult
var r1 error
- if rf, ok := ret.Get(0).(func(context.Context, client.DocKey, string) (*client.UpdateResult, error)); ok {
- return rf(ctx, key, updater)
+ if rf, ok := ret.Get(0).(func(context.Context, []client.DocID, string) (*client.UpdateResult, error)); ok {
+ return rf(_a0, _a1, _a2)
}
- if rf, ok := ret.Get(0).(func(context.Context, client.DocKey, string) *client.UpdateResult); ok {
- r0 = rf(ctx, key, updater)
+ if rf, ok := ret.Get(0).(func(context.Context, []client.DocID, string) *client.UpdateResult); ok {
+ r0 = rf(_a0, _a1, _a2)
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).(*client.UpdateResult)
}
}
- if rf, ok := ret.Get(1).(func(context.Context, client.DocKey, string) error); ok {
- r1 = rf(ctx, key, updater)
+ if rf, ok := ret.Get(1).(func(context.Context, []client.DocID, string) error); ok {
+ r1 = rf(_a0, _a1, _a2)
} else {
r1 = ret.Error(1)
}
@@ -1167,55 +1167,55 @@ func (_m *Collection) UpdateWithKey(ctx context.Context, key client.DocKey, upda
return r0, r1
}
-// Collection_UpdateWithKey_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'UpdateWithKey'
-type Collection_UpdateWithKey_Call struct {
+// Collection_UpdateWithDocIDs_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'UpdateWithDocIDs'
+type Collection_UpdateWithDocIDs_Call struct {
*mock.Call
}
-// UpdateWithKey is a helper method to define mock.On call
-// - ctx context.Context
-// - key client.DocKey
-// - updater string
-func (_e *Collection_Expecter) UpdateWithKey(ctx interface{}, key interface{}, updater interface{}) *Collection_UpdateWithKey_Call {
- return &Collection_UpdateWithKey_Call{Call: _e.mock.On("UpdateWithKey", ctx, key, updater)}
+// UpdateWithDocIDs is a helper method to define mock.On call
+// - _a0 context.Context
+// - _a1 []client.DocID
+// - _a2 string
+func (_e *Collection_Expecter) UpdateWithDocIDs(_a0 interface{}, _a1 interface{}, _a2 interface{}) *Collection_UpdateWithDocIDs_Call {
+ return &Collection_UpdateWithDocIDs_Call{Call: _e.mock.On("UpdateWithDocIDs", _a0, _a1, _a2)}
}
-func (_c *Collection_UpdateWithKey_Call) Run(run func(ctx context.Context, key client.DocKey, updater string)) *Collection_UpdateWithKey_Call {
+func (_c *Collection_UpdateWithDocIDs_Call) Run(run func(_a0 context.Context, _a1 []client.DocID, _a2 string)) *Collection_UpdateWithDocIDs_Call {
_c.Call.Run(func(args mock.Arguments) {
- run(args[0].(context.Context), args[1].(client.DocKey), args[2].(string))
+ run(args[0].(context.Context), args[1].([]client.DocID), args[2].(string))
})
return _c
}
-func (_c *Collection_UpdateWithKey_Call) Return(_a0 *client.UpdateResult, _a1 error) *Collection_UpdateWithKey_Call {
+func (_c *Collection_UpdateWithDocIDs_Call) Return(_a0 *client.UpdateResult, _a1 error) *Collection_UpdateWithDocIDs_Call {
_c.Call.Return(_a0, _a1)
return _c
}
-func (_c *Collection_UpdateWithKey_Call) RunAndReturn(run func(context.Context, client.DocKey, string) (*client.UpdateResult, error)) *Collection_UpdateWithKey_Call {
+func (_c *Collection_UpdateWithDocIDs_Call) RunAndReturn(run func(context.Context, []client.DocID, string) (*client.UpdateResult, error)) *Collection_UpdateWithDocIDs_Call {
_c.Call.Return(run)
return _c
}
-// UpdateWithKeys provides a mock function with given fields: _a0, _a1, _a2
-func (_m *Collection) UpdateWithKeys(_a0 context.Context, _a1 []client.DocKey, _a2 string) (*client.UpdateResult, error) {
- ret := _m.Called(_a0, _a1, _a2)
+// UpdateWithFilter provides a mock function with given fields: ctx, filter, updater
+func (_m *Collection) UpdateWithFilter(ctx context.Context, filter interface{}, updater string) (*client.UpdateResult, error) {
+ ret := _m.Called(ctx, filter, updater)
var r0 *client.UpdateResult
var r1 error
- if rf, ok := ret.Get(0).(func(context.Context, []client.DocKey, string) (*client.UpdateResult, error)); ok {
- return rf(_a0, _a1, _a2)
+ if rf, ok := ret.Get(0).(func(context.Context, interface{}, string) (*client.UpdateResult, error)); ok {
+ return rf(ctx, filter, updater)
}
- if rf, ok := ret.Get(0).(func(context.Context, []client.DocKey, string) *client.UpdateResult); ok {
- r0 = rf(_a0, _a1, _a2)
+ if rf, ok := ret.Get(0).(func(context.Context, interface{}, string) *client.UpdateResult); ok {
+ r0 = rf(ctx, filter, updater)
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).(*client.UpdateResult)
}
}
- if rf, ok := ret.Get(1).(func(context.Context, []client.DocKey, string) error); ok {
- r1 = rf(_a0, _a1, _a2)
+ if rf, ok := ret.Get(1).(func(context.Context, interface{}, string) error); ok {
+ r1 = rf(ctx, filter, updater)
} else {
r1 = ret.Error(1)
}
@@ -1223,32 +1223,32 @@ func (_m *Collection) UpdateWithKeys(_a0 context.Context, _a1 []client.DocKey, _
return r0, r1
}
-// Collection_UpdateWithKeys_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'UpdateWithKeys'
-type Collection_UpdateWithKeys_Call struct {
+// Collection_UpdateWithFilter_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'UpdateWithFilter'
+type Collection_UpdateWithFilter_Call struct {
*mock.Call
}
-// UpdateWithKeys is a helper method to define mock.On call
-// - _a0 context.Context
-// - _a1 []client.DocKey
-// - _a2 string
-func (_e *Collection_Expecter) UpdateWithKeys(_a0 interface{}, _a1 interface{}, _a2 interface{}) *Collection_UpdateWithKeys_Call {
- return &Collection_UpdateWithKeys_Call{Call: _e.mock.On("UpdateWithKeys", _a0, _a1, _a2)}
+// UpdateWithFilter is a helper method to define mock.On call
+// - ctx context.Context
+// - filter interface{}
+// - updater string
+func (_e *Collection_Expecter) UpdateWithFilter(ctx interface{}, filter interface{}, updater interface{}) *Collection_UpdateWithFilter_Call {
+ return &Collection_UpdateWithFilter_Call{Call: _e.mock.On("UpdateWithFilter", ctx, filter, updater)}
}
-func (_c *Collection_UpdateWithKeys_Call) Run(run func(_a0 context.Context, _a1 []client.DocKey, _a2 string)) *Collection_UpdateWithKeys_Call {
+func (_c *Collection_UpdateWithFilter_Call) Run(run func(ctx context.Context, filter interface{}, updater string)) *Collection_UpdateWithFilter_Call {
_c.Call.Run(func(args mock.Arguments) {
- run(args[0].(context.Context), args[1].([]client.DocKey), args[2].(string))
+ run(args[0].(context.Context), args[1].(interface{}), args[2].(string))
})
return _c
}
-func (_c *Collection_UpdateWithKeys_Call) Return(_a0 *client.UpdateResult, _a1 error) *Collection_UpdateWithKeys_Call {
+func (_c *Collection_UpdateWithFilter_Call) Return(_a0 *client.UpdateResult, _a1 error) *Collection_UpdateWithFilter_Call {
_c.Call.Return(_a0, _a1)
return _c
}
-func (_c *Collection_UpdateWithKeys_Call) RunAndReturn(run func(context.Context, []client.DocKey, string) (*client.UpdateResult, error)) *Collection_UpdateWithKeys_Call {
+func (_c *Collection_UpdateWithFilter_Call) RunAndReturn(run func(context.Context, interface{}, string) (*client.UpdateResult, error)) *Collection_UpdateWithFilter_Call {
_c.Call.Return(run)
return _c
}
diff --git a/client/mocks/db.go b/client/mocks/db.go
index df7b53fb5a..90dc8986d0 100644
--- a/client/mocks/db.go
+++ b/client/mocks/db.go
@@ -83,6 +83,62 @@ func (_c *DB_AddSchema_Call) RunAndReturn(run func(context.Context, string) ([]c
return _c
}
+// AddView provides a mock function with given fields: ctx, gqlQuery, sdl
+func (_m *DB) AddView(ctx context.Context, gqlQuery string, sdl string) ([]client.CollectionDefinition, error) {
+ ret := _m.Called(ctx, gqlQuery, sdl)
+
+ var r0 []client.CollectionDefinition
+ var r1 error
+ if rf, ok := ret.Get(0).(func(context.Context, string, string) ([]client.CollectionDefinition, error)); ok {
+ return rf(ctx, gqlQuery, sdl)
+ }
+ if rf, ok := ret.Get(0).(func(context.Context, string, string) []client.CollectionDefinition); ok {
+ r0 = rf(ctx, gqlQuery, sdl)
+ } else {
+ if ret.Get(0) != nil {
+ r0 = ret.Get(0).([]client.CollectionDefinition)
+ }
+ }
+
+ if rf, ok := ret.Get(1).(func(context.Context, string, string) error); ok {
+ r1 = rf(ctx, gqlQuery, sdl)
+ } else {
+ r1 = ret.Error(1)
+ }
+
+ return r0, r1
+}
+
+// DB_AddView_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'AddView'
+type DB_AddView_Call struct {
+ *mock.Call
+}
+
+// AddView is a helper method to define mock.On call
+// - ctx context.Context
+// - gqlQuery string
+// - sdl string
+func (_e *DB_Expecter) AddView(ctx interface{}, gqlQuery interface{}, sdl interface{}) *DB_AddView_Call {
+ return &DB_AddView_Call{Call: _e.mock.On("AddView", ctx, gqlQuery, sdl)}
+}
+
+func (_c *DB_AddView_Call) Run(run func(ctx context.Context, gqlQuery string, sdl string)) *DB_AddView_Call {
+ _c.Call.Run(func(args mock.Arguments) {
+ run(args[0].(context.Context), args[1].(string), args[2].(string))
+ })
+ return _c
+}
+
+func (_c *DB_AddView_Call) Return(_a0 []client.CollectionDefinition, _a1 error) *DB_AddView_Call {
+ _c.Call.Return(_a0, _a1)
+ return _c
+}
+
+func (_c *DB_AddView_Call) RunAndReturn(run func(context.Context, string, string) ([]client.CollectionDefinition, error)) *DB_AddView_Call {
+ _c.Call.Return(run)
+ return _c
+}
+
// BasicExport provides a mock function with given fields: ctx, config
func (_m *DB) BasicExport(ctx context.Context, config *client.BackupConfig) error {
ret := _m.Called(ctx, config)
@@ -464,30 +520,30 @@ func (_m *DB) GetAllSchemas(_a0 context.Context) ([]client.SchemaDescription, er
return r0, r1
}
-// DB_GetAllSchema_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'GetAllSchemas'
-type DB_GetAllSchema_Call struct {
+// DB_GetAllSchemas_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'GetAllSchemas'
+type DB_GetAllSchemas_Call struct {
*mock.Call
}
// GetAllSchemas is a helper method to define mock.On call
// - _a0 context.Context
-func (_e *DB_Expecter) GetAllSchemas(_a0 interface{}) *DB_GetAllSchema_Call {
- return &DB_GetAllSchema_Call{Call: _e.mock.On("GetAllSchemas", _a0)}
+func (_e *DB_Expecter) GetAllSchemas(_a0 interface{}) *DB_GetAllSchemas_Call {
+ return &DB_GetAllSchemas_Call{Call: _e.mock.On("GetAllSchemas", _a0)}
}
-func (_c *DB_GetAllSchema_Call) Run(run func(_a0 context.Context)) *DB_GetAllSchema_Call {
+func (_c *DB_GetAllSchemas_Call) Run(run func(_a0 context.Context)) *DB_GetAllSchemas_Call {
_c.Call.Run(func(args mock.Arguments) {
run(args[0].(context.Context))
})
return _c
}
-func (_c *DB_GetAllSchema_Call) Return(_a0 []client.SchemaDescription, _a1 error) *DB_GetAllSchema_Call {
+func (_c *DB_GetAllSchemas_Call) Return(_a0 []client.SchemaDescription, _a1 error) *DB_GetAllSchemas_Call {
_c.Call.Return(_a0, _a1)
return _c
}
-func (_c *DB_GetAllSchema_Call) RunAndReturn(run func(context.Context) ([]client.SchemaDescription, error)) *DB_GetAllSchema_Call {
+func (_c *DB_GetAllSchemas_Call) RunAndReturn(run func(context.Context) ([]client.SchemaDescription, error)) *DB_GetAllSchemas_Call {
_c.Call.Return(run)
return _c
}
@@ -657,21 +713,19 @@ func (_c *DB_GetCollectionsByVersionID_Call) RunAndReturn(run func(context.Conte
return _c
}
-// GetSchemasByName provides a mock function with given fields: _a0, _a1
-func (_m *DB) GetSchemasByName(_a0 context.Context, _a1 string) ([]client.SchemaDescription, error) {
+// GetSchemaByVersionID provides a mock function with given fields: _a0, _a1
+func (_m *DB) GetSchemaByVersionID(_a0 context.Context, _a1 string) (client.SchemaDescription, error) {
ret := _m.Called(_a0, _a1)
- var r0 []client.SchemaDescription
+ var r0 client.SchemaDescription
var r1 error
- if rf, ok := ret.Get(0).(func(context.Context, string) ([]client.SchemaDescription, error)); ok {
+ if rf, ok := ret.Get(0).(func(context.Context, string) (client.SchemaDescription, error)); ok {
return rf(_a0, _a1)
}
- if rf, ok := ret.Get(0).(func(context.Context, string) []client.SchemaDescription); ok {
+ if rf, ok := ret.Get(0).(func(context.Context, string) client.SchemaDescription); ok {
r0 = rf(_a0, _a1)
} else {
- if ret.Get(0) != nil {
- r0 = ret.Get(0).([]client.SchemaDescription)
- }
+ r0 = ret.Get(0).(client.SchemaDescription)
}
if rf, ok := ret.Get(1).(func(context.Context, string) error); ok {
@@ -683,37 +737,37 @@ func (_m *DB) GetSchemasByName(_a0 context.Context, _a1 string) ([]client.Schema
return r0, r1
}
-// DB_GetSchemaByName_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'GetSchemasByName'
-type DB_GetSchemaByName_Call struct {
+// DB_GetSchemaByVersionID_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'GetSchemaByVersionID'
+type DB_GetSchemaByVersionID_Call struct {
*mock.Call
}
-// GetSchemasByName is a helper method to define mock.On call
+// GetSchemaByVersionID is a helper method to define mock.On call
// - _a0 context.Context
// - _a1 string
-func (_e *DB_Expecter) GetSchemasByName(_a0 interface{}, _a1 interface{}) *DB_GetSchemaByName_Call {
- return &DB_GetSchemaByName_Call{Call: _e.mock.On("GetSchemasByName", _a0, _a1)}
+func (_e *DB_Expecter) GetSchemaByVersionID(_a0 interface{}, _a1 interface{}) *DB_GetSchemaByVersionID_Call {
+ return &DB_GetSchemaByVersionID_Call{Call: _e.mock.On("GetSchemaByVersionID", _a0, _a1)}
}
-func (_c *DB_GetSchemaByName_Call) Run(run func(_a0 context.Context, _a1 string)) *DB_GetSchemaByName_Call {
+func (_c *DB_GetSchemaByVersionID_Call) Run(run func(_a0 context.Context, _a1 string)) *DB_GetSchemaByVersionID_Call {
_c.Call.Run(func(args mock.Arguments) {
run(args[0].(context.Context), args[1].(string))
})
return _c
}
-func (_c *DB_GetSchemaByName_Call) Return(_a0 []client.SchemaDescription, _a1 error) *DB_GetSchemaByName_Call {
+func (_c *DB_GetSchemaByVersionID_Call) Return(_a0 client.SchemaDescription, _a1 error) *DB_GetSchemaByVersionID_Call {
_c.Call.Return(_a0, _a1)
return _c
}
-func (_c *DB_GetSchemaByName_Call) RunAndReturn(run func(context.Context, string) ([]client.SchemaDescription, error)) *DB_GetSchemaByName_Call {
+func (_c *DB_GetSchemaByVersionID_Call) RunAndReturn(run func(context.Context, string) (client.SchemaDescription, error)) *DB_GetSchemaByVersionID_Call {
_c.Call.Return(run)
return _c
}
-// GetSchemasByRoot provides a mock function with given fields: _a0, _a1
-func (_m *DB) GetSchemasByRoot(_a0 context.Context, _a1 string) ([]client.SchemaDescription, error) {
+// GetSchemasByName provides a mock function with given fields: _a0, _a1
+func (_m *DB) GetSchemasByName(_a0 context.Context, _a1 string) ([]client.SchemaDescription, error) {
ret := _m.Called(_a0, _a1)
var r0 []client.SchemaDescription
@@ -738,48 +792,50 @@ func (_m *DB) GetSchemasByRoot(_a0 context.Context, _a1 string) ([]client.Schema
return r0, r1
}
-// DB_GetSchemaByRoot_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'GetSchemasByRoot'
-type DB_GetSchemaByRoot_Call struct {
+// DB_GetSchemasByName_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'GetSchemasByName'
+type DB_GetSchemasByName_Call struct {
*mock.Call
}
-// GetSchemasByRoot is a helper method to define mock.On call
+// GetSchemasByName is a helper method to define mock.On call
// - _a0 context.Context
// - _a1 string
-func (_e *DB_Expecter) GetSchemasByRoot(_a0 interface{}, _a1 interface{}) *DB_GetSchemaByRoot_Call {
- return &DB_GetSchemaByRoot_Call{Call: _e.mock.On("GetSchemasByRoot", _a0, _a1)}
+func (_e *DB_Expecter) GetSchemasByName(_a0 interface{}, _a1 interface{}) *DB_GetSchemasByName_Call {
+ return &DB_GetSchemasByName_Call{Call: _e.mock.On("GetSchemasByName", _a0, _a1)}
}
-func (_c *DB_GetSchemaByRoot_Call) Run(run func(_a0 context.Context, _a1 string)) *DB_GetSchemaByRoot_Call {
+func (_c *DB_GetSchemasByName_Call) Run(run func(_a0 context.Context, _a1 string)) *DB_GetSchemasByName_Call {
_c.Call.Run(func(args mock.Arguments) {
run(args[0].(context.Context), args[1].(string))
})
return _c
}
-func (_c *DB_GetSchemaByRoot_Call) Return(_a0 []client.SchemaDescription, _a1 error) *DB_GetSchemaByRoot_Call {
+func (_c *DB_GetSchemasByName_Call) Return(_a0 []client.SchemaDescription, _a1 error) *DB_GetSchemasByName_Call {
_c.Call.Return(_a0, _a1)
return _c
}
-func (_c *DB_GetSchemaByRoot_Call) RunAndReturn(run func(context.Context, string) ([]client.SchemaDescription, error)) *DB_GetSchemaByRoot_Call {
+func (_c *DB_GetSchemasByName_Call) RunAndReturn(run func(context.Context, string) ([]client.SchemaDescription, error)) *DB_GetSchemasByName_Call {
_c.Call.Return(run)
return _c
}
-// GetSchemaByVersionID provides a mock function with given fields: _a0, _a1
-func (_m *DB) GetSchemaByVersionID(_a0 context.Context, _a1 string) (client.SchemaDescription, error) {
+// GetSchemasByRoot provides a mock function with given fields: _a0, _a1
+func (_m *DB) GetSchemasByRoot(_a0 context.Context, _a1 string) ([]client.SchemaDescription, error) {
ret := _m.Called(_a0, _a1)
- var r0 client.SchemaDescription
+ var r0 []client.SchemaDescription
var r1 error
- if rf, ok := ret.Get(0).(func(context.Context, string) (client.SchemaDescription, error)); ok {
+ if rf, ok := ret.Get(0).(func(context.Context, string) ([]client.SchemaDescription, error)); ok {
return rf(_a0, _a1)
}
- if rf, ok := ret.Get(0).(func(context.Context, string) client.SchemaDescription); ok {
+ if rf, ok := ret.Get(0).(func(context.Context, string) []client.SchemaDescription); ok {
r0 = rf(_a0, _a1)
} else {
- r0 = ret.Get(0).(client.SchemaDescription)
+ if ret.Get(0) != nil {
+ r0 = ret.Get(0).([]client.SchemaDescription)
+ }
}
if rf, ok := ret.Get(1).(func(context.Context, string) error); ok {
@@ -791,31 +847,31 @@ func (_m *DB) GetSchemaByVersionID(_a0 context.Context, _a1 string) (client.Sche
return r0, r1
}
-// DB_GetSchemaByVersionID_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'GetSchemaByVersionID'
-type DB_GetSchemaByVersionID_Call struct {
+// DB_GetSchemasByRoot_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'GetSchemasByRoot'
+type DB_GetSchemasByRoot_Call struct {
*mock.Call
}
-// GetSchemaByVersionID is a helper method to define mock.On call
+// GetSchemasByRoot is a helper method to define mock.On call
// - _a0 context.Context
// - _a1 string
-func (_e *DB_Expecter) GetSchemaByVersionID(_a0 interface{}, _a1 interface{}) *DB_GetSchemaByVersionID_Call {
- return &DB_GetSchemaByVersionID_Call{Call: _e.mock.On("GetSchemaByVersionID", _a0, _a1)}
+func (_e *DB_Expecter) GetSchemasByRoot(_a0 interface{}, _a1 interface{}) *DB_GetSchemasByRoot_Call {
+ return &DB_GetSchemasByRoot_Call{Call: _e.mock.On("GetSchemasByRoot", _a0, _a1)}
}
-func (_c *DB_GetSchemaByVersionID_Call) Run(run func(_a0 context.Context, _a1 string)) *DB_GetSchemaByVersionID_Call {
+func (_c *DB_GetSchemasByRoot_Call) Run(run func(_a0 context.Context, _a1 string)) *DB_GetSchemasByRoot_Call {
_c.Call.Run(func(args mock.Arguments) {
run(args[0].(context.Context), args[1].(string))
})
return _c
}
-func (_c *DB_GetSchemaByVersionID_Call) Return(_a0 client.SchemaDescription, _a1 error) *DB_GetSchemaByVersionID_Call {
+func (_c *DB_GetSchemasByRoot_Call) Return(_a0 []client.SchemaDescription, _a1 error) *DB_GetSchemasByRoot_Call {
_c.Call.Return(_a0, _a1)
return _c
}
-func (_c *DB_GetSchemaByVersionID_Call) RunAndReturn(run func(context.Context, string) (client.SchemaDescription, error)) *DB_GetSchemaByVersionID_Call {
+func (_c *DB_GetSchemasByRoot_Call) RunAndReturn(run func(context.Context, string) ([]client.SchemaDescription, error)) *DB_GetSchemasByRoot_Call {
_c.Call.Return(run)
return _c
}
diff --git a/client/request/commit.go b/client/request/commit.go
index 0715276547..ff65e20822 100644
--- a/client/request/commit.go
+++ b/client/request/commit.go
@@ -19,7 +19,7 @@ var (
type CommitSelect struct {
Field
- DocKey immutable.Option[string]
+ DocID immutable.Option[string]
FieldID immutable.Option[string]
Cid immutable.Option[string]
Depth immutable.Option[uint64]
diff --git a/client/request/consts.go b/client/request/consts.go
index 7287a49ac3..1a1d653a25 100644
--- a/client/request/consts.go
+++ b/client/request/consts.go
@@ -20,13 +20,9 @@ const (
RelatedObjectID = "_id"
Cid = "cid"
- Data = "data"
- DocKey = "dockey"
- DocKeys = "dockeys"
+ Input = "input"
FieldName = "field"
FieldIDName = "fieldId"
- Id = "id"
- Ids = "ids"
ShowDeleted = "showDeleted"
FilterClause = "filter"
@@ -36,14 +32,21 @@ const (
OrderClause = "order"
DepthClause = "depth"
+ DocIDArgName = "docID"
+ DocIDsArgName = "docIDs"
+
AverageFieldName = "_avg"
CountFieldName = "_count"
- KeyFieldName = "_key"
+ DocIDFieldName = "_docID"
GroupFieldName = "_group"
DeletedFieldName = "_deleted"
SumFieldName = "_sum"
VersionFieldName = "_version"
+ // New generated document id from a backed up document,
+ // which might have a different _docID originally.
+ NewDocIDFieldName = "_docIDNew"
+
ExplainLabel = "explain"
LatestCommitsName = "latestCommits"
@@ -53,13 +56,18 @@ const (
LinksFieldName = "links"
HeightFieldName = "height"
CidFieldName = "cid"
- DockeyFieldName = "dockey"
CollectionIDFieldName = "collectionID"
SchemaVersionIDFieldName = "schemaVersionId"
FieldNameFieldName = "fieldName"
FieldIDFieldName = "fieldId"
DeltaFieldName = "delta"
+ DeltaArgFieldName = "FieldName"
+ DeltaArgData = "Data"
+ DeltaArgSchemaVersionID = "SchemaVersionID"
+ DeltaArgPriority = "Priority"
+ DeltaArgDocID = "DocID"
+
LinksNameFieldName = "name"
LinksCidFieldName = "cid"
@@ -80,7 +88,7 @@ var (
CountFieldName: true,
SumFieldName: true,
AverageFieldName: true,
- KeyFieldName: true,
+ DocIDFieldName: true,
DeletedFieldName: true,
}
@@ -98,7 +106,7 @@ var (
VersionFields = []string{
HeightFieldName,
CidFieldName,
- DockeyFieldName,
+ DocIDArgName,
CollectionIDFieldName,
SchemaVersionIDFieldName,
FieldNameFieldName,
diff --git a/client/request/mutation.go b/client/request/mutation.go
index c7f0e07ee8..6bff180dd9 100644
--- a/client/request/mutation.go
+++ b/client/request/mutation.go
@@ -33,7 +33,7 @@ type ObjectMutation struct {
IDs immutable.Option[[]string]
Filter immutable.Option[Filter]
- Data string
+ Input map[string]any
Fields []Selection
}
@@ -46,8 +46,8 @@ func (m ObjectMutation) ToSelect() *Select {
Name: m.Collection,
Alias: m.Alias,
},
- Fields: m.Fields,
- DocKeys: m.IDs,
- Filter: m.Filter,
+ Fields: m.Fields,
+ DocIDs: m.IDs,
+ Filter: m.Filter,
}
}
diff --git a/client/request/select.go b/client/request/select.go
index fb842228aa..863bba2aeb 100644
--- a/client/request/select.go
+++ b/client/request/select.go
@@ -11,6 +11,8 @@
package request
import (
+ "encoding/json"
+
"github.com/sourcenetwork/immutable"
)
@@ -28,8 +30,8 @@ const (
type Select struct {
Field
- DocKeys immutable.Option[[]string]
- CID immutable.Option[string]
+ DocIDs immutable.Option[[]string]
+ CID immutable.Option[string]
// Root is the top level type of parsed request
Root SelectionType
@@ -107,3 +109,88 @@ func (s *Select) validateGroupBy() []error {
return result
}
+
+// selectJson is a private object used for handling json deserialization
+// of `Select` objects.
+type selectJson struct {
+ Field
+ DocIDs immutable.Option[[]string]
+ CID immutable.Option[string]
+ Root SelectionType
+ Limit immutable.Option[uint64]
+ Offset immutable.Option[uint64]
+ OrderBy immutable.Option[OrderBy]
+ GroupBy immutable.Option[GroupBy]
+ Filter immutable.Option[Filter]
+ ShowDeleted bool
+
+ // Properties above this line match the `Select` object and
+ // are deserialized using the normal/default logic.
+ // Properties below this line require custom logic in `UnmarshalJSON`
+ // in order to be deserialized correctly.
+
+ Fields []map[string]json.RawMessage
+}
+
+func (s *Select) UnmarshalJSON(bytes []byte) error {
+ var selectMap selectJson
+ err := json.Unmarshal(bytes, &selectMap)
+ if err != nil {
+ return err
+ }
+
+ s.Field = selectMap.Field
+ s.DocIDs = selectMap.DocIDs
+ s.CID = selectMap.CID
+ s.Root = selectMap.Root
+ s.Limit = selectMap.Limit
+ s.Offset = selectMap.Offset
+ s.OrderBy = selectMap.OrderBy
+ s.GroupBy = selectMap.GroupBy
+ s.Filter = selectMap.Filter
+ s.ShowDeleted = selectMap.ShowDeleted
+ s.Fields = make([]Selection, len(selectMap.Fields))
+
+ for i, field := range selectMap.Fields {
+ fieldJson, err := json.Marshal(field)
+ if err != nil {
+ return err
+ }
+
+ var fieldValue Selection
+ // We detect which concrete type each `Selection` object is by detecting
+ // non-nillable fields, if the key is present it must be of that type.
+ // They must be non-nillable as nil values may have their keys omitted from
+ // the json. This also relies on the fields being unique. We may wish to change
+ // this later to custom-serialize with a `_type` property.
+ if _, ok := field["Root"]; ok {
+ // This must be a Select, as only the `Select` type has a `Root` field
+ var fieldSelect Select
+ err := json.Unmarshal(fieldJson, &fieldSelect)
+ if err != nil {
+ return err
+ }
+ fieldValue = &fieldSelect
+ } else if _, ok := field["Targets"]; ok {
+ // This must be an Aggregate, as only the `Aggregate` type has a `Targets` field
+ var fieldAggregate Aggregate
+ err := json.Unmarshal(fieldJson, &fieldAggregate)
+ if err != nil {
+ return err
+ }
+ fieldValue = &fieldAggregate
+ } else {
+ // This must be a Field
+ var fieldField Field
+ err := json.Unmarshal(fieldJson, &fieldField)
+ if err != nil {
+ return err
+ }
+ fieldValue = &fieldField
+ }
+
+ s.Fields[i] = fieldValue
+ }
+
+ return nil
+}
diff --git a/client/request/subscription.go b/client/request/subscription.go
index c788efbb4c..bb4e01156c 100644
--- a/client/request/subscription.go
+++ b/client/request/subscription.go
@@ -30,15 +30,15 @@ type ObjectSubscription struct {
// ToSelect returns a basic Select object, with the same Name, Alias, and Fields as
// the Subscription object. Used to create a Select planNode for the event stream return objects.
-func (m ObjectSubscription) ToSelect(docKey, cid string) *Select {
+func (m ObjectSubscription) ToSelect(docID, cid string) *Select {
return &Select{
Field: Field{
Name: m.Collection,
Alias: m.Alias,
},
- DocKeys: immutable.Some([]string{docKey}),
- CID: immutable.Some(cid),
- Fields: m.Fields,
- Filter: m.Filter,
+ DocIDs: immutable.Some([]string{docID}),
+ CID: immutable.Some(cid),
+ Fields: m.Fields,
+ Filter: m.Filter,
}
}
diff --git a/client/value.go b/client/value.go
index c07f265957..ae462b74f2 100644
--- a/client/value.go
+++ b/client/value.go
@@ -12,100 +12,91 @@ package client
import (
"github.com/fxamacker/cbor/v2"
+ "github.com/sourcenetwork/immutable"
)
-// Value is an interface that points to a concrete Value implementation.
-// (TODO May collapse this down without an interface)
-type Value interface {
- Value() any
- IsDocument() bool
- Type() CType
- IsDirty() bool
- Clean()
- IsDelete() bool //todo: Update IsDelete naming
- Delete()
-}
-
-// WriteableValue defines a simple interface with a Bytes() method
-// which is used to indicate if a Value is writeable type versus
-// a composite type like a Sub-Document.
-// Writeable types include simple Strings/Ints/Floats/Binary
-// that can be loaded into a CRDT Register, Set, Counter, etc.
-type WriteableValue interface {
- Value
-
- Bytes() ([]byte, error)
-}
-
-type ReadableValue interface {
- Value
-
- Read() (any, error)
-}
-
-type simpleValue struct {
+type FieldValue struct {
t CType
value any
isDirty bool
delete bool
}
-func newValue(t CType, val any) simpleValue {
- return simpleValue{
+func NewFieldValue(t CType, val any) *FieldValue {
+ return &FieldValue{
t: t,
value: val,
isDirty: true,
}
}
-// func (val simpleValue) Set(val any)
-
-func (val simpleValue) Value() any {
+func (val FieldValue) Value() any {
return val.value
}
-func (val simpleValue) Type() CType {
+func (val FieldValue) Type() CType {
return val.t
}
-func (val simpleValue) IsDocument() bool {
+func (val FieldValue) IsDocument() bool {
_, ok := val.value.(*Document)
return ok
}
// IsDirty returns if the value is marked as dirty (unsaved/changed)
-func (val simpleValue) IsDirty() bool {
+func (val FieldValue) IsDirty() bool {
return val.isDirty
}
-func (val *simpleValue) Clean() {
+func (val *FieldValue) Clean() {
val.isDirty = false
val.delete = false
}
-func (val *simpleValue) Delete() {
+func (val *FieldValue) Delete() {
val.delete = true
val.isDirty = true
}
-func (val simpleValue) IsDelete() bool {
+func (val FieldValue) IsDelete() bool {
return val.delete
}
-type cborValue struct {
- *simpleValue
+func (val *FieldValue) SetType(t CType) {
+ val.t = t
}
-// NewCBORValue creates a new CBOR value from a CRDT type and a value.
-func NewCBORValue(t CType, val any) WriteableValue {
- return newCBORValue(t, val)
-}
+func (val FieldValue) Bytes() ([]byte, error) {
+ em, err := cbor.EncOptions{Time: cbor.TimeRFC3339}.EncMode()
+ if err != nil {
+ return nil, err
+ }
-func newCBORValue(t CType, val any) WriteableValue {
- v := newValue(t, val)
- return cborValue{&v}
+ var value any
+ switch tempVal := val.value.(type) {
+ case []immutable.Option[string]:
+ value = convertImmutable(tempVal)
+ case []immutable.Option[int64]:
+ value = convertImmutable(tempVal)
+ case []immutable.Option[float64]:
+ value = convertImmutable(tempVal)
+ case []immutable.Option[bool]:
+ value = convertImmutable(tempVal)
+ default:
+ value = val.value
+ }
+
+ return em.Marshal(value)
}
-func (v cborValue) Bytes() ([]byte, error) {
- return cbor.Marshal(v.value)
+func convertImmutable[T any](vals []immutable.Option[T]) []any {
+ var out []any
+ for _, val := range vals {
+ if !val.HasValue() {
+ out = append(out, nil)
+ continue
+ }
+ out = append(out, val.Value())
+ }
+ return out
}
diff --git a/cmd/gendocs/main.go b/cmd/gendocs/main.go
new file mode 100644
index 0000000000..44901b0faf
--- /dev/null
+++ b/cmd/gendocs/main.go
@@ -0,0 +1,34 @@
+// Copyright 2023 Democratized Data Foundation
+//
+// Use of this software is governed by the Business Source License
+// included in the file licenses/BSL.txt.
+//
+// As of the Change Date specified in that file, in accordance with
+// the Business Source License, use of this software will be governed
+// by the Apache License, Version 2.0, included in the file
+// licenses/APL.txt.
+
+/*
+gendocs is a tool to generate the collections' documents automatically.
+*/
+package main
+
+import (
+ "os"
+
+ "github.com/sourcenetwork/defradb/config"
+ "github.com/sourcenetwork/defradb/tests/gen/cli"
+)
+
+func main() {
+ conf := config.DefaultConfig()
+ gendocsCmd := cli.MakeGenDocCommand(conf)
+ if err := gendocsCmd.Execute(); err != nil {
+ // this error is okay to discard because cobra
+ // logs any errors encountered during execution
+ //
+ // exiting with a non-zero status code signals
+ // that an error has ocurred during execution
+ os.Exit(1)
+ }
+}
diff --git a/config/config.go b/config/config.go
index 24d5ee73e7..c179fc5db3 100644
--- a/config/config.go
+++ b/config/config.go
@@ -101,7 +101,14 @@ func DefaultConfig() *Config {
cfg.v.SetConfigName(DefaultConfigFileName)
cfg.v.SetConfigType(configType)
- // Load default values in viper.
+ cfg.Persist()
+
+ return cfg
+}
+
+// Persist persists manually set config parameters to the viper config.
+func (cfg *Config) Persist() {
+ // Load new values in viper.
b, err := cfg.toBytes()
if err != nil {
panic(err)
@@ -109,8 +116,6 @@ func DefaultConfig() *Config {
if err = cfg.v.ReadConfig(bytes.NewReader(b)); err != nil {
panic(NewErrReadingConfigFile(err))
}
-
- return cfg
}
// LoadWithRootdir loads a Config with parameters from defaults, config file, environment variables, and CLI flags.
diff --git a/config/configfile_test.go b/config/configfile_test.go
index a9ec874f9f..5f7aed26aa 100644
--- a/config/configfile_test.go
+++ b/config/configfile_test.go
@@ -14,6 +14,7 @@ import (
"bytes"
"os"
"path/filepath"
+ "runtime"
"testing"
"text/template"
@@ -61,6 +62,10 @@ func TestWritesConfigFileErroneousPath(t *testing.T) {
}
func TestReadConfigFileForLogger(t *testing.T) {
+ if runtime.GOOS == "windows" {
+ t.Skipf("Test is not supported on windows as it leaks resources, see https://github.com/sourcenetwork/defradb/issues/2057")
+ }
+
cfg := DefaultConfig()
tmpdir := t.TempDir()
cfg.Rootdir = tmpdir
diff --git a/core/crdt/base.go b/core/crdt/base.go
index d24b263645..a0d8b5375f 100644
--- a/core/crdt/base.go
+++ b/core/crdt/base.go
@@ -28,13 +28,26 @@ import (
type baseCRDT struct {
store datastore.DSReaderWriter
key core.DataStoreKey
+
+ // schemaVersionKey is the schema version datastore key at the time of commit.
+ //
+ // It can be used to identify the collection datastructure state at the time of commit.
+ schemaVersionKey core.CollectionSchemaVersionKey
+
+ fieldName string
}
-// @TODO paramaterize ns/suffix
-func newBaseCRDT(store datastore.DSReaderWriter, key core.DataStoreKey) baseCRDT {
+func newBaseCRDT(
+ store datastore.DSReaderWriter,
+ key core.DataStoreKey,
+ schemaVersionKey core.CollectionSchemaVersionKey,
+ fieldName string,
+) baseCRDT {
return baseCRDT{
- store: store,
- key: key,
+ store: store,
+ key: key,
+ schemaVersionKey: schemaVersionKey,
+ fieldName: fieldName,
}
}
diff --git a/core/crdt/base_test.go b/core/crdt/base_test.go
index 5fd7d9248e..c3db4af3d6 100644
--- a/core/crdt/base_test.go
+++ b/core/crdt/base_test.go
@@ -29,11 +29,11 @@ func newSeededDS() datastore.DSReaderWriter {
}
func exampleBaseCRDT() baseCRDT {
- return newBaseCRDT(newSeededDS(), core.DataStoreKey{})
+ return newBaseCRDT(newSeededDS(), core.DataStoreKey{}, core.CollectionSchemaVersionKey{}, "")
}
func TestBaseCRDTNew(t *testing.T) {
- base := newBaseCRDT(newDS(), core.DataStoreKey{})
+ base := newBaseCRDT(newDS(), core.DataStoreKey{}, core.CollectionSchemaVersionKey{}, "")
if base.store == nil {
t.Error("newBaseCRDT needs to init store")
}
@@ -41,7 +41,7 @@ func TestBaseCRDTNew(t *testing.T) {
func TestBaseCRDTvalueKey(t *testing.T) {
base := exampleBaseCRDT()
- vk := base.key.WithDocKey("mykey").WithValueFlag()
+ vk := base.key.WithDocID("mykey").WithValueFlag()
if vk.ToString() != "/v/mykey" {
t.Errorf("Incorrect valueKey. Have %v, want %v", vk.ToString(), "/v/mykey")
}
@@ -49,7 +49,7 @@ func TestBaseCRDTvalueKey(t *testing.T) {
func TestBaseCRDTprioryKey(t *testing.T) {
base := exampleBaseCRDT()
- pk := base.key.WithDocKey("mykey").WithPriorityFlag()
+ pk := base.key.WithDocID("mykey").WithPriorityFlag()
if pk.ToString() != "/p/mykey" {
t.Errorf("Incorrect priorityKey. Have %v, want %v", pk.ToString(), "/p/mykey")
}
@@ -58,13 +58,13 @@ func TestBaseCRDTprioryKey(t *testing.T) {
func TestBaseCRDTSetGetPriority(t *testing.T) {
base := exampleBaseCRDT()
ctx := context.Background()
- err := base.setPriority(ctx, base.key.WithDocKey("mykey"), 10)
+ err := base.setPriority(ctx, base.key.WithDocID("mykey"), 10)
if err != nil {
t.Errorf("baseCRDT failed to set Priority. err: %v", err)
return
}
- priority, err := base.getPriority(ctx, base.key.WithDocKey("mykey"))
+ priority, err := base.getPriority(ctx, base.key.WithDocID("mykey"))
if err != nil {
t.Errorf("baseCRDT failed to get priority. err: %v", err)
return
diff --git a/core/crdt/composite.go b/core/crdt/composite.go
index 68f7824329..6900387db3 100644
--- a/core/crdt/composite.go
+++ b/core/crdt/composite.go
@@ -29,28 +29,26 @@ import (
"github.com/sourcenetwork/defradb/errors"
)
-var (
- _ core.ReplicatedData = (*CompositeDAG)(nil)
- _ core.CompositeDelta = (*CompositeDAGDelta)(nil)
-)
-
// CompositeDAGDelta represents a delta-state update made of sub-MerkleCRDTs.
type CompositeDAGDelta struct {
+ DocID []byte
+ FieldName string
+ Priority uint64
// SchemaVersionID is the schema version datastore key at the time of commit.
//
- // It can be used to identify the collection datastructure state at time of commit.
+ // It can be used to identify the collection datastructure state at the time of commit.
SchemaVersionID string
- Priority uint64
- Data []byte
- DocKey []byte
- SubDAGs []core.DAGLink
// Status represents the status of the document. By default it is `Active`.
// Alternatively, if can be set to `Deleted`.
Status client.DocumentStatus
-
- FieldName string
+ // SubDAGS should not be marshalled as they are already
+ // stored as links in the DAG blocks. They are needed here to
+ // hold on to them for the block creation.
+ SubDAGs []core.DAGLink `json:"-"`
}
+var _ core.CompositeDelta = (*CompositeDAGDelta)(nil)
+
// GetPriority gets the current priority for this delta.
func (delta *CompositeDAGDelta) GetPriority() uint64 {
return delta.Priority
@@ -66,23 +64,18 @@ func (delta *CompositeDAGDelta) Marshal() ([]byte, error) {
h := &codec.CborHandle{}
buf := bytes.NewBuffer(nil)
enc := codec.NewEncoder(buf, h)
- err := enc.Encode(struct {
- SchemaVersionID string
- Priority uint64
- Data []byte
- DocKey []byte
- Status uint8
- FieldName string
- }{delta.SchemaVersionID, delta.Priority, delta.Data, delta.DocKey, delta.Status.UInt8(), delta.FieldName})
+ err := enc.Encode(delta)
if err != nil {
return nil, err
}
return buf.Bytes(), nil
}
-// Value returns the value of this delta.
-func (delta *CompositeDAGDelta) Value() any {
- return delta.Data
+// Unmarshal decodes the delta from CBOR.
+func (delta *CompositeDAGDelta) Unmarshal(b []byte) error {
+ h := &codec.CborHandle{}
+ dec := codec.NewDecoderBytes(b, h)
+ return dec.Decode(delta)
}
// Links returns the links for this delta.
@@ -92,55 +85,36 @@ func (delta *CompositeDAGDelta) Links() []core.DAGLink {
// CompositeDAG is a CRDT structure that is used to track a collection of sub MerkleCRDTs.
type CompositeDAG struct {
- store datastore.DSReaderWriter
- key core.DataStoreKey
- // schemaVersionKey is the schema version datastore key at the time of commit.
- //
- // It can be used to identify the collection datastructure state at time of commit.
- schemaVersionKey core.CollectionSchemaVersionKey
-
- fieldName string
+ baseCRDT
}
-var _ core.ReplicatedData = CompositeDAG{}
+var _ core.ReplicatedData = (*CompositeDAG)(nil)
func NewCompositeDAG(
store datastore.DSReaderWriter,
schemaVersionKey core.CollectionSchemaVersionKey,
- namespace core.Key,
key core.DataStoreKey,
fieldName string,
) CompositeDAG {
- return CompositeDAG{
- store: store,
- key: key,
- schemaVersionKey: schemaVersionKey,
- fieldName: fieldName,
- }
+ return CompositeDAG{newBaseCRDT(store, key, schemaVersionKey, fieldName)}
}
-// ID returns the schema ID of the composite DAG CRDT.
-func (c CompositeDAG) ID() string {
- return c.key.ToString()
-}
-
-// Value returns the schema ID of the composite DAG CRDT.
+// Value is a no-op for a CompositeDAG.
func (c CompositeDAG) Value(ctx context.Context) ([]byte, error) {
return nil, nil
}
// Set applies a delta to the composite DAG CRDT. TBD
-func (c CompositeDAG) Set(patch []byte, links []core.DAGLink) *CompositeDAGDelta {
+func (c CompositeDAG) Set(links []core.DAGLink) *CompositeDAGDelta {
// make sure the links are sorted lexicographically by CID
sort.Slice(links, func(i, j int) bool {
return strings.Compare(links[i].Cid.String(), links[j].Cid.String()) < 0
})
return &CompositeDAGDelta{
- Data: patch,
- DocKey: []byte(c.key.DocKey),
- SubDAGs: links,
- SchemaVersionID: c.schemaVersionKey.SchemaVersionId,
+ DocID: []byte(c.key.DocID),
FieldName: c.fieldName,
+ SchemaVersionID: c.schemaVersionKey.SchemaVersionId,
+ SubDAGs: links,
}
}
@@ -226,18 +200,16 @@ func (c CompositeDAG) deleteWithPrefix(ctx context.Context, key core.DataStoreKe
}
// DeltaDecode is a typed helper to extract.
-// a LWWRegDelta from a ipld.Node
+// a CompositeDAGDelta from a ipld.Node
// for now let's do cbor (quick to implement)
func (c CompositeDAG) DeltaDecode(node ipld.Node) (core.Delta, error) {
- delta := &CompositeDAGDelta{}
pbNode, ok := node.(*dag.ProtoNode)
if !ok {
return nil, client.NewErrUnexpectedType[*dag.ProtoNode]("ipld.Node", node)
}
- data := pbNode.Data()
- h := &codec.CborHandle{}
- dec := codec.NewDecoderBytes(data, h)
- err := dec.Decode(delta)
+
+ delta := &CompositeDAGDelta{}
+ err := delta.Unmarshal(pbNode.Data())
if err != nil {
return nil, err
}
diff --git a/core/crdt/lwwreg.go b/core/crdt/lwwreg.go
index 60df739319..937552d868 100644
--- a/core/crdt/lwwreg.go
+++ b/core/crdt/lwwreg.go
@@ -26,22 +26,21 @@ import (
"github.com/sourcenetwork/defradb/errors"
)
-var (
- // ensure types implements core interfaces
- _ core.ReplicatedData = (*LWWRegister)(nil)
- _ core.Delta = (*LWWRegDelta)(nil)
-)
-
// LWWRegDelta is a single delta operation for an LWWRegister
// @todo: Expand delta metadata (investigate if needed)
type LWWRegDelta struct {
+ DocID []byte
+ FieldName string
+ Priority uint64
+ // SchemaVersionID is the schema version datastore key at the time of commit.
+ //
+ // It can be used to identify the collection datastructure state at the time of commit.
SchemaVersionID string
- Priority uint64
Data []byte
- DocKey []byte
- FieldName string
}
+var _ core.Delta = (*LWWRegDelta)(nil)
+
// GetPriority gets the current priority for this delta.
func (delta *LWWRegDelta) GetPriority() uint64 {
return delta.Priority
@@ -58,36 +57,28 @@ func (delta *LWWRegDelta) Marshal() ([]byte, error) {
h := &codec.CborHandle{}
buf := bytes.NewBuffer(nil)
enc := codec.NewEncoder(buf, h)
- err := enc.Encode(struct {
- SchemaVersionID string
- Priority uint64
- Data []byte
- DocKey []byte
- FieldName string
- }{delta.SchemaVersionID, delta.Priority, delta.Data, delta.DocKey, delta.FieldName})
+ err := enc.Encode(delta)
if err != nil {
return nil, err
}
return buf.Bytes(), nil
}
-func (delta *LWWRegDelta) Value() any {
- return delta.Data
+// Unmarshal decodes the delta from CBOR.
+func (delta *LWWRegDelta) Unmarshal(b []byte) error {
+ h := &codec.CborHandle{}
+ dec := codec.NewDecoderBytes(b, h)
+ return dec.Decode(delta)
}
// LWWRegister, Last-Writer-Wins Register, is a simple CRDT type that allows set/get
// of an arbitrary data type that ensures convergence.
type LWWRegister struct {
baseCRDT
-
- // schemaVersionKey is the schema version datastore key at the time of commit.
- //
- // It can be used to identify the collection datastructure state at time of commit.
- schemaVersionKey core.CollectionSchemaVersionKey
-
- fieldName string
}
+var _ core.ReplicatedData = (*LWWRegister)(nil)
+
// NewLWWRegister returns a new instance of the LWWReg with the given ID.
func NewLWWRegister(
store datastore.DSReaderWriter,
@@ -95,15 +86,7 @@ func NewLWWRegister(
key core.DataStoreKey,
fieldName string,
) LWWRegister {
- return LWWRegister{
- baseCRDT: newBaseCRDT(store, key),
- schemaVersionKey: schemaVersionKey,
- fieldName: fieldName,
- // id: id,
- // data: data,
- // ts: ts,
- // clock: clock,
- }
+ return LWWRegister{newBaseCRDT(store, key, schemaVersionKey, fieldName)}
}
// Value gets the current register value
@@ -120,27 +103,14 @@ func (reg LWWRegister) Value(ctx context.Context) ([]byte, error) {
// Set generates a new delta with the supplied value
// RETURN DELTA
func (reg LWWRegister) Set(value []byte) *LWWRegDelta {
- // return NewLWWRegister(reg.id, value, reg.clock.Apply(), reg.clock)
return &LWWRegDelta{
Data: value,
- DocKey: []byte(reg.key.DocKey),
+ DocID: []byte(reg.key.DocID),
FieldName: reg.fieldName,
SchemaVersionID: reg.schemaVersionKey.SchemaVersionId,
}
}
-func (reg LWWRegister) ID() string {
- return reg.key.ToString()
-}
-
-// RETURN DELTA
-// func (reg LWWRegister) setWithClock(value []byte, clock Clock) LWWRegDelta {
-// // return NewLWWRegister(reg.id, value, clock.Apply(), clock)
-// return LWWRegDelta{
-// data: value,
-// }
-// }
-
// Merge implements ReplicatedData interface
// Merge two LWWRegisty based on the order of the timestamp (ts),
// if they are equal, compare IDs
@@ -196,15 +166,13 @@ func (reg LWWRegister) setValue(ctx context.Context, val []byte, priority uint64
// a LWWRegDelta from a ipld.Node
// for now let's do cbor (quick to implement)
func (reg LWWRegister) DeltaDecode(node ipld.Node) (core.Delta, error) {
- delta := &LWWRegDelta{}
pbNode, ok := node.(*dag.ProtoNode)
if !ok {
return nil, client.NewErrUnexpectedType[*dag.ProtoNode]("ipld.Node", node)
}
- data := pbNode.Data()
- h := &codec.CborHandle{}
- dec := codec.NewDecoderBytes(data, h)
- err := dec.Decode(delta)
+
+ delta := &LWWRegDelta{}
+ err := delta.Unmarshal(pbNode.Data())
if err != nil {
return nil, err
}
diff --git a/core/crdt/lwwreg_test.go b/core/crdt/lwwreg_test.go
index 2b978feb2d..5e6e1b27a4 100644
--- a/core/crdt/lwwreg_test.go
+++ b/core/crdt/lwwreg_test.go
@@ -32,7 +32,7 @@ func newMockStore() datastore.DSReaderWriter {
func setupLWWRegister() LWWRegister {
store := newMockStore()
- key := core.DataStoreKey{DocKey: "AAAA-BBBB"}
+ key := core.DataStoreKey{DocID: "AAAA-BBBB"}
return NewLWWRegister(store, core.CollectionSchemaVersionKey{}, key, "")
}
diff --git a/core/crdt/pncounter.go b/core/crdt/pncounter.go
new file mode 100644
index 0000000000..7d8b02c1a4
--- /dev/null
+++ b/core/crdt/pncounter.go
@@ -0,0 +1,218 @@
+// Copyright 2023 Democratized Data Foundation
+//
+// Use of this software is governed by the Business Source License
+// included in the file licenses/BSL.txt.
+//
+// As of the Change Date specified in that file, in accordance with
+// the Business Source License, use of this software will be governed
+// by the Apache License, Version 2.0, included in the file
+// licenses/APL.txt.
+
+package crdt
+
+import (
+ "bytes"
+ "context"
+ "crypto/rand"
+ "math"
+ "math/big"
+
+ "github.com/fxamacker/cbor/v2"
+ dag "github.com/ipfs/boxo/ipld/merkledag"
+ ds "github.com/ipfs/go-datastore"
+ ipld "github.com/ipfs/go-ipld-format"
+ "github.com/ugorji/go/codec"
+ "golang.org/x/exp/constraints"
+
+ "github.com/sourcenetwork/defradb/client"
+ "github.com/sourcenetwork/defradb/core"
+ "github.com/sourcenetwork/defradb/datastore"
+ "github.com/sourcenetwork/defradb/db/base"
+ "github.com/sourcenetwork/defradb/errors"
+)
+
+var (
+ // ensure types implements core interfaces
+ _ core.ReplicatedData = (*PNCounter[float64])(nil)
+ _ core.ReplicatedData = (*PNCounter[int64])(nil)
+ _ core.Delta = (*PNCounterDelta[float64])(nil)
+ _ core.Delta = (*PNCounterDelta[int64])(nil)
+)
+
+type Incrementable interface {
+ constraints.Integer | constraints.Float
+}
+
+// PNCounterDelta is a single delta operation for an PNCounter
+type PNCounterDelta[T Incrementable] struct {
+ DocID []byte
+ FieldName string
+ Priority uint64
+ // Nonce is an added randomly generated number that ensures
+ // that each increment operation is unique.
+ Nonce int64
+ // SchemaVersionID is the schema version datastore key at the time of commit.
+ //
+ // It can be used to identify the collection datastructure state at the time of commit.
+ SchemaVersionID string
+ Data T
+}
+
+// GetPriority gets the current priority for this delta.
+func (delta *PNCounterDelta[T]) GetPriority() uint64 {
+ return delta.Priority
+}
+
+// SetPriority will set the priority for this delta.
+func (delta *PNCounterDelta[T]) SetPriority(prio uint64) {
+ delta.Priority = prio
+}
+
+// Marshal encodes the delta using CBOR.
+func (delta *PNCounterDelta[T]) Marshal() ([]byte, error) {
+ h := &codec.CborHandle{}
+ buf := bytes.NewBuffer(nil)
+ enc := codec.NewEncoder(buf, h)
+ err := enc.Encode(delta)
+ if err != nil {
+ return nil, err
+ }
+ return buf.Bytes(), nil
+}
+
+// Unmarshal decodes the delta from CBOR.
+func (delta *PNCounterDelta[T]) Unmarshal(b []byte) error {
+ h := &codec.CborHandle{}
+ dec := codec.NewDecoderBytes(b, h)
+ return dec.Decode(delta)
+}
+
+// PNCounter, is a simple CRDT type that allows increment/decrement
+// of an Int and Float data types that ensures convergence.
+type PNCounter[T Incrementable] struct {
+ baseCRDT
+}
+
+// NewPNCounter returns a new instance of the PNCounter with the given ID.
+func NewPNCounter[T Incrementable](
+ store datastore.DSReaderWriter,
+ schemaVersionKey core.CollectionSchemaVersionKey,
+ key core.DataStoreKey,
+ fieldName string,
+) PNCounter[T] {
+ return PNCounter[T]{newBaseCRDT(store, key, schemaVersionKey, fieldName)}
+}
+
+// Value gets the current register value
+func (reg PNCounter[T]) Value(ctx context.Context) ([]byte, error) {
+ valueK := reg.key.WithValueFlag()
+ buf, err := reg.store.Get(ctx, valueK.ToDS())
+ if err != nil {
+ return nil, err
+ }
+ return buf, nil
+}
+
+// Set generates a new delta with the supplied value
+func (reg PNCounter[T]) Increment(ctx context.Context, value T) (*PNCounterDelta[T], error) {
+ // To ensure that the dag block is unique, we add a random number to the delta.
+ // This is done only on update (if the doc doesn't already exist) to ensure that the
+ // initial dag block of a document can be reproducible.
+ exists, err := reg.store.Has(ctx, reg.key.ToPrimaryDataStoreKey().ToDS())
+ if err != nil {
+ return nil, err
+ }
+ var nonce int64
+ if exists {
+ r, err := rand.Int(rand.Reader, big.NewInt(math.MaxInt64))
+ if err != nil {
+ return nil, err
+ }
+ nonce = r.Int64()
+ }
+
+ return &PNCounterDelta[T]{
+ DocID: []byte(reg.key.DocID),
+ FieldName: reg.fieldName,
+ Data: value,
+ SchemaVersionID: reg.schemaVersionKey.SchemaVersionId,
+ Nonce: nonce,
+ }, nil
+}
+
+// Merge implements ReplicatedData interface.
+// It merges two PNCounterRegisty by adding the values together.
+func (reg PNCounter[T]) Merge(ctx context.Context, delta core.Delta) error {
+ d, ok := delta.(*PNCounterDelta[T])
+ if !ok {
+ return ErrMismatchedMergeType
+ }
+
+ return reg.incrementValue(ctx, d.Data, d.GetPriority())
+}
+
+func (reg PNCounter[T]) incrementValue(ctx context.Context, value T, priority uint64) error {
+ key := reg.key.WithValueFlag()
+ marker, err := reg.store.Get(ctx, reg.key.ToPrimaryDataStoreKey().ToDS())
+ if err != nil && !errors.Is(err, ds.ErrNotFound) {
+ return err
+ }
+ if bytes.Equal(marker, []byte{base.DeletedObjectMarker}) {
+ key = key.WithDeletedFlag()
+ }
+
+ curValue, err := reg.getCurrentValue(ctx, key)
+ if err != nil {
+ return err
+ }
+
+ newValue := curValue + value
+ b, err := cbor.Marshal(newValue)
+ if err != nil {
+ return err
+ }
+
+ err = reg.store.Put(ctx, key.ToDS(), b)
+ if err != nil {
+ return NewErrFailedToStoreValue(err)
+ }
+
+ return reg.setPriority(ctx, reg.key, priority)
+}
+
+func (reg PNCounter[T]) getCurrentValue(ctx context.Context, key core.DataStoreKey) (T, error) {
+ curValue, err := reg.store.Get(ctx, key.ToDS())
+ if err != nil {
+ if errors.Is(err, ds.ErrNotFound) {
+ return 0, nil
+ }
+ return 0, err
+ }
+
+ return getNumericFromBytes[T](curValue)
+}
+
+// DeltaDecode is a typed helper to extract a PNCounterDelta from a ipld.Node
+func (reg PNCounter[T]) DeltaDecode(node ipld.Node) (core.Delta, error) {
+ pbNode, ok := node.(*dag.ProtoNode)
+ if !ok {
+ return nil, client.NewErrUnexpectedType[*dag.ProtoNode]("ipld.Node", node)
+ }
+
+ delta := &PNCounterDelta[T]{}
+ err := delta.Unmarshal(pbNode.Data())
+ if err != nil {
+ return nil, err
+ }
+
+ return delta, nil
+}
+
+func getNumericFromBytes[T Incrementable](b []byte) (T, error) {
+ var val T
+ err := cbor.Unmarshal(b, &val)
+ if err != nil {
+ return val, err
+ }
+ return val, nil
+}
diff --git a/core/delta.go b/core/delta.go
index cddf51b71c..fda7dd13ae 100644
--- a/core/delta.go
+++ b/core/delta.go
@@ -20,7 +20,7 @@ type Delta interface {
GetPriority() uint64
SetPriority(uint64)
Marshal() ([]byte, error)
- Value() any
+ Unmarshal(b []byte) error
}
// CompositeDelta represents a delta-state update to a composite CRDT.
diff --git a/core/doc.go b/core/doc.go
index 8f6700f50c..379ac79bf9 100644
--- a/core/doc.go
+++ b/core/doc.go
@@ -20,8 +20,8 @@ import (
"github.com/sourcenetwork/defradb/client/request"
)
-// DocKeyFieldIndex is the index of the key field in a document.
-const DocKeyFieldIndex int = 0
+// DocIDFieldIndex is the index of the DocID field in a document.
+const DocIDFieldIndex int = 0
// DocFields is a slice of fields in a document.
type DocFields []any
@@ -39,19 +39,19 @@ type Doc struct {
SchemaVersionID string
}
-// GetKey returns the DocKey for this document.
+// GetID returns the DocID for this document.
//
// Will panic if the document is empty.
-func (d *Doc) GetKey() string {
- key, _ := d.Fields[DocKeyFieldIndex].(string)
- return key
+func (d *Doc) GetID() string {
+ docID, _ := d.Fields[DocIDFieldIndex].(string)
+ return docID
}
-// SetKey sets the DocKey for this document.
+// SetID sets the DocID for this document.
//
// Will panic if the document has not been initialised with fields.
-func (d *Doc) SetKey(key string) {
- d.Fields[DocKeyFieldIndex] = key
+func (d *Doc) SetID(docID string) {
+ d.Fields[DocIDFieldIndex] = docID
}
// Clone returns a deep copy of this document.
@@ -180,6 +180,21 @@ func (mapping *DocumentMapping) SetFirstOfName(d *Doc, name string, value any) {
d.Fields[mapping.IndexesByName[name][0]] = value
}
+// TrySetFirstOfName overwrites the first field of this name with the given value.
+//
+// Will return false if the field does not exist, otherwise will return true.
+func (mapping *DocumentMapping) TrySetFirstOfName(d *Doc, name string, value any) bool {
+ if indexes, ok := mapping.IndexesByName[name]; ok && len(indexes) > 0 {
+ index := indexes[0]
+ // Panicing here should be impossible unless there is something very wrong in
+ // the mapper code.
+ d.Fields[index] = value
+ return true
+ }
+
+ return false
+}
+
// FirstOfName returns the value of the first field of the given name.
//
// Will panic if the field does not exist (but not if it's value is default).
diff --git a/core/encoding.go b/core/encoding.go
index ba4926ffc5..f6b46a4381 100644
--- a/core/encoding.go
+++ b/core/encoding.go
@@ -12,6 +12,7 @@ package core
import (
"fmt"
+ "time"
"github.com/sourcenetwork/immutable"
@@ -119,6 +120,11 @@ func DecodeFieldValue(fieldDesc client.FieldDescription, val any) (any, error) {
case uint:
return int64(v), nil
}
+ case client.FieldKind_DATETIME:
+ switch v := val.(type) {
+ case string:
+ return time.Parse(time.RFC3339, v)
+ }
}
}
diff --git a/core/key.go b/core/key.go
index 0a2529338a..0c038b11dd 100644
--- a/core/key.go
+++ b/core/key.go
@@ -67,7 +67,7 @@ type Key interface {
type DataStoreKey struct {
CollectionID string
InstanceType InstanceType
- DocKey string
+ DocID string
FieldId string
}
@@ -87,13 +87,13 @@ var _ Key = (*IndexDataStoreKey)(nil)
type PrimaryDataStoreKey struct {
CollectionId string
- DocKey string
+ DocID string
}
var _ Key = (*PrimaryDataStoreKey)(nil)
type HeadStoreKey struct {
- DocKey string
+ DocID string
FieldId string //can be 'C'
Cid cid.Cid
}
@@ -192,7 +192,7 @@ var _ Key = (*ReplicatorKey)(nil)
// splitting the input using '/' as a field deliminator. It assumes
// that the input string is in the following format:
//
-// /[CollectionId]/[InstanceType]/[DocKey]/[FieldId]
+// /[CollectionId]/[InstanceType]/[DocID]/[FieldId]
//
// Any properties before the above (assuming a '/' deliminator) are ignored
func NewDataStoreKey(key string) (DataStoreKey, error) {
@@ -212,7 +212,7 @@ func NewDataStoreKey(key string) (DataStoreKey, error) {
dataStoreKey.CollectionID = elements[0]
dataStoreKey.InstanceType = InstanceType(elements[1])
- dataStoreKey.DocKey = elements[2]
+ dataStoreKey.DocID = elements[2]
if numberOfElements == 4 {
dataStoreKey.FieldId = elements[3]
}
@@ -228,9 +228,9 @@ func MustNewDataStoreKey(key string) DataStoreKey {
return dsKey
}
-func DataStoreKeyFromDocKey(dockey client.DocKey) DataStoreKey {
+func DataStoreKeyFromDocID(docID client.DocID) DataStoreKey {
return DataStoreKey{
- DocKey: dockey.String(),
+ DocID: docID.String(),
}
}
@@ -238,7 +238,7 @@ func DataStoreKeyFromDocKey(dockey client.DocKey) DataStoreKey {
// splitting the input using '/' as a field deliminator. It assumes
// that the input string is in the following format:
//
-// /[DocKey]/[FieldId]/[Cid]
+// /[DocID]/[FieldId]/[Cid]
//
// Any properties before the above are ignored
func NewHeadStoreKey(key string) (HeadStoreKey, error) {
@@ -254,7 +254,7 @@ func NewHeadStoreKey(key string) (HeadStoreKey, error) {
return HeadStoreKey{
// elements[0] is empty (key has leading '/')
- DocKey: elements[1],
+ DocID: elements[1],
FieldId: elements[2],
Cid: cid,
}, nil
@@ -390,15 +390,15 @@ func (k DataStoreKey) WithDeletedFlag() DataStoreKey {
return newKey
}
-func (k DataStoreKey) WithDocKey(docKey string) DataStoreKey {
+func (k DataStoreKey) WithDocID(docID string) DataStoreKey {
newKey := k
- newKey.DocKey = docKey
+ newKey.DocID = docID
return newKey
}
func (k DataStoreKey) WithInstanceInfo(key DataStoreKey) DataStoreKey {
newKey := k
- newKey.DocKey = key.DocKey
+ newKey.DocID = key.DocID
newKey.FieldId = key.FieldId
newKey.InstanceType = key.InstanceType
return newKey
@@ -412,14 +412,14 @@ func (k DataStoreKey) WithFieldId(fieldId string) DataStoreKey {
func (k DataStoreKey) ToHeadStoreKey() HeadStoreKey {
return HeadStoreKey{
- DocKey: k.DocKey,
+ DocID: k.DocID,
FieldId: k.FieldId,
}
}
-func (k HeadStoreKey) WithDocKey(docKey string) HeadStoreKey {
+func (k HeadStoreKey) WithDocID(docID string) HeadStoreKey {
newKey := k
- newKey.DocKey = docKey
+ newKey.DocID = docID
return newKey
}
@@ -444,8 +444,8 @@ func (k DataStoreKey) ToString() string {
if k.InstanceType != "" {
result = result + "/" + string(k.InstanceType)
}
- if k.DocKey != "" {
- result = result + "/" + k.DocKey
+ if k.DocID != "" {
+ result = result + "/" + k.DocID
}
if k.FieldId != "" {
result = result + "/" + k.FieldId
@@ -464,7 +464,7 @@ func (k DataStoreKey) ToDS() ds.Key {
func (k DataStoreKey) Equal(other DataStoreKey) bool {
return k.CollectionID == other.CollectionID &&
- k.DocKey == other.DocKey &&
+ k.DocID == other.DocID &&
k.FieldId == other.FieldId &&
k.InstanceType == other.InstanceType
}
@@ -472,7 +472,7 @@ func (k DataStoreKey) Equal(other DataStoreKey) bool {
func (k DataStoreKey) ToPrimaryDataStoreKey() PrimaryDataStoreKey {
return PrimaryDataStoreKey{
CollectionId: k.CollectionID,
- DocKey: k.DocKey,
+ DocID: k.DocID,
}
}
@@ -582,7 +582,7 @@ func (k IndexDataStoreKey) Equal(other IndexDataStoreKey) bool {
func (k PrimaryDataStoreKey) ToDataStoreKey() DataStoreKey {
return DataStoreKey{
CollectionID: k.CollectionId,
- DocKey: k.DocKey,
+ DocID: k.DocID,
}
}
@@ -601,8 +601,8 @@ func (k PrimaryDataStoreKey) ToString() string {
result = result + "/" + k.CollectionId
}
result = result + PRIMARY_KEY
- if k.DocKey != "" {
- result = result + "/" + k.DocKey
+ if k.DocID != "" {
+ result = result + "/" + k.DocID
}
return result
@@ -786,8 +786,8 @@ func (k ReplicatorKey) ToDS() ds.Key {
func (k HeadStoreKey) ToString() string {
var result string
- if k.DocKey != "" {
- result = result + "/" + k.DocKey
+ if k.DocID != "" {
+ result = result + "/" + k.DocID
}
if k.FieldId != "" {
result = result + "/" + k.FieldId
@@ -817,8 +817,8 @@ func (k DataStoreKey) PrefixEnd() DataStoreKey {
newKey.FieldId = string(bytesPrefixEnd([]byte(k.FieldId)))
return newKey
}
- if k.DocKey != "" {
- newKey.DocKey = string(bytesPrefixEnd([]byte(k.DocKey)))
+ if k.DocID != "" {
+ newKey.DocID = string(bytesPrefixEnd([]byte(k.DocID)))
return newKey
}
if k.InstanceType != "" {
diff --git a/core/key_test.go b/core/key_test.go
index d22498bd8c..4984c5b14f 100644
--- a/core/key_test.go
+++ b/core/key_test.go
@@ -29,14 +29,14 @@ func TestNewDataStoreKey_ReturnsEmptyStruct_GivenEmptyString(t *testing.T) {
assert.ErrorIs(t, ErrEmptyKey, err)
}
-func TestNewDataStoreKey_ReturnsCollectionIdAndIndexIdAndDocKeyAndFieldIdAndInstanceType_GivenFourItemsWithType(
+func TestNewDataStoreKey_ReturnsCollectionIdAndIndexIdAndDocIDAndFieldIdAndInstanceType_GivenFourItemsWithType(
t *testing.T,
) {
instanceType := "anyType"
fieldId := "f1"
- docKey := "docKey"
+ docID := "docID"
collectionId := "1"
- inputString := collectionId + "/" + instanceType + "/" + docKey + "/" + fieldId
+ inputString := collectionId + "/" + instanceType + "/" + docID + "/" + fieldId
result, err := NewDataStoreKey(inputString)
if err != nil {
@@ -48,11 +48,11 @@ func TestNewDataStoreKey_ReturnsCollectionIdAndIndexIdAndDocKeyAndFieldIdAndInst
t,
DataStoreKey{
CollectionID: collectionId,
- DocKey: docKey,
+ DocID: docID,
FieldId: fieldId,
InstanceType: InstanceType(instanceType)},
result)
- assert.Equal(t, "/"+collectionId+"/"+instanceType+"/"+docKey+"/"+fieldId, resultString)
+ assert.Equal(t, "/"+collectionId+"/"+instanceType+"/"+docID+"/"+fieldId, resultString)
}
func TestNewDataStoreKey_ReturnsEmptyStruct_GivenAStringWithMissingElements(t *testing.T) {
@@ -65,9 +65,9 @@ func TestNewDataStoreKey_ReturnsEmptyStruct_GivenAStringWithMissingElements(t *t
func TestNewDataStoreKey_GivenAShortObjectMarker(t *testing.T) {
instanceType := "anyType"
- docKey := "docKey"
+ docID := "docID"
collectionId := "1"
- inputString := collectionId + "/" + instanceType + "/" + docKey
+ inputString := collectionId + "/" + instanceType + "/" + docID
result, err := NewDataStoreKey(inputString)
if err != nil {
@@ -79,18 +79,18 @@ func TestNewDataStoreKey_GivenAShortObjectMarker(t *testing.T) {
t,
DataStoreKey{
CollectionID: collectionId,
- DocKey: docKey,
+ DocID: docID,
InstanceType: InstanceType(instanceType)},
result)
- assert.Equal(t, "/"+collectionId+"/"+instanceType+"/"+docKey, resultString)
+ assert.Equal(t, "/"+collectionId+"/"+instanceType+"/"+docID, resultString)
}
func TestNewDataStoreKey_GivenAStringWithExtraPrefixes(t *testing.T) {
instanceType := "anyType"
fieldId := "f1"
- docKey := "docKey"
+ docID := "docID"
collectionId := "1"
- inputString := "/db/my_database_name/data/" + collectionId + "/" + instanceType + "/" + docKey + "/" + fieldId
+ inputString := "/db/my_database_name/data/" + collectionId + "/" + instanceType + "/" + docID + "/" + fieldId
_, err := NewDataStoreKey(inputString)
@@ -100,9 +100,9 @@ func TestNewDataStoreKey_GivenAStringWithExtraPrefixes(t *testing.T) {
func TestNewDataStoreKey_GivenAStringWithExtraSuffix(t *testing.T) {
instanceType := "anyType"
fieldId := "f1"
- docKey := "docKey"
+ docID := "docID"
collectionId := "1"
- inputString := "/db/data/" + collectionId + "/" + instanceType + "/" + docKey + "/" + fieldId + "/version_number"
+ inputString := "/db/data/" + collectionId + "/" + instanceType + "/" + docID + "/" + fieldId + "/version_number"
_, err := NewDataStoreKey(inputString)
diff --git a/core/replicated.go b/core/replicated.go
index 86d0523e42..75a72ece7f 100644
--- a/core/replicated.go
+++ b/core/replicated.go
@@ -20,7 +20,6 @@ import (
// ReplicatedData is a data type that allows concurrent writers to deterministically merge other
// replicated data so as to converge on the same state.
type ReplicatedData interface {
- ID() string
Merge(ctx context.Context, other Delta) error
DeltaDecode(node ipld.Node) (Delta, error) // possibly rename to just Decode
Value(ctx context.Context) ([]byte, error)
@@ -31,8 +30,3 @@ type PersistedReplicatedData interface {
ReplicatedData
Publish(Delta) (cid.Cid, error)
}
-
-// type EmbedableReplicatedData interface {
-// ReplicatedData
-// Apply(Operation) error
-// }
diff --git a/datastore/badger/v4/datastore_test.go b/datastore/badger/v4/datastore_test.go
index 1f4f0bb5d2..69a24981df 100644
--- a/datastore/badger/v4/datastore_test.go
+++ b/datastore/badger/v4/datastore_test.go
@@ -60,6 +60,10 @@ func TestNewDatastoreWithOptions(t *testing.T) {
s, err := NewDatastore(dir, &opt)
require.NoError(t, err)
+ defer func() {
+ err := s.Close()
+ require.NoError(t, err)
+ }()
s.Put(ctx, testKey1, testValue1)
s.Put(ctx, testKey2, testValue2)
@@ -68,6 +72,10 @@ func TestNewDatastoreWithOptions(t *testing.T) {
func TestNewBatch(t *testing.T) {
ctx := context.Background()
s := newLoadedDatastore(ctx, t)
+ defer func() {
+ err := s.Close()
+ require.NoError(t, err)
+ }()
b, err := s.Batch(ctx)
require.NoError(t, err)
@@ -77,6 +85,10 @@ func TestNewBatch(t *testing.T) {
func TestBatchOperations(t *testing.T) {
ctx := context.Background()
s := newLoadedDatastore(ctx, t)
+ defer func() {
+ err := s.Close()
+ require.NoError(t, err)
+ }()
b, err := s.Batch(ctx)
require.NoError(t, err)
@@ -154,6 +166,10 @@ func TestBatchCommitWithStoreClosed(t *testing.T) {
func TestBatchConsecutiveCommit(t *testing.T) {
ctx := context.Background()
s := newLoadedDatastore(ctx, t)
+ defer func() {
+ err := s.Close()
+ require.NoError(t, err)
+ }()
b, err := s.Batch(ctx)
require.NoError(t, err)
@@ -168,6 +184,10 @@ func TestBatchConsecutiveCommit(t *testing.T) {
func TestCollectGarbage(t *testing.T) {
ctx := context.Background()
s := newLoadedDatastore(ctx, t)
+ defer func() {
+ err := s.Close()
+ require.NoError(t, err)
+ }()
err := s.CollectGarbage(ctx)
require.NoError(t, err)
@@ -206,6 +226,10 @@ func TestConsecutiveClose(t *testing.T) {
func TestGetOperation(t *testing.T) {
ctx := context.Background()
s := newLoadedDatastore(ctx, t)
+ defer func() {
+ err := s.Close()
+ require.NoError(t, err)
+ }()
resp, err := s.Get(ctx, testKey1)
require.NoError(t, err)
@@ -225,6 +249,10 @@ func TestGetOperationWithStoreClosed(t *testing.T) {
func TestGetOperationNotFound(t *testing.T) {
ctx := context.Background()
s := newLoadedDatastore(ctx, t)
+ defer func() {
+ err := s.Close()
+ require.NoError(t, err)
+ }()
_, err := s.Get(ctx, testKey3)
require.ErrorIs(t, err, ds.ErrNotFound)
@@ -233,6 +261,10 @@ func TestGetOperationNotFound(t *testing.T) {
func TestDeleteOperation(t *testing.T) {
ctx := context.Background()
s := newLoadedDatastore(ctx, t)
+ defer func() {
+ err := s.Close()
+ require.NoError(t, err)
+ }()
err := s.Delete(ctx, testKey1)
require.NoError(t, err)
@@ -244,6 +276,10 @@ func TestDeleteOperation(t *testing.T) {
func TestDeleteOperation2(t *testing.T) {
ctx := context.Background()
s := newLoadedDatastore(ctx, t)
+ defer func() {
+ err := s.Close()
+ require.NoError(t, err)
+ }()
err := s.Put(ctx, testKey1, testValue1)
require.NoError(t, err)
@@ -269,6 +305,10 @@ func TestDeleteOperationWithStoreClosed(t *testing.T) {
func TestGetSizeOperation(t *testing.T) {
ctx := context.Background()
s := newLoadedDatastore(ctx, t)
+ defer func() {
+ err := s.Close()
+ require.NoError(t, err)
+ }()
resp, err := s.GetSize(ctx, testKey1)
require.NoError(t, err)
@@ -278,6 +318,10 @@ func TestGetSizeOperation(t *testing.T) {
func TestGetSizeOperationNotFound(t *testing.T) {
ctx := context.Background()
s := newLoadedDatastore(ctx, t)
+ defer func() {
+ err := s.Close()
+ require.NoError(t, err)
+ }()
_, err := s.GetSize(ctx, testKey3)
require.ErrorIs(t, err, ds.ErrNotFound)
@@ -297,6 +341,10 @@ func TestGetSizeOperationWithStoreClosed(t *testing.T) {
func TestHasOperation(t *testing.T) {
ctx := context.Background()
s := newLoadedDatastore(ctx, t)
+ defer func() {
+ err := s.Close()
+ require.NoError(t, err)
+ }()
resp, err := s.Has(ctx, testKey1)
require.NoError(t, err)
@@ -306,6 +354,10 @@ func TestHasOperation(t *testing.T) {
func TestHasOperationNotFound(t *testing.T) {
ctx := context.Background()
s := newLoadedDatastore(ctx, t)
+ defer func() {
+ err := s.Close()
+ require.NoError(t, err)
+ }()
resp, err := s.Has(ctx, testKey3)
require.NoError(t, err)
@@ -326,6 +378,10 @@ func TestHasOperationWithStoreClosed(t *testing.T) {
func TestPutOperation(t *testing.T) {
ctx := context.Background()
s := newLoadedDatastore(ctx, t)
+ defer func() {
+ err := s.Close()
+ require.NoError(t, err)
+ }()
err := s.Put(ctx, testKey3, testValue3)
require.NoError(t, err)
@@ -349,6 +405,10 @@ func TestPutOperationWithStoreClosed(t *testing.T) {
func TestQueryOperation(t *testing.T) {
ctx := context.Background()
s := newLoadedDatastore(ctx, t)
+ defer func() {
+ err := s.Close()
+ require.NoError(t, err)
+ }()
results, err := s.Query(ctx, dsq.Query{
Limit: 1,
@@ -379,6 +439,10 @@ func TestQueryOperationWithStoreClosed(t *testing.T) {
func TestDiskUsage(t *testing.T) {
ctx := context.Background()
s := newLoadedDatastore(ctx, t)
+ defer func() {
+ err := s.Close()
+ require.NoError(t, err)
+ }()
size, err := s.DiskUsage(ctx)
require.NoError(t, err)
@@ -399,6 +463,10 @@ func TestDiskUsageWithStoreClosed(t *testing.T) {
func TestSync(t *testing.T) {
ctx := context.Background()
s := newLoadedDatastore(ctx, t)
+ defer func() {
+ err := s.Close()
+ require.NoError(t, err)
+ }()
err := s.Sync(ctx, testKey1)
require.NoError(t, err)
}
diff --git a/datastore/memory/memory_test.go b/datastore/memory/memory_test.go
index 1dbf7a825e..8cfa26823f 100644
--- a/datastore/memory/memory_test.go
+++ b/datastore/memory/memory_test.go
@@ -475,7 +475,9 @@ func TestClearOldFlightTransactions(t *testing.T) {
s.inFlightTxn.Set(dsTxn{
dsVersion: s.getVersion(),
txnVersion: s.getVersion() + 1,
- expiresAt: time.Now(),
+ // Ensure expiresAt is before the value returned from the later call in `clearOldInFlightTxn`,
+ // in windows in particular it seems that the two `time.Now` calls can return the same value
+ expiresAt: time.Now().Add(-1 * time.Minute),
})
require.Equal(t, 1, s.inFlightTxn.Len())
diff --git a/db/backup.go b/db/backup.go
index e2958d1c96..d3a1138686 100644
--- a/db/backup.go
+++ b/db/backup.go
@@ -74,7 +74,7 @@ func (db *db) basicImport(ctx context.Context, txn datastore.Txn, filepath strin
for _, field := range col.Schema().Fields {
if field.Kind == client.FieldKind_FOREIGN_OBJECT {
if val, ok := docMap[field.Name+request.RelatedObjectID]; ok {
- if docMap["_newKey"] == val {
+ if docMap[request.NewDocIDFieldName] == val {
resetMap[field.Name+request.RelatedObjectID] = val
delete(docMap, field.Name+request.RelatedObjectID)
}
@@ -82,10 +82,10 @@ func (db *db) basicImport(ctx context.Context, txn datastore.Txn, filepath strin
}
}
- delete(docMap, "_key")
- delete(docMap, "_newKey")
+ delete(docMap, request.DocIDFieldName)
+ delete(docMap, request.NewDocIDFieldName)
- doc, err := client.NewDocFromMap(docMap)
+ doc, err := client.NewDocFromMap(docMap, col.Schema())
if err != nil {
return NewErrDocFromMap(err)
}
@@ -189,13 +189,13 @@ func (db *db) basicExport(ctx context.Context, txn datastore.Txn, config *client
return err
}
colTxn := col.WithTxn(txn)
- keysCh, err := colTxn.GetAllDocKeys(ctx)
+ docIDsCh, err := colTxn.GetAllDocIDs(ctx)
if err != nil {
return err
}
firstDoc := true
- for key := range keysCh {
+ for docResultWithID := range docIDsCh {
if firstDoc {
firstDoc = false
} else {
@@ -205,7 +205,7 @@ func (db *db) basicExport(ctx context.Context, txn datastore.Txn, config *client
return err
}
}
- doc, err := colTxn.Get(ctx, key.Key, false)
+ doc, err := colTxn.Get(ctx, docResultWithID.ID, false)
if err != nil {
return err
}
@@ -225,7 +225,7 @@ func (db *db) basicExport(ctx context.Context, txn datastore.Txn, config *client
if err != nil {
return err
}
- if foreignKey.(string) == doc.Key().String() {
+ if foreignKey.(string) == doc.ID().String() {
isSelfReference = true
refFieldName = field.Name + request.RelatedObjectID
}
@@ -234,11 +234,11 @@ func (db *db) basicExport(ctx context.Context, txn datastore.Txn, config *client
if err != nil {
return NewErrFailedToGetCollection(field.Schema, err)
}
- foreignDocKey, err := client.NewDocKeyFromString(foreignKey.(string))
+ foreignDocID, err := client.NewDocIDFromString(foreignKey.(string))
if err != nil {
return err
}
- foreignDoc, err := foreignCol.Get(ctx, foreignDocKey, false)
+ foreignDoc, err := foreignCol.Get(ctx, foreignDocID, false)
if err != nil {
err := doc.Set(field.Name+request.RelatedObjectID, nil)
if err != nil {
@@ -250,30 +250,30 @@ func (db *db) basicExport(ctx context.Context, txn datastore.Txn, config *client
return err
}
- delete(oldForeignDoc, "_key")
- if foreignDoc.Key().String() == foreignDocKey.String() {
+ delete(oldForeignDoc, request.DocIDFieldName)
+ if foreignDoc.ID().String() == foreignDocID.String() {
delete(oldForeignDoc, field.Name+request.RelatedObjectID)
}
- if foreignDoc.Key().String() == doc.Key().String() {
+ if foreignDoc.ID().String() == doc.ID().String() {
isSelfReference = true
refFieldName = field.Name + request.RelatedObjectID
}
- newForeignDoc, err := client.NewDocFromMap(oldForeignDoc)
+ newForeignDoc, err := client.NewDocFromMap(oldForeignDoc, foreignCol.Schema())
if err != nil {
return err
}
- if foreignDoc.Key().String() != doc.Key().String() {
- err = doc.Set(field.Name+request.RelatedObjectID, newForeignDoc.Key().String())
+ if foreignDoc.ID().String() != doc.ID().String() {
+ err = doc.Set(field.Name+request.RelatedObjectID, newForeignDoc.ID().String())
if err != nil {
return err
}
}
- if newForeignDoc.Key().String() != foreignDoc.Key().String() {
- keyChangeCache[foreignDoc.Key().String()] = newForeignDoc.Key().String()
+ if newForeignDoc.ID().String() != foreignDoc.ID().String() {
+ keyChangeCache[foreignDoc.ID().String()] = newForeignDoc.ID().String()
}
}
}
@@ -286,26 +286,26 @@ func (db *db) basicExport(ctx context.Context, txn datastore.Txn, config *client
return err
}
- delete(docM, "_key")
+ delete(docM, request.DocIDFieldName)
if isSelfReference {
delete(docM, refFieldName)
}
- newDoc, err := client.NewDocFromMap(docM)
+ newDoc, err := client.NewDocFromMap(docM, col.Schema())
if err != nil {
return err
}
- // newKey is needed to let the user know what will be the key of the imported document.
- docM["_newKey"] = newDoc.Key().String()
- // NewDocFromMap removes the "_key" map item so we add it back.
- docM["_key"] = doc.Key().String()
+ // a new docID is needed to let the user know what will be the docID of the imported document.
+ docM[request.NewDocIDFieldName] = newDoc.ID().String()
+ // NewDocFromMap removes the "_docID" map item so we add it back.
+ docM[request.DocIDFieldName] = doc.ID().String()
if isSelfReference {
- docM[refFieldName] = newDoc.Key().String()
+ docM[refFieldName] = newDoc.ID().String()
}
- if newDoc.Key().String() != doc.Key().String() {
- keyChangeCache[doc.Key().String()] = newDoc.Key().String()
+ if newDoc.ID().String() != doc.ID().String() {
+ keyChangeCache[doc.ID().String()] = newDoc.ID().String()
}
var b []byte
diff --git a/db/backup_test.go b/db/backup_test.go
index f0e7a6e338..093b1a1a3f 100644
--- a/db/backup_test.go
+++ b/db/backup_test.go
@@ -37,14 +37,13 @@ func TestBasicExport_WithNormalFormatting_NoError(t *testing.T) {
city: String
}`)
require.NoError(t, err)
-
- doc1, err := client.NewDocFromJSON([]byte(`{"name": "John", "age": 30}`))
+ col1, err := db.GetCollectionByName(ctx, "User")
require.NoError(t, err)
- doc2, err := client.NewDocFromJSON([]byte(`{"name": "Bob", "age": 40}`))
+ doc1, err := client.NewDocFromJSON([]byte(`{"name": "John", "age": 30}`), col1.Schema())
require.NoError(t, err)
- col1, err := db.GetCollectionByName(ctx, "User")
+ doc2, err := client.NewDocFromJSON([]byte(`{"name": "Bob", "age": 40}`), col1.Schema())
require.NoError(t, err)
err = col1.Create(ctx, doc1)
@@ -53,10 +52,10 @@ func TestBasicExport_WithNormalFormatting_NoError(t *testing.T) {
err = col1.Create(ctx, doc2)
require.NoError(t, err)
- doc3, err := client.NewDocFromJSON([]byte(`{"street": "101 Maple St", "city": "Toronto"}`))
+ col2, err := db.GetCollectionByName(ctx, "Address")
require.NoError(t, err)
- col2, err := db.GetCollectionByName(ctx, "Address")
+ doc3, err := client.NewDocFromJSON([]byte(`{"street": "101 Maple St", "city": "Toronto"}`), col2.Schema())
require.NoError(t, err)
err = col2.Create(ctx, doc3)
@@ -77,7 +76,7 @@ func TestBasicExport_WithNormalFormatting_NoError(t *testing.T) {
require.NoError(t, err)
expectedMap := map[string]any{}
- data := []byte(`{"Address":[{"_key":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","_newKey":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","city":"Toronto","street":"101 Maple St"}],"User":[{"_key":"bae-b94880d1-e6d2-542f-b9e0-5a369fafd0df","_newKey":"bae-b94880d1-e6d2-542f-b9e0-5a369fafd0df","age":40,"name":"Bob"},{"_key":"bae-e933420a-988a-56f8-8952-6c245aebd519","_newKey":"bae-e933420a-988a-56f8-8952-6c245aebd519","age":30,"name":"John"}]}`)
+ data := []byte(`{"Address":[{"_docID":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","_docIDNew":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","city":"Toronto","street":"101 Maple St"}],"User":[{"_docID":"bae-b94880d1-e6d2-542f-b9e0-5a369fafd0df","_docIDNew":"bae-b94880d1-e6d2-542f-b9e0-5a369fafd0df","age":40,"name":"Bob"},{"_docID":"bae-e933420a-988a-56f8-8952-6c245aebd519","_docIDNew":"bae-e933420a-988a-56f8-8952-6c245aebd519","age":30,"name":"John"}]}`)
err = json.Unmarshal(data, &expectedMap)
require.NoError(t, err)
require.EqualValues(t, expectedMap, fileMap)
@@ -100,13 +99,13 @@ func TestBasicExport_WithPrettyFormatting_NoError(t *testing.T) {
}`)
require.NoError(t, err)
- doc1, err := client.NewDocFromJSON([]byte(`{"name": "John", "age": 30}`))
+ col1, err := db.GetCollectionByName(ctx, "User")
require.NoError(t, err)
- doc2, err := client.NewDocFromJSON([]byte(`{"name": "Bob", "age": 40}`))
+ doc1, err := client.NewDocFromJSON([]byte(`{"name": "John", "age": 30}`), col1.Schema())
require.NoError(t, err)
- col1, err := db.GetCollectionByName(ctx, "User")
+ doc2, err := client.NewDocFromJSON([]byte(`{"name": "Bob", "age": 40}`), col1.Schema())
require.NoError(t, err)
err = col1.Create(ctx, doc1)
@@ -115,10 +114,10 @@ func TestBasicExport_WithPrettyFormatting_NoError(t *testing.T) {
err = col1.Create(ctx, doc2)
require.NoError(t, err)
- doc3, err := client.NewDocFromJSON([]byte(`{"street": "101 Maple St", "city": "Toronto"}`))
+ col2, err := db.GetCollectionByName(ctx, "Address")
require.NoError(t, err)
- col2, err := db.GetCollectionByName(ctx, "Address")
+ doc3, err := client.NewDocFromJSON([]byte(`{"street": "101 Maple St", "city": "Toronto"}`), col2.Schema())
require.NoError(t, err)
err = col2.Create(ctx, doc3)
@@ -139,7 +138,7 @@ func TestBasicExport_WithPrettyFormatting_NoError(t *testing.T) {
require.NoError(t, err)
expectedMap := map[string]any{}
- data := []byte(`{"Address":[{"_key":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","_newKey":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","city":"Toronto","street":"101 Maple St"}],"User":[{"_key":"bae-b94880d1-e6d2-542f-b9e0-5a369fafd0df","_newKey":"bae-b94880d1-e6d2-542f-b9e0-5a369fafd0df","age":40,"name":"Bob"},{"_key":"bae-e933420a-988a-56f8-8952-6c245aebd519","_newKey":"bae-e933420a-988a-56f8-8952-6c245aebd519","age":30,"name":"John"}]}`)
+ data := []byte(`{"Address":[{"_docID":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","_docIDNew":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","city":"Toronto","street":"101 Maple St"}],"User":[{"_docID":"bae-b94880d1-e6d2-542f-b9e0-5a369fafd0df","_docIDNew":"bae-b94880d1-e6d2-542f-b9e0-5a369fafd0df","age":40,"name":"Bob"},{"_docID":"bae-e933420a-988a-56f8-8952-6c245aebd519","_docIDNew":"bae-e933420a-988a-56f8-8952-6c245aebd519","age":30,"name":"John"}]}`)
err = json.Unmarshal(data, &expectedMap)
require.NoError(t, err)
require.EqualValues(t, expectedMap, fileMap)
@@ -162,13 +161,13 @@ func TestBasicExport_WithSingleCollection_NoError(t *testing.T) {
}`)
require.NoError(t, err)
- doc1, err := client.NewDocFromJSON([]byte(`{"name": "John", "age": 30}`))
+ col1, err := db.GetCollectionByName(ctx, "User")
require.NoError(t, err)
- doc2, err := client.NewDocFromJSON([]byte(`{"name": "Bob", "age": 40}`))
+ doc1, err := client.NewDocFromJSON([]byte(`{"name": "John", "age": 30}`), col1.Schema())
require.NoError(t, err)
- col1, err := db.GetCollectionByName(ctx, "User")
+ doc2, err := client.NewDocFromJSON([]byte(`{"name": "Bob", "age": 40}`), col1.Schema())
require.NoError(t, err)
err = col1.Create(ctx, doc1)
@@ -177,10 +176,10 @@ func TestBasicExport_WithSingleCollection_NoError(t *testing.T) {
err = col1.Create(ctx, doc2)
require.NoError(t, err)
- doc3, err := client.NewDocFromJSON([]byte(`{"street": "101 Maple St", "city": "Toronto"}`))
+ col2, err := db.GetCollectionByName(ctx, "Address")
require.NoError(t, err)
- col2, err := db.GetCollectionByName(ctx, "Address")
+ doc3, err := client.NewDocFromJSON([]byte(`{"street": "101 Maple St", "city": "Toronto"}`), col2.Schema())
require.NoError(t, err)
err = col2.Create(ctx, doc3)
@@ -201,7 +200,7 @@ func TestBasicExport_WithSingleCollection_NoError(t *testing.T) {
require.NoError(t, err)
expectedMap := map[string]any{}
- data := []byte(`{"Address":[{"_key":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","_newKey":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","city":"Toronto","street":"101 Maple St"}]}`)
+ data := []byte(`{"Address":[{"_docID":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","_docIDNew":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","city":"Toronto","street":"101 Maple St"}]}`)
err = json.Unmarshal(data, &expectedMap)
require.NoError(t, err)
require.EqualValues(t, expectedMap, fileMap)
@@ -225,13 +224,13 @@ func TestBasicExport_WithMultipleCollectionsAndUpdate_NoError(t *testing.T) {
}`)
require.NoError(t, err)
- doc1, err := client.NewDocFromJSON([]byte(`{"name": "John", "age": 30}`))
+ col1, err := db.GetCollectionByName(ctx, "User")
require.NoError(t, err)
- doc2, err := client.NewDocFromJSON([]byte(`{"name": "Bob", "age": 31}`))
+ doc1, err := client.NewDocFromJSON([]byte(`{"name": "John", "age": 30}`), col1.Schema())
require.NoError(t, err)
- col1, err := db.GetCollectionByName(ctx, "User")
+ doc2, err := client.NewDocFromJSON([]byte(`{"name": "Bob", "age": 31}`), col1.Schema())
require.NoError(t, err)
err = col1.Create(ctx, doc1)
@@ -240,13 +239,13 @@ func TestBasicExport_WithMultipleCollectionsAndUpdate_NoError(t *testing.T) {
err = col1.Create(ctx, doc2)
require.NoError(t, err)
- doc3, err := client.NewDocFromJSON([]byte(`{"name": "John and the sourcerers' stone", "author": "bae-e933420a-988a-56f8-8952-6c245aebd519"}`))
+ col2, err := db.GetCollectionByName(ctx, "Book")
require.NoError(t, err)
- doc4, err := client.NewDocFromJSON([]byte(`{"name": "Game of chains", "author": "bae-e933420a-988a-56f8-8952-6c245aebd519"}`))
+ doc3, err := client.NewDocFromJSON([]byte(`{"name": "John and the sourcerers' stone", "author": "bae-e933420a-988a-56f8-8952-6c245aebd519"}`), col2.Schema())
require.NoError(t, err)
- col2, err := db.GetCollectionByName(ctx, "Book")
+ doc4, err := client.NewDocFromJSON([]byte(`{"name": "Game of chains", "author": "bae-e933420a-988a-56f8-8952-6c245aebd519"}`), col2.Schema())
require.NoError(t, err)
err = col2.Create(ctx, doc3)
@@ -275,7 +274,7 @@ func TestBasicExport_WithMultipleCollectionsAndUpdate_NoError(t *testing.T) {
require.NoError(t, err)
expectedMap := map[string]any{}
- data := []byte(`{"Book":[{"_key":"bae-4399f189-138d-5d49-9e25-82e78463677b","_newKey":"bae-78a40f28-a4b8-5dca-be44-392b0f96d0ff","author_id":"bae-807ea028-6c13-5f86-a72b-46e8b715a162","name":"Game of chains"},{"_key":"bae-5cf2fec3-d8ed-50d5-8286-39109853d2da","_newKey":"bae-edeade01-2d21-5d6d-aadf-efc5a5279de5","author_id":"bae-807ea028-6c13-5f86-a72b-46e8b715a162","name":"John and the sourcerers' stone"}],"User":[{"_key":"bae-0648f44e-74e8-593b-a662-3310ec278927","_newKey":"bae-0648f44e-74e8-593b-a662-3310ec278927","age":31,"name":"Bob"},{"_key":"bae-e933420a-988a-56f8-8952-6c245aebd519","_newKey":"bae-807ea028-6c13-5f86-a72b-46e8b715a162","age":31,"name":"John"}]}`)
+ data := []byte(`{"Book":[{"_docID":"bae-4399f189-138d-5d49-9e25-82e78463677b","_docIDNew":"bae-78a40f28-a4b8-5dca-be44-392b0f96d0ff","author_id":"bae-807ea028-6c13-5f86-a72b-46e8b715a162","name":"Game of chains"},{"_docID":"bae-5cf2fec3-d8ed-50d5-8286-39109853d2da","_docIDNew":"bae-edeade01-2d21-5d6d-aadf-efc5a5279de5","author_id":"bae-807ea028-6c13-5f86-a72b-46e8b715a162","name":"John and the sourcerers' stone"}],"User":[{"_docID":"bae-0648f44e-74e8-593b-a662-3310ec278927","_docIDNew":"bae-0648f44e-74e8-593b-a662-3310ec278927","age":31,"name":"Bob"},{"_docID":"bae-e933420a-988a-56f8-8952-6c245aebd519","_docIDNew":"bae-807ea028-6c13-5f86-a72b-46e8b715a162","age":31,"name":"John"}]}`)
err = json.Unmarshal(data, &expectedMap)
require.NoError(t, err)
require.EqualValues(t, expectedMap, fileMap)
@@ -298,13 +297,13 @@ func TestBasicExport_EnsureFileOverwrite_NoError(t *testing.T) {
}`)
require.NoError(t, err)
- doc1, err := client.NewDocFromJSON([]byte(`{"name": "John", "age": 30}`))
+ col1, err := db.GetCollectionByName(ctx, "User")
require.NoError(t, err)
- doc2, err := client.NewDocFromJSON([]byte(`{"name": "Bob", "age": 40}`))
+ doc1, err := client.NewDocFromJSON([]byte(`{"name": "John", "age": 30}`), col1.Schema())
require.NoError(t, err)
- col1, err := db.GetCollectionByName(ctx, "User")
+ doc2, err := client.NewDocFromJSON([]byte(`{"name": "Bob", "age": 40}`), col1.Schema())
require.NoError(t, err)
err = col1.Create(ctx, doc1)
@@ -313,10 +312,10 @@ func TestBasicExport_EnsureFileOverwrite_NoError(t *testing.T) {
err = col1.Create(ctx, doc2)
require.NoError(t, err)
- doc3, err := client.NewDocFromJSON([]byte(`{"street": "101 Maple St", "city": "Toronto"}`))
+ col2, err := db.GetCollectionByName(ctx, "Address")
require.NoError(t, err)
- col2, err := db.GetCollectionByName(ctx, "Address")
+ doc3, err := client.NewDocFromJSON([]byte(`{"street": "101 Maple St", "city": "Toronto"}`), col2.Schema())
require.NoError(t, err)
err = col2.Create(ctx, doc3)
@@ -330,7 +329,7 @@ func TestBasicExport_EnsureFileOverwrite_NoError(t *testing.T) {
err = os.WriteFile(
filepath,
- []byte(`{"Address":[{"_key":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","_newKey":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","city":"Toronto","street":"101 Maple St"}],"User":[{"_key":"bae-b94880d1-e6d2-542f-b9e0-5a369fafd0df","_newKey":"bae-b94880d1-e6d2-542f-b9e0-5a369fafd0df","age":40,"name":"Bob"},{"_key":"bae-e933420a-988a-56f8-8952-6c245aebd519","_newKey":"bae-e933420a-988a-56f8-8952-6c245aebd519","age":30,"name":"John"}]}`),
+ []byte(`{"Address":[{"_docID":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","_docIDNew":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","city":"Toronto","street":"101 Maple St"}],"User":[{"_docID":"bae-b94880d1-e6d2-542f-b9e0-5a369fafd0df","_docIDNew":"bae-b94880d1-e6d2-542f-b9e0-5a369fafd0df","age":40,"name":"Bob"},{"_docID":"bae-e933420a-988a-56f8-8952-6c245aebd519","_docIDNew":"bae-e933420a-988a-56f8-8952-6c245aebd519","age":30,"name":"John"}]}`),
0664,
)
require.NoError(t, err)
@@ -345,7 +344,7 @@ func TestBasicExport_EnsureFileOverwrite_NoError(t *testing.T) {
require.NoError(t, err)
expectedMap := map[string]any{}
- data := []byte(`{"Address":[{"_key":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","_newKey":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","city":"Toronto","street":"101 Maple St"}]}`)
+ data := []byte(`{"Address":[{"_docID":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","_docIDNew":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","city":"Toronto","street":"101 Maple St"}]}`)
err = json.Unmarshal(data, &expectedMap)
require.NoError(t, err)
require.EqualValues(t, expectedMap, fileMap)
@@ -375,7 +374,7 @@ func TestBasicImport_WithMultipleCollectionsAndObjects_NoError(t *testing.T) {
err = os.WriteFile(
filepath,
- []byte(`{"Address":[{"_key":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","_newKey":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","city":"Toronto","street":"101 Maple St"}],"User":[{"_key":"bae-b94880d1-e6d2-542f-b9e0-5a369fafd0df","_newKey":"bae-b94880d1-e6d2-542f-b9e0-5a369fafd0df","age":40,"name":"Bob"},{"_key":"bae-e933420a-988a-56f8-8952-6c245aebd519","_newKey":"bae-e933420a-988a-56f8-8952-6c245aebd519","age":30,"name":"John"}]}`),
+ []byte(`{"Address":[{"_docID":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","_docIDNew":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","city":"Toronto","street":"101 Maple St"}],"User":[{"_docID":"bae-b94880d1-e6d2-542f-b9e0-5a369fafd0df","_docIDNew":"bae-b94880d1-e6d2-542f-b9e0-5a369fafd0df","age":40,"name":"Bob"},{"_docID":"bae-e933420a-988a-56f8-8952-6c245aebd519","_docIDNew":"bae-e933420a-988a-56f8-8952-6c245aebd519","age":30,"name":"John"}]}`),
0664,
)
require.NoError(t, err)
@@ -391,7 +390,7 @@ func TestBasicImport_WithMultipleCollectionsAndObjects_NoError(t *testing.T) {
col1, err := db.getCollectionByName(ctx, txn, "Address")
require.NoError(t, err)
- key1, err := client.NewDocKeyFromString("bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f")
+ key1, err := client.NewDocIDFromString("bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f")
require.NoError(t, err)
_, err = col1.Get(ctx, key1, false)
require.NoError(t, err)
@@ -399,12 +398,12 @@ func TestBasicImport_WithMultipleCollectionsAndObjects_NoError(t *testing.T) {
col2, err := db.getCollectionByName(ctx, txn, "User")
require.NoError(t, err)
- key2, err := client.NewDocKeyFromString("bae-b94880d1-e6d2-542f-b9e0-5a369fafd0df")
+ key2, err := client.NewDocIDFromString("bae-b94880d1-e6d2-542f-b9e0-5a369fafd0df")
require.NoError(t, err)
_, err = col2.Get(ctx, key2, false)
require.NoError(t, err)
- key3, err := client.NewDocKeyFromString("bae-e933420a-988a-56f8-8952-6c245aebd519")
+ key3, err := client.NewDocIDFromString("bae-e933420a-988a-56f8-8952-6c245aebd519")
require.NoError(t, err)
_, err = col2.Get(ctx, key3, false)
require.NoError(t, err)
@@ -434,7 +433,7 @@ func TestBasicImport_WithJSONArray_ReturnError(t *testing.T) {
err = os.WriteFile(
filepath,
- []byte(`["Address":[{"_key":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","_newKey":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","city":"Toronto","street":"101 Maple St"}],"User":[{"_key":"bae-b94880d1-e6d2-542f-b9e0-5a369fafd0df","_newKey":"bae-b94880d1-e6d2-542f-b9e0-5a369fafd0df","age":40,"name":"Bob"},{"_key":"bae-e933420a-988a-56f8-8952-6c245aebd519","_newKey":"bae-e933420a-988a-56f8-8952-6c245aebd519","age":30,"name":"John"}]]`),
+ []byte(`["Address":[{"_docID":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","_docIDNew":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","city":"Toronto","street":"101 Maple St"}],"User":[{"_docID":"bae-b94880d1-e6d2-542f-b9e0-5a369fafd0df","_docIDNew":"bae-b94880d1-e6d2-542f-b9e0-5a369fafd0df","age":40,"name":"Bob"},{"_docID":"bae-e933420a-988a-56f8-8952-6c245aebd519","_docIDNew":"bae-e933420a-988a-56f8-8952-6c245aebd519","age":30,"name":"John"}]]`),
0664,
)
require.NoError(t, err)
@@ -469,7 +468,7 @@ func TestBasicImport_WithObjectCollection_ReturnError(t *testing.T) {
err = os.WriteFile(
filepath,
- []byte(`{"Address":{"_key":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","_newKey":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","city":"Toronto","street":"101 Maple St"}}`),
+ []byte(`{"Address":{"_docID":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","_docIDNew":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","city":"Toronto","street":"101 Maple St"}}`),
0664,
)
require.NoError(t, err)
@@ -504,7 +503,7 @@ func TestBasicImport_WithInvalidFilepath_ReturnError(t *testing.T) {
err = os.WriteFile(
filepath,
- []byte(`{"Address":{"_key":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","_newKey":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","city":"Toronto","street":"101 Maple St"}}`),
+ []byte(`{"Address":{"_docID":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","_docIDNew":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","city":"Toronto","street":"101 Maple St"}}`),
0664,
)
require.NoError(t, err)
@@ -540,7 +539,7 @@ func TestBasicImport_WithInvalidCollection_ReturnError(t *testing.T) {
err = os.WriteFile(
filepath,
- []byte(`{"Addresses":{"_key":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","_newKey":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","city":"Toronto","street":"101 Maple St"}}`),
+ []byte(`{"Addresses":{"_docID":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","_docIDNew":"bae-8096f2c1-ea4c-5226-8ba5-17fc4b68ac1f","city":"Toronto","street":"101 Maple St"}}`),
0664,
)
require.NoError(t, err)
diff --git a/db/base/collection_keys.go b/db/base/collection_keys.go
index 6a762ff180..b2adc2f9e7 100644
--- a/db/base/collection_keys.go
+++ b/db/base/collection_keys.go
@@ -17,18 +17,21 @@ import (
"github.com/sourcenetwork/defradb/core"
)
-// MakeIndexPrefix generates a key prefix for the given collection/index descriptions
-func MakeCollectionKey(col client.CollectionDescription) core.DataStoreKey {
+// MakeDataStoreKeyWithCollectionDescription returns the datastore key for the given collection description.
+func MakeDataStoreKeyWithCollectionDescription(col client.CollectionDescription) core.DataStoreKey {
return core.DataStoreKey{
CollectionID: col.IDString(),
}
}
-// MakeIndexKey generates a key for the target dockey, using the collection/index description
-func MakeDocKey(col client.CollectionDescription, docKey string) core.DataStoreKey {
+// MakeDataStoreKeyWithCollectionAndDocID returns the datastore key for the given docID and collection description.
+func MakeDataStoreKeyWithCollectionAndDocID(
+ col client.CollectionDescription,
+ docID string,
+) core.DataStoreKey {
return core.DataStoreKey{
CollectionID: col.IDString(),
- DocKey: docKey,
+ DocID: docID,
}
}
@@ -41,14 +44,14 @@ func MakePrimaryIndexKeyForCRDT(
) (core.DataStoreKey, error) {
switch ctype {
case client.COMPOSITE:
- return MakeCollectionKey(c).WithInstanceInfo(key).WithFieldId(core.COMPOSITE_NAMESPACE), nil
- case client.LWW_REGISTER:
+ return MakeDataStoreKeyWithCollectionDescription(c).WithInstanceInfo(key).WithFieldId(core.COMPOSITE_NAMESPACE), nil
+ case client.LWW_REGISTER, client.PN_COUNTER:
field, ok := c.GetFieldByName(fieldName, &schema)
if !ok {
return core.DataStoreKey{}, client.NewErrFieldNotExist(fieldName)
}
- return MakeCollectionKey(c).WithInstanceInfo(key).WithFieldId(fmt.Sprint(field.ID)), nil
+ return MakeDataStoreKeyWithCollectionDescription(c).WithInstanceInfo(key).WithFieldId(fmt.Sprint(field.ID)), nil
}
return core.DataStoreKey{}, ErrInvalidCrdtType
}
diff --git a/db/collection.go b/db/collection.go
index b4586be89b..f066c1d9fe 100644
--- a/db/collection.go
+++ b/db/collection.go
@@ -17,7 +17,6 @@ import (
"strconv"
"strings"
- "github.com/fxamacker/cbor/v2"
"github.com/ipfs/go-cid"
ds "github.com/ipfs/go-datastore"
"github.com/ipfs/go-datastore/query"
@@ -34,7 +33,7 @@ import (
"github.com/sourcenetwork/defradb/errors"
"github.com/sourcenetwork/defradb/events"
"github.com/sourcenetwork/defradb/lens"
- "github.com/sourcenetwork/defradb/merkle/crdt"
+ merklecrdt "github.com/sourcenetwork/defradb/merkle/crdt"
)
var _ client.Collection = (*collection)(nil)
@@ -173,7 +172,7 @@ func (db *db) updateSchema(
if _, ok := schema.GetField(idFieldName); !ok {
schema.Fields = append(schema.Fields, client.FieldDescription{
Name: idFieldName,
- Kind: client.FieldKind_DocKey,
+ Kind: client.FieldKind_DocID,
RelationType: client.Relation_Type_INTERNAL_ID,
RelationName: field.RelationName,
})
@@ -285,7 +284,7 @@ func validateUpdateSchemaFields(
var existingField client.FieldDescription
var fieldAlreadyExists bool
if proposedField.ID != client.FieldID(0) ||
- proposedField.Name == request.KeyFieldName {
+ proposedField.Name == request.DocIDFieldName {
existingField, fieldAlreadyExists = existingFieldsByID[proposedField.ID]
}
@@ -351,8 +350,8 @@ func validateUpdateSchemaFields(
idFieldName := proposedField.Name + request.RelatedObjectID
idField, idFieldFound := proposedDesc.GetField(idFieldName)
if idFieldFound {
- if idField.Kind != client.FieldKind_DocKey {
- return false, NewErrRelationalFieldIDInvalidType(idField.Name, client.FieldKind_DocKey, idField.Kind)
+ if idField.Kind != client.FieldKind_DocID {
+ return false, NewErrRelationalFieldIDInvalidType(idField.Name, client.FieldKind_DocID, idField.Kind)
}
if idField.RelationType != client.Relation_Type_INTERNAL_ID {
@@ -437,8 +436,12 @@ func validateUpdateSchemaFields(
return false, NewErrCannotMoveField(proposedField.Name, proposedIndex, existingIndex)
}
- if proposedField.Typ != client.NONE_CRDT && proposedField.Typ != client.LWW_REGISTER {
- return false, NewErrInvalidCRDTType(proposedField.Name, proposedField.Typ)
+ if !proposedField.Typ.IsSupportedFieldCType() {
+ return false, client.NewErrInvalidCRDTType(proposedField.Name, proposedField.Typ.String())
+ }
+
+ if !proposedField.Typ.IsCompatibleWith(proposedField.Kind) {
+ return false, client.NewErrCRDTKindMismatch(proposedField.Typ.String(), proposedField.Kind.String())
}
newFieldNames[proposedField.Name] = struct{}{}
@@ -480,17 +483,7 @@ func (db *db) setDefaultSchemaVersion(
}
}
- cols, err := db.getAllCollections(ctx, txn)
- if err != nil {
- return err
- }
-
- definitions := make([]client.CollectionDefinition, len(cols))
- for i, col := range cols {
- definitions[i] = col.Definition()
- }
-
- return db.parser.SetSchema(ctx, txn, definitions)
+ return db.loadSchema(ctx, txn)
}
func (db *db) setDefaultSchemaVersionExplicit(
@@ -630,23 +623,64 @@ func (db *db) getAllCollections(ctx context.Context, txn datastore.Txn) ([]clien
return collections, nil
}
-// GetAllDocKeys returns all the document keys that exist in the collection.
+// getAllActiveDefinitions returns all queryable collection/views and any embedded schema used by them.
+func (db *db) getAllActiveDefinitions(ctx context.Context, txn datastore.Txn) ([]client.CollectionDefinition, error) {
+ cols, err := description.GetCollections(ctx, txn)
+ if err != nil {
+ return nil, err
+ }
+
+ definitions := make([]client.CollectionDefinition, len(cols))
+ for i, col := range cols {
+ schema, err := description.GetSchemaVersion(ctx, txn, col.SchemaVersionID)
+ if err != nil {
+ return nil, err
+ }
+
+ collection := db.newCollection(col, schema)
+
+ err = collection.loadIndexes(ctx, txn)
+ if err != nil {
+ return nil, err
+ }
+
+ definitions[i] = collection.Definition()
+ }
+
+ schemas, err := description.GetCollectionlessSchemas(ctx, txn)
+ if err != nil {
+ return nil, err
+ }
+
+ for _, schema := range schemas {
+ definitions = append(
+ definitions,
+ client.CollectionDefinition{
+ Schema: schema,
+ },
+ )
+ }
+
+ return definitions, nil
+}
+
+// GetAllDocIDs returns all the document IDs that exist in the collection.
//
// @todo: We probably need a lock on the collection for this kind of op since
// it hits every key and will cause Tx conflicts for concurrent Txs
-func (c *collection) GetAllDocKeys(ctx context.Context) (<-chan client.DocKeysResult, error) {
+func (c *collection) GetAllDocIDs(ctx context.Context) (<-chan client.DocIDResult, error) {
txn, err := c.getTxn(ctx, true)
if err != nil {
return nil, err
}
- return c.getAllDocKeysChan(ctx, txn)
+ return c.getAllDocIDsChan(ctx, txn)
}
-func (c *collection) getAllDocKeysChan(
+func (c *collection) getAllDocIDsChan(
ctx context.Context,
txn datastore.Txn,
-) (<-chan client.DocKeysResult, error) {
+) (<-chan client.DocIDResult, error) {
prefix := core.PrimaryDataStoreKey{ // empty path for all keys prefix
CollectionId: fmt.Sprint(c.ID()),
}
@@ -658,11 +692,11 @@ func (c *collection) getAllDocKeysChan(
return nil, err
}
- resCh := make(chan client.DocKeysResult)
+ resCh := make(chan client.DocIDResult)
go func() {
defer func() {
if err := q.Close(); err != nil {
- log.ErrorE(ctx, "Failed to close AllDocKeys query", err)
+ log.ErrorE(ctx, errFailedtoCloseQueryReqAllIDs, err)
}
close(resCh)
c.discardImplicitTxn(ctx, txn)
@@ -677,23 +711,22 @@ func (c *collection) getAllDocKeysChan(
// noop, just continue on the with the for loop
}
if res.Error != nil {
- resCh <- client.DocKeysResult{
+ resCh <- client.DocIDResult{
Err: res.Error,
}
return
}
- // now we have a doc key
- rawDocKey := ds.NewKey(res.Key).BaseNamespace()
- key, err := client.NewDocKeyFromString(rawDocKey)
+ rawDocID := ds.NewKey(res.Key).BaseNamespace()
+ docID, err := client.NewDocIDFromString(rawDocID)
if err != nil {
- resCh <- client.DocKeysResult{
+ resCh <- client.DocIDResult{
Err: res.Error,
}
return
}
- resCh <- client.DocKeysResult{
- Key: key,
+ resCh <- client.DocIDResult{
+ ID: docID,
}
}
}()
@@ -742,7 +775,7 @@ func (c *collection) WithTxn(txn datastore.Txn) client.Collection {
}
// Create a new document.
-// Will verify the DocKey/CID to ensure that the new document is correctly formatted.
+// Will verify the DocID/CID to ensure that the new document is correctly formatted.
func (c *collection) Create(ctx context.Context, doc *client.Document) error {
txn, err := c.getTxn(ctx, false)
if err != nil {
@@ -758,7 +791,7 @@ func (c *collection) Create(ctx context.Context, doc *client.Document) error {
}
// CreateMany creates a collection of documents at once.
-// Will verify the DocKey/CID to ensure that the new documents are correctly formatted.
+// Will verify the DocID/CID to ensure that the new documents are correctly formatted.
func (c *collection) CreateMany(ctx context.Context, docs []*client.Document) error {
txn, err := c.getTxn(ctx, false)
if err != nil {
@@ -775,29 +808,24 @@ func (c *collection) CreateMany(ctx context.Context, docs []*client.Document) er
return c.commitImplicitTxn(ctx, txn)
}
-func (c *collection) getKeysFromDoc(
+func (c *collection) getDocIDAndPrimaryKeyFromDoc(
doc *client.Document,
-) (client.DocKey, core.PrimaryDataStoreKey, error) {
- docKey, err := doc.GenerateDocKey()
+) (client.DocID, core.PrimaryDataStoreKey, error) {
+ docID, err := doc.GenerateDocID()
if err != nil {
- return client.DocKey{}, core.PrimaryDataStoreKey{}, err
+ return client.DocID{}, core.PrimaryDataStoreKey{}, err
}
- primaryKey := c.getPrimaryKeyFromDocKey(docKey)
- if primaryKey.DocKey != doc.Key().String() {
- return client.DocKey{}, core.PrimaryDataStoreKey{},
- NewErrDocVerification(doc.Key().String(), primaryKey.DocKey)
+ primaryKey := c.getPrimaryKeyFromDocID(docID)
+ if primaryKey.DocID != doc.ID().String() {
+ return client.DocID{}, core.PrimaryDataStoreKey{},
+ NewErrDocVerification(doc.ID().String(), primaryKey.DocID)
}
- return docKey, primaryKey, nil
+ return docID, primaryKey, nil
}
func (c *collection) create(ctx context.Context, txn datastore.Txn, doc *client.Document) error {
- // This has to be done before dockey verification happens in the next step.
- if err := doc.RemapAliasFieldsAndDockey(c.Schema().Fields); err != nil {
- return err
- }
-
- dockey, primaryKey, err := c.getKeysFromDoc(doc)
+ docID, primaryKey, err := c.getDocIDAndPrimaryKeyFromDoc(doc)
if err != nil {
return err
}
@@ -808,15 +836,15 @@ func (c *collection) create(ctx context.Context, txn datastore.Txn, doc *client.
return err
}
if exists {
- return NewErrDocumentAlreadyExists(primaryKey.DocKey)
+ return NewErrDocumentAlreadyExists(primaryKey.DocID)
}
if isDeleted {
- return NewErrDocumentDeleted(primaryKey.DocKey)
+ return NewErrDocumentDeleted(primaryKey.DocID)
}
// write value object marker if we have an empty doc
if len(doc.Values()) == 0 {
- valueKey := c.getDSKeyFromDockey(dockey)
+ valueKey := c.getDataStoreKeyFromDocID(docID)
err = txn.Datastore().Put(ctx, valueKey.ToDS(), []byte{base.ObjectMarker})
if err != nil {
return err
@@ -842,7 +870,7 @@ func (c *collection) Update(ctx context.Context, doc *client.Document) error {
}
defer c.discardImplicitTxn(ctx, txn)
- primaryKey := c.getPrimaryKeyFromDocKey(doc.Key())
+ primaryKey := c.getPrimaryKeyFromDocID(doc.ID())
exists, isDeleted, err := c.exists(ctx, txn, primaryKey)
if err != nil {
return err
@@ -851,7 +879,7 @@ func (c *collection) Update(ctx context.Context, doc *client.Document) error {
return client.ErrDocumentNotFound
}
if isDeleted {
- return NewErrDocumentDeleted(primaryKey.DocKey)
+ return NewErrDocumentDeleted(primaryKey.DocID)
}
err = c.update(ctx, txn, doc)
@@ -862,7 +890,7 @@ func (c *collection) Update(ctx context.Context, doc *client.Document) error {
return c.commitImplicitTxn(ctx, txn)
}
-// Contract: DB Exists check is already performed, and a doc with the given key exists.
+// Contract: DB Exists check is already performed, and a doc with the given ID exists.
// Note: Should we CompareAndSet the update, IE: Query(read-only) the state, and update if changed
// or, just update everything regardless.
// Should probably be smart about the update due to the MerkleCRDT overhead, shouldn't
@@ -884,15 +912,15 @@ func (c *collection) Save(ctx context.Context, doc *client.Document) error {
}
defer c.discardImplicitTxn(ctx, txn)
- // Check if document already exists with key
- primaryKey := c.getPrimaryKeyFromDocKey(doc.Key())
+ // Check if document already exists with primary DS key.
+ primaryKey := c.getPrimaryKeyFromDocID(doc.ID())
exists, isDeleted, err := c.exists(ctx, txn, primaryKey)
if err != nil {
return err
}
if isDeleted {
- return NewErrDocumentDeleted(doc.Key().String())
+ return NewErrDocumentDeleted(doc.ID().String())
}
if exists {
@@ -933,9 +961,8 @@ func (c *collection) save(
// Loop through doc values
// => instantiate MerkleCRDT objects
// => Set/Publish new CRDT values
- primaryKey := c.getPrimaryKeyFromDocKey(doc.Key())
+ primaryKey := c.getPrimaryKeyFromDocID(doc.ID())
links := make([]core.DAGLink, 0)
- docProperties := make(map[string]any)
for k, v := range doc.Fields() {
val, err := doc.GetValueWithField(v)
if err != nil {
@@ -954,11 +981,15 @@ func (c *collection) save(
return cid.Undef, client.NewErrFieldNotExist(k)
}
+ // by default the type will have been set to LWW_REGISTER. We need to ensure
+ // that it's set to the same as the field description CRDT type.
+ val.SetType(fieldDescription.Typ)
+
relationFieldDescription, isSecondaryRelationID := c.isSecondaryIDField(fieldDescription)
if isSecondaryRelationID {
primaryId := val.Value().(string)
- err = c.patchPrimaryDoc(ctx, txn, c.Name(), relationFieldDescription, primaryKey.DocKey, primaryId)
+ err = c.patchPrimaryDoc(ctx, txn, c.Name(), relationFieldDescription, primaryKey.DocID, primaryId)
if err != nil {
return cid.Undef, err
}
@@ -968,19 +999,26 @@ func (c *collection) save(
continue
}
- err = c.validateOneToOneLinkDoesntAlreadyExist(ctx, txn, doc.Key().String(), fieldDescription, val.Value())
+ err = c.validateOneToOneLinkDoesntAlreadyExist(ctx, txn, doc.ID().String(), fieldDescription, val.Value())
if err != nil {
return cid.Undef, err
}
- node, _, err := c.saveDocValue(ctx, txn, fieldKey, val)
+ merkleCRDT, err := merklecrdt.InstanceWithStore(
+ txn,
+ core.NewCollectionSchemaVersionKey(c.Schema().VersionID, c.ID()),
+ val.Type(),
+ fieldDescription.Kind,
+ fieldKey,
+ fieldDescription.Name,
+ )
if err != nil {
return cid.Undef, err
}
- if val.IsDelete() {
- docProperties[k] = nil
- } else {
- docProperties[k] = val.Value()
+
+ node, _, err := merkleCRDT.Save(ctx, val)
+ if err != nil {
+ return cid.Undef, err
}
link := core.DAGLink{
@@ -990,22 +1028,11 @@ func (c *collection) save(
links = append(links, link)
}
}
- // Update CompositeDAG
- em, err := cbor.CanonicalEncOptions().EncMode()
- if err != nil {
- return cid.Undef, err
- }
- buf, err := em.Marshal(docProperties)
- if err != nil {
- return cid.Undef, nil
- }
- headNode, priority, err := c.saveValueToMerkleCRDT(
+ headNode, priority, err := c.saveCompositeToMerkleCRDT(
ctx,
txn,
primaryKey.ToDataStoreKey(),
- client.COMPOSITE,
- buf,
links,
client.Active,
)
@@ -1018,7 +1045,7 @@ func (c *collection) save(
func() {
c.db.events.Updates.Value().Publish(
events.Update{
- DocKey: doc.Key().String(),
+ DocID: doc.ID().String(),
Cid: headNode.Cid(),
SchemaRoot: c.Schema().Root,
Block: headNode,
@@ -1039,7 +1066,7 @@ func (c *collection) save(
func (c *collection) validateOneToOneLinkDoesntAlreadyExist(
ctx context.Context,
txn datastore.Txn,
- docKey string,
+ docID string,
fieldDescription client.FieldDescription,
value any,
) error {
@@ -1061,8 +1088,8 @@ func (c *collection) validateOneToOneLinkDoesntAlreadyExist(
filter := fmt.Sprintf(
`{_and: [{%s: {_ne: "%s"}}, {%s: {_eq: "%s"}}]}`,
- request.KeyFieldName,
- docKey,
+ request.DocIDFieldName,
+ docID,
fieldDescription.Name,
value,
)
@@ -1103,7 +1130,7 @@ func (c *collection) validateOneToOneLinkDoesntAlreadyExist(
if err != nil {
return err
}
- return NewErrOneOneAlreadyLinked(docKey, existingDocument.GetKey(), objFieldDescription.RelationName)
+ return NewErrOneOneAlreadyLinked(docID, existingDocument.GetID(), objFieldDescription.RelationName)
}
err = selectionPlan.Close()
@@ -1114,18 +1141,18 @@ func (c *collection) validateOneToOneLinkDoesntAlreadyExist(
return nil
}
-// Delete will attempt to delete a document by key will return true if a deletion is successful,
-// and return false, along with an error, if it cannot.
+// Delete will attempt to delete a document by docID and return true if a deletion is successful,
+// otherwise will return false, along with an error, if it cannot.
// If the document doesn't exist, then it will return false, and a ErrDocumentNotFound error.
-// This operation will all state relating to the given DocKey. This includes data, block, and head storage.
-func (c *collection) Delete(ctx context.Context, key client.DocKey) (bool, error) {
+// This operation will all state relating to the given DocID. This includes data, block, and head storage.
+func (c *collection) Delete(ctx context.Context, docID client.DocID) (bool, error) {
txn, err := c.getTxn(ctx, false)
if err != nil {
return false, err
}
defer c.discardImplicitTxn(ctx, txn)
- primaryKey := c.getPrimaryKeyFromDocKey(key)
+ primaryKey := c.getPrimaryKeyFromDocID(docID)
exists, isDeleted, err := c.exists(ctx, txn, primaryKey)
if err != nil {
return false, err
@@ -1134,7 +1161,7 @@ func (c *collection) Delete(ctx context.Context, key client.DocKey) (bool, error
return false, client.ErrDocumentNotFound
}
if isDeleted {
- return false, NewErrDocumentDeleted(primaryKey.DocKey)
+ return false, NewErrDocumentDeleted(primaryKey.DocID)
}
err = c.applyDelete(ctx, txn, primaryKey)
@@ -1144,15 +1171,15 @@ func (c *collection) Delete(ctx context.Context, key client.DocKey) (bool, error
return true, c.commitImplicitTxn(ctx, txn)
}
-// Exists checks if a given document exists with supplied DocKey.
-func (c *collection) Exists(ctx context.Context, key client.DocKey) (bool, error) {
+// Exists checks if a given document exists with supplied DocID.
+func (c *collection) Exists(ctx context.Context, docID client.DocID) (bool, error) {
txn, err := c.getTxn(ctx, false)
if err != nil {
return false, err
}
defer c.discardImplicitTxn(ctx, txn)
- primaryKey := c.getPrimaryKeyFromDocKey(key)
+ primaryKey := c.getPrimaryKeyFromDocID(docID)
exists, isDeleted, err := c.exists(ctx, txn, primaryKey)
if err != nil && !errors.Is(err, ds.ErrNotFound) {
return false, err
@@ -1160,13 +1187,13 @@ func (c *collection) Exists(ctx context.Context, key client.DocKey) (bool, error
return exists && !isDeleted, c.commitImplicitTxn(ctx, txn)
}
-// check if a document exists with the given key
+// check if a document exists with the given primary key
func (c *collection) exists(
ctx context.Context,
txn datastore.Txn,
- key core.PrimaryDataStoreKey,
+ primaryKey core.PrimaryDataStoreKey,
) (exists bool, isDeleted bool, err error) {
- val, err := txn.Datastore().Get(ctx, key.ToDS())
+ val, err := txn.Datastore().Get(ctx, primaryKey.ToDS())
if err != nil && errors.Is(err, ds.ErrNotFound) {
return false, false, nil
} else if err != nil {
@@ -1179,116 +1206,26 @@ func (c *collection) exists(
return true, false, nil
}
-func (c *collection) saveDocValue(
+func (c *collection) saveCompositeToMerkleCRDT(
ctx context.Context,
txn datastore.Txn,
- key core.DataStoreKey,
- val client.Value,
+ dsKey core.DataStoreKey,
+ links []core.DAGLink,
+ status client.DocumentStatus,
) (ipld.Node, uint64, error) {
- switch val.Type() {
- case client.LWW_REGISTER:
- wval, ok := val.(client.WriteableValue)
- if !ok {
- return nil, 0, client.ErrValueTypeMismatch
- }
- var bytes []byte
- var err error
- if val.IsDelete() { // empty byte array
- bytes = []byte{}
- } else {
- bytes, err = wval.Bytes()
- if err != nil {
- return nil, 0, err
- }
- }
- return c.saveValueToMerkleCRDT(ctx, txn, key, client.LWW_REGISTER, bytes)
- default:
- return nil, 0, ErrUnknownCRDT
- }
-}
-
-func (c *collection) saveValueToMerkleCRDT(
- ctx context.Context,
- txn datastore.Txn,
- key core.DataStoreKey,
- ctype client.CType,
- args ...any) (ipld.Node, uint64, error) {
- switch ctype {
- case client.LWW_REGISTER:
- fieldID, err := strconv.Atoi(key.FieldId)
- if err != nil {
- return nil, 0, err
- }
-
- schema := c.Schema()
-
- field, ok := c.Description().GetFieldByID(client.FieldID(fieldID), &schema)
- if !ok {
- return nil, 0, client.NewErrFieldIndexNotExist(fieldID)
- }
-
- merkleCRDT, err := c.db.crdtFactory.InstanceWithStores(
- txn,
- core.NewCollectionSchemaVersionKey(schema.VersionID, c.ID()),
- c.db.events.Updates,
- ctype,
- key,
- field.Name,
- )
- if err != nil {
- return nil, 0, err
- }
-
- var bytes []byte
- // parse args
- if len(args) != 1 {
- return nil, 0, ErrUnknownCRDTArgument
- }
- bytes, ok = args[0].([]byte)
- if !ok {
- return nil, 0, ErrUnknownCRDTArgument
- }
- lwwreg := merkleCRDT.(*crdt.MerkleLWWRegister)
- return lwwreg.Set(ctx, bytes)
- case client.COMPOSITE:
- key = key.WithFieldId(core.COMPOSITE_NAMESPACE)
- merkleCRDT, err := c.db.crdtFactory.InstanceWithStores(
- txn,
- core.NewCollectionSchemaVersionKey(c.Schema().VersionID, c.ID()),
- c.db.events.Updates,
- ctype,
- key,
- "",
- )
- if err != nil {
- return nil, 0, err
- }
+ dsKey = dsKey.WithFieldId(core.COMPOSITE_NAMESPACE)
+ merkleCRDT := merklecrdt.NewMerkleCompositeDAG(
+ txn,
+ core.NewCollectionSchemaVersionKey(c.Schema().VersionID, c.ID()),
+ dsKey,
+ "",
+ )
- // parse args
- if len(args) < 2 {
- return nil, 0, ErrUnknownCRDTArgument
- }
- bytes, ok := args[0].([]byte)
- if !ok {
- return nil, 0, ErrUnknownCRDTArgument
- }
- links, ok := args[1].([]core.DAGLink)
- if !ok {
- return nil, 0, ErrUnknownCRDTArgument
- }
- comp := merkleCRDT.(*crdt.MerkleCompositeDAG)
- if len(args) > 2 {
- status, ok := args[2].(client.DocumentStatus)
- if !ok {
- return nil, 0, ErrUnknownCRDTArgument
- }
- if status.IsDeleted() {
- return comp.Delete(ctx, links)
- }
- }
- return comp.Set(ctx, bytes, links)
+ if status.IsDeleted() {
+ return merkleCRDT.Delete(ctx, links)
}
- return nil, 0, ErrUnknownCRDT
+
+ return merkleCRDT.Save(ctx, links)
}
// getTxn gets or creates a new transaction from the underlying db.
@@ -1320,30 +1257,30 @@ func (c *collection) commitImplicitTxn(ctx context.Context, txn datastore.Txn) e
return nil
}
-func (c *collection) getPrimaryKeyFromDocKey(docKey client.DocKey) core.PrimaryDataStoreKey {
+func (c *collection) getPrimaryKeyFromDocID(docID client.DocID) core.PrimaryDataStoreKey {
return core.PrimaryDataStoreKey{
CollectionId: fmt.Sprint(c.ID()),
- DocKey: docKey.String(),
+ DocID: docID.String(),
}
}
-func (c *collection) getDSKeyFromDockey(docKey client.DocKey) core.DataStoreKey {
+func (c *collection) getDataStoreKeyFromDocID(docID client.DocID) core.DataStoreKey {
return core.DataStoreKey{
CollectionID: fmt.Sprint(c.ID()),
- DocKey: docKey.String(),
+ DocID: docID.String(),
InstanceType: core.ValueKey,
}
}
-func (c *collection) tryGetFieldKey(key core.PrimaryDataStoreKey, fieldName string) (core.DataStoreKey, bool) {
+func (c *collection) tryGetFieldKey(primaryKey core.PrimaryDataStoreKey, fieldName string) (core.DataStoreKey, bool) {
fieldId, hasField := c.tryGetSchemaFieldID(fieldName)
if !hasField {
return core.DataStoreKey{}, false
}
return core.DataStoreKey{
- CollectionID: key.CollectionId,
- DocKey: key.DocKey,
+ CollectionID: primaryKey.CollectionId,
+ DocID: primaryKey.DocID,
FieldId: strconv.FormatUint(uint64(fieldId), 10),
}, true
}
diff --git a/db/collection_delete.go b/db/collection_delete.go
index 7f6a968a97..f91b8e38f2 100644
--- a/db/collection_delete.go
+++ b/db/collection_delete.go
@@ -24,11 +24,11 @@ import (
// DeleteWith deletes a target document.
//
-// Target can be a Filter statement, a single docKey, a single document,
-// an array of docKeys, or an array of documents.
+// Target can be a Filter statement, a single DocID, a single document,
+// an array of DocIDs, or an array of documents.
//
// If you want more type safety, use the respective typed versions of Delete.
-// Eg: DeleteWithFilter or DeleteWithKey
+// Eg: DeleteWithFilter or DeleteWithDocID
func (c *collection) DeleteWith(
ctx context.Context,
target any,
@@ -36,19 +36,19 @@ func (c *collection) DeleteWith(
switch t := target.(type) {
case string, map[string]any, *request.Filter:
return c.DeleteWithFilter(ctx, t)
- case client.DocKey:
- return c.DeleteWithKey(ctx, t)
- case []client.DocKey:
- return c.DeleteWithKeys(ctx, t)
+ case client.DocID:
+ return c.DeleteWithDocID(ctx, t)
+ case []client.DocID:
+ return c.DeleteWithDocIDs(ctx, t)
default:
return nil, client.ErrInvalidDeleteTarget
}
}
-// DeleteWithKey deletes using a DocKey to target a single document for delete.
-func (c *collection) DeleteWithKey(
+// DeleteWithDocID deletes using a DocID to target a single document for delete.
+func (c *collection) DeleteWithDocID(
ctx context.Context,
- key client.DocKey,
+ docID client.DocID,
) (*client.DeleteResult, error) {
txn, err := c.getTxn(ctx, false)
if err != nil {
@@ -57,7 +57,7 @@ func (c *collection) DeleteWithKey(
defer c.discardImplicitTxn(ctx, txn)
- dsKey := c.getPrimaryKeyFromDocKey(key)
+ dsKey := c.getPrimaryKeyFromDocID(docID)
res, err := c.deleteWithKey(ctx, txn, dsKey, client.Deleted)
if err != nil {
return nil, err
@@ -66,10 +66,10 @@ func (c *collection) DeleteWithKey(
return res, c.commitImplicitTxn(ctx, txn)
}
-// DeleteWithKeys is the same as DeleteWithKey but accepts multiple keys as a slice.
-func (c *collection) DeleteWithKeys(
+// DeleteWithDocIDs is the same as DeleteWithDocID but accepts multiple DocIDs as a slice.
+func (c *collection) DeleteWithDocIDs(
ctx context.Context,
- keys []client.DocKey,
+ docIDs []client.DocID,
) (*client.DeleteResult, error) {
txn, err := c.getTxn(ctx, false)
if err != nil {
@@ -78,7 +78,7 @@ func (c *collection) DeleteWithKeys(
defer c.discardImplicitTxn(ctx, txn)
- res, err := c.deleteWithKeys(ctx, txn, keys, client.Deleted)
+ res, err := c.deleteWithIDs(ctx, txn, docIDs, client.Deleted)
if err != nil {
return nil, err
}
@@ -112,7 +112,7 @@ func (c *collection) deleteWithKey(
key core.PrimaryDataStoreKey,
status client.DocumentStatus,
) (*client.DeleteResult, error) {
- // Check the docKey we have been given to delete with actually has a corresponding
+ // Check the key we have been given to delete with actually has a corresponding
// document (i.e. document actually exists in the collection).
err := c.applyDelete(ctx, txn, key)
if err != nil {
@@ -121,38 +121,38 @@ func (c *collection) deleteWithKey(
// Upon successfull deletion, record a summary.
results := &client.DeleteResult{
- Count: 1,
- DocKeys: []string{key.DocKey},
+ Count: 1,
+ DocIDs: []string{key.DocID},
}
return results, nil
}
-func (c *collection) deleteWithKeys(
+func (c *collection) deleteWithIDs(
ctx context.Context,
txn datastore.Txn,
- keys []client.DocKey,
+ docIDs []client.DocID,
status client.DocumentStatus,
) (*client.DeleteResult, error) {
results := &client.DeleteResult{
- DocKeys: make([]string, 0),
+ DocIDs: make([]string, 0),
}
- for _, key := range keys {
- dsKey := c.getPrimaryKeyFromDocKey(key)
+ for _, docID := range docIDs {
+ primaryKey := c.getPrimaryKeyFromDocID(docID)
// Apply the function that will perform the full deletion of this document.
- err := c.applyDelete(ctx, txn, dsKey)
+ err := c.applyDelete(ctx, txn, primaryKey)
if err != nil {
return nil, err
}
- // Add this deleted key to our list.
- results.DocKeys = append(results.DocKeys, key.String())
+ // Add this deleted docID to our list.
+ results.DocIDs = append(results.DocIDs, docID.String())
}
// Upon successfull deletion, record a summary of how many we deleted.
- results.Count = int64(len(results.DocKeys))
+ results.Count = int64(len(results.DocIDs))
return results, nil
}
@@ -186,7 +186,7 @@ func (c *collection) deleteWithFilter(
}()
results := &client.DeleteResult{
- DocKeys: make([]string, 0),
+ DocIDs: make([]string, 0),
}
// Keep looping until results from the selection plan have been iterated through.
@@ -202,26 +202,26 @@ func (c *collection) deleteWithFilter(
}
doc := selectionPlan.Value()
- // Extract the dockey in the string format from the document value.
- docKey := doc.GetKey()
- // Convert from string to client.DocKey.
- key := core.PrimaryDataStoreKey{
+ // Extract the docID in the string format from the document value.
+ docID := doc.GetID()
+
+ primaryKey := core.PrimaryDataStoreKey{
CollectionId: fmt.Sprint(c.ID()),
- DocKey: docKey,
+ DocID: docID,
}
- // Delete the document that is associated with this key we got from the filter.
- err = c.applyDelete(ctx, txn, key)
+ // Delete the document that is associated with this DS key we got from the filter.
+ err = c.applyDelete(ctx, txn, primaryKey)
if err != nil {
return nil, err
}
- // Add key of successfully deleted document to our list.
- results.DocKeys = append(results.DocKeys, docKey)
+ // Add docID of successfully deleted document to our list.
+ results.DocIDs = append(results.DocIDs, docID)
}
- results.Count = int64(len(results.DocKeys))
+ results.Count = int64(len(results.DocIDs))
return results, nil
}
@@ -229,9 +229,9 @@ func (c *collection) deleteWithFilter(
func (c *collection) applyDelete(
ctx context.Context,
txn datastore.Txn,
- key core.PrimaryDataStoreKey,
+ primaryKey core.PrimaryDataStoreKey,
) error {
- found, isDeleted, err := c.exists(ctx, txn, key)
+ found, isDeleted, err := c.exists(ctx, txn, primaryKey)
if err != nil {
return err
}
@@ -239,10 +239,10 @@ func (c *collection) applyDelete(
return client.ErrDocumentNotFound
}
if isDeleted {
- return NewErrDocumentDeleted(key.DocKey)
+ return NewErrDocumentDeleted(primaryKey.DocID)
}
- dsKey := key.ToDataStoreKey()
+ dsKey := primaryKey.ToDataStoreKey()
headset := clock.NewHeadSet(
txn.Headstore(),
@@ -261,12 +261,10 @@ func (c *collection) applyDelete(
}
}
- headNode, priority, err := c.saveValueToMerkleCRDT(
+ headNode, priority, err := c.saveCompositeToMerkleCRDT(
ctx,
txn,
dsKey,
- client.COMPOSITE,
- []byte{},
dagLinks,
client.Deleted,
)
@@ -279,7 +277,7 @@ func (c *collection) applyDelete(
func() {
c.db.events.Updates.Value().Publish(
events.Update{
- DocKey: key.DocKey,
+ DocID: primaryKey.DocID,
Cid: headNode.Cid(),
SchemaRoot: c.Schema().Root,
Block: headNode,
diff --git a/db/collection_get.go b/db/collection_get.go
index d210072793..e19ccd58c0 100644
--- a/db/collection_get.go
+++ b/db/collection_get.go
@@ -20,16 +20,16 @@ import (
"github.com/sourcenetwork/defradb/db/fetcher"
)
-func (c *collection) Get(ctx context.Context, key client.DocKey, showDeleted bool) (*client.Document, error) {
+func (c *collection) Get(ctx context.Context, docID client.DocID, showDeleted bool) (*client.Document, error) {
// create txn
txn, err := c.getTxn(ctx, true)
if err != nil {
return nil, err
}
defer c.discardImplicitTxn(ctx, txn)
- dsKey := c.getPrimaryKeyFromDocKey(key)
+ primaryKey := c.getPrimaryKeyFromDocID(docID)
- found, isDeleted, err := c.exists(ctx, txn, dsKey)
+ found, isDeleted, err := c.exists(ctx, txn, primaryKey)
if err != nil {
return nil, err
}
@@ -37,7 +37,7 @@ func (c *collection) Get(ctx context.Context, key client.DocKey, showDeleted boo
return nil, client.ErrDocumentNotFound
}
- doc, err := c.get(ctx, txn, dsKey, nil, showDeleted)
+ doc, err := c.get(ctx, txn, primaryKey, nil, showDeleted)
if err != nil {
return nil, err
}
@@ -47,7 +47,7 @@ func (c *collection) Get(ctx context.Context, key client.DocKey, showDeleted boo
func (c *collection) get(
ctx context.Context,
txn datastore.Txn,
- key core.PrimaryDataStoreKey,
+ primaryKey core.PrimaryDataStoreKey,
fields []client.FieldDescription,
showDeleted bool,
) (*client.Document, error) {
@@ -60,8 +60,8 @@ func (c *collection) get(
return nil, err
}
- // construct target key for DocKey
- targetKey := base.MakeDocKey(c.Description(), key.DocKey)
+ // construct target DS key from DocID.
+ targetKey := base.MakeDataStoreKeyWithCollectionAndDocID(c.Description(), primaryKey.DocID)
// run the doc fetcher
err = df.Start(ctx, core.NewSpans(core.NewSpan(targetKey, targetKey.PrefixEnd())))
if err != nil {
@@ -85,7 +85,7 @@ func (c *collection) get(
return nil, nil
}
- doc, err := fetcher.Decode(encodedDoc)
+ doc, err := fetcher.Decode(encodedDoc, c.Schema())
if err != nil {
return nil, err
}
diff --git a/db/collection_index.go b/db/collection_index.go
index 278586902b..4367d8ebdf 100644
--- a/db/collection_index.go
+++ b/db/collection_index.go
@@ -13,6 +13,7 @@ package db
import (
"context"
"encoding/json"
+ "errors"
"fmt"
"strconv"
"strings"
@@ -125,7 +126,7 @@ func (c *collection) updateIndexedDoc(
oldDoc, err := c.get(
ctx,
txn,
- c.getPrimaryKeyFromDocKey(doc.Key()), desc.CollectIndexedFields(&schema),
+ c.getPrimaryKeyFromDocID(doc.ID()), desc.CollectIndexedFields(&schema),
false,
)
if err != nil {
@@ -236,36 +237,33 @@ func (c *collection) iterateAllDocs(
df := c.newFetcher()
err := df.Init(ctx, txn, c, fields, nil, nil, false, false)
if err != nil {
- _ = df.Close()
- return err
+ return errors.Join(err, df.Close())
}
- start := base.MakeCollectionKey(c.Description())
+ start := base.MakeDataStoreKeyWithCollectionDescription(c.Description())
spans := core.NewSpans(core.NewSpan(start, start.PrefixEnd()))
err = df.Start(ctx, spans)
if err != nil {
- _ = df.Close()
- return err
+ return errors.Join(err, df.Close())
}
for {
encodedDoc, _, err := df.FetchNext(ctx)
if err != nil {
- _ = df.Close()
- return err
+ return errors.Join(err, df.Close())
}
if encodedDoc == nil {
break
}
- doc, err := fetcher.Decode(encodedDoc)
+ doc, err := fetcher.Decode(encodedDoc, c.Schema())
if err != nil {
- return err
+ return errors.Join(err, df.Close())
}
err = exec(doc)
if err != nil {
- return err
+ return errors.Join(err, df.Close())
}
}
diff --git a/db/collection_update.go b/db/collection_update.go
index c68902db44..4c1895602b 100644
--- a/db/collection_update.go
+++ b/db/collection_update.go
@@ -26,10 +26,10 @@ import (
)
// UpdateWith updates a target document using the given updater type. Target
-// can be a Filter statement, a single docKey, a single document,
-// an array of docKeys, or an array of documents.
+// can be a Filter statement, a single DocID, a single document,
+// an array of DocIDs, or an array of documents.
// If you want more type safety, use the respective typed versions of Update.
-// Eg: UpdateWithFilter or UpdateWithKey
+// Eg: UpdateWithFilter or UpdateWithDocID
func (c *collection) UpdateWith(
ctx context.Context,
target any,
@@ -38,10 +38,10 @@ func (c *collection) UpdateWith(
switch t := target.(type) {
case string, map[string]any, *request.Filter:
return c.UpdateWithFilter(ctx, t, updater)
- case client.DocKey:
- return c.UpdateWithKey(ctx, t, updater)
- case []client.DocKey:
- return c.UpdateWithKeys(ctx, t, updater)
+ case client.DocID:
+ return c.UpdateWithDocID(ctx, t, updater)
+ case []client.DocID:
+ return c.UpdateWithDocIDs(ctx, t, updater)
default:
return nil, client.ErrInvalidUpdateTarget
}
@@ -67,12 +67,12 @@ func (c *collection) UpdateWithFilter(
return res, c.commitImplicitTxn(ctx, txn)
}
-// UpdateWithKey updates using a DocKey to target a single document for update.
+// UpdateWithDocID updates using a DocID to target a single document for update.
// An updater value is provided, which could be a string Patch, string Merge Patch
// or a parsed Patch, or parsed Merge Patch.
-func (c *collection) UpdateWithKey(
+func (c *collection) UpdateWithDocID(
ctx context.Context,
- key client.DocKey,
+ docID client.DocID,
updater string,
) (*client.UpdateResult, error) {
txn, err := c.getTxn(ctx, false)
@@ -80,7 +80,7 @@ func (c *collection) UpdateWithKey(
return nil, err
}
defer c.discardImplicitTxn(ctx, txn)
- res, err := c.updateWithKey(ctx, txn, key, updater)
+ res, err := c.updateWithDocID(ctx, txn, docID, updater)
if err != nil {
return nil, err
}
@@ -88,12 +88,12 @@ func (c *collection) UpdateWithKey(
return res, c.commitImplicitTxn(ctx, txn)
}
-// UpdateWithKeys is the same as UpdateWithKey but accepts multiple keys as a slice.
+// UpdateWithDocIDs is the same as UpdateWithDocID but accepts multiple DocIDs as a slice.
// An updater value is provided, which could be a string Patch, string Merge Patch
// or a parsed Patch, or parsed Merge Patch.
-func (c *collection) UpdateWithKeys(
+func (c *collection) UpdateWithDocIDs(
ctx context.Context,
- keys []client.DocKey,
+ docIDs []client.DocID,
updater string,
) (*client.UpdateResult, error) {
txn, err := c.getTxn(ctx, false)
@@ -101,7 +101,7 @@ func (c *collection) UpdateWithKeys(
return nil, err
}
defer c.discardImplicitTxn(ctx, txn)
- res, err := c.updateWithKeys(ctx, txn, keys, updater)
+ res, err := c.updateWithIDs(ctx, txn, docIDs, updater)
if err != nil {
return nil, err
}
@@ -109,10 +109,10 @@ func (c *collection) UpdateWithKeys(
return res, c.commitImplicitTxn(ctx, txn)
}
-func (c *collection) updateWithKey(
+func (c *collection) updateWithDocID(
ctx context.Context,
txn datastore.Txn,
- key client.DocKey,
+ docID client.DocID,
updater string,
) (*client.UpdateResult, error) {
parsedUpdater, err := fastjson.Parse(updater)
@@ -127,7 +127,7 @@ func (c *collection) updateWithKey(
return nil, client.ErrInvalidUpdater
}
- doc, err := c.Get(ctx, key, false)
+ doc, err := c.Get(ctx, docID, false)
if err != nil {
return nil, err
}
@@ -135,7 +135,7 @@ func (c *collection) updateWithKey(
if isPatch {
// todo
} else {
- err = c.applyMergeToDoc(doc, parsedUpdater.GetObject())
+ err = doc.SetWithJSON([]byte(updater))
}
if err != nil {
return nil, err
@@ -147,16 +147,16 @@ func (c *collection) updateWithKey(
}
results := &client.UpdateResult{
- Count: 1,
- DocKeys: []string{key.String()},
+ Count: 1,
+ DocIDs: []string{docID.String()},
}
return results, nil
}
-func (c *collection) updateWithKeys(
+func (c *collection) updateWithIDs(
ctx context.Context,
txn datastore.Txn,
- keys []client.DocKey,
+ docIDs []client.DocID,
updater string,
) (*client.UpdateResult, error) {
parsedUpdater, err := fastjson.Parse(updater)
@@ -172,10 +172,10 @@ func (c *collection) updateWithKeys(
}
results := &client.UpdateResult{
- DocKeys: make([]string, len(keys)),
+ DocIDs: make([]string, len(docIDs)),
}
- for i, key := range keys {
- doc, err := c.Get(ctx, key, false)
+ for i, docIDs := range docIDs {
+ doc, err := c.Get(ctx, docIDs, false)
if err != nil {
return nil, err
}
@@ -183,7 +183,7 @@ func (c *collection) updateWithKeys(
if isPatch {
// todo
} else {
- err = c.applyMergeToDoc(doc, parsedUpdater.GetObject())
+ err = doc.SetWithJSON([]byte(updater))
}
if err != nil {
return nil, err
@@ -194,7 +194,7 @@ func (c *collection) updateWithKeys(
return nil, err
}
- results.DocKeys[i] = key.String()
+ results.DocIDs[i] = docIDs.String()
results.Count++
}
return results, nil
@@ -245,7 +245,7 @@ func (c *collection) updateWithFilter(
}()
results := &client.UpdateResult{
- DocKeys: make([]string, 0),
+ DocIDs: make([]string, 0),
}
docMap := selectionPlan.DocumentMap()
@@ -263,7 +263,7 @@ func (c *collection) updateWithFilter(
// Get the document, and apply the patch
docAsMap := docMap.ToMap(selectionPlan.Value())
- doc, err := client.NewDocFromMap(docAsMap)
+ doc, err := client.NewDocFromMap(docAsMap, c.Schema())
if err != nil {
return nil, err
}
@@ -271,10 +271,10 @@ func (c *collection) updateWithFilter(
if isPatch {
// todo
} else if isMerge { // else is fine here
- err = c.applyMergeToDoc(doc, parsedUpdater.GetObject())
- }
- if err != nil {
- return nil, err
+ err := doc.SetWithJSON([]byte(updater))
+ if err != nil {
+ return nil, err
+ }
}
_, err = c.save(ctx, txn, doc, false)
@@ -283,52 +283,13 @@ func (c *collection) updateWithFilter(
}
// add successful updated doc to results
- results.DocKeys = append(results.DocKeys, doc.Key().String())
+ results.DocIDs = append(results.DocIDs, doc.ID().String())
results.Count++
}
return results, nil
}
-// applyMergeToDoc applies the given json merge to the given Defra doc.
-//
-// It does not save the document.
-func (c *collection) applyMergeToDoc(
- doc *client.Document,
- merge *fastjson.Object,
-) error {
- mergeMap := make(map[string]*fastjson.Value)
- merge.Visit(func(k []byte, v *fastjson.Value) {
- mergeMap[string(k)] = v
- })
-
- for mfield, mval := range mergeMap {
- fd, isValidField := c.Schema().GetField(mfield)
- if !isValidField {
- return client.NewErrFieldNotExist(mfield)
- }
-
- if fd.Kind == client.FieldKind_FOREIGN_OBJECT {
- fd, isValidField = c.Schema().GetField(mfield + request.RelatedObjectID)
- if !isValidField {
- return client.NewErrFieldNotExist(mfield)
- }
- }
-
- cborVal, err := validateFieldSchema(mval, fd)
- if err != nil {
- return err
- }
-
- err = doc.Set(fd.Name, cborVal)
- if err != nil {
- return err
- }
- }
-
- return nil
-}
-
// isSecondaryIDField returns true if the given field description represents a secondary relation field ID.
func (c *collection) isSecondaryIDField(fieldDesc client.FieldDescription) (client.FieldDescription, bool) {
if fieldDesc.RelationType != client.Relation_Type_INTERNAL_ID {
@@ -341,21 +302,21 @@ func (c *collection) isSecondaryIDField(fieldDesc client.FieldDescription) (clie
return relationFieldDescription, valid && !relationFieldDescription.IsPrimaryRelation()
}
-// patchPrimaryDoc patches the (primary) document linked to from the document of the given dockey via the
+// patchPrimaryDoc patches the (primary) document linked to from the document of the given DocID via the
// given (secondary) relationship field description (hosted on the collection of the document matching the
-// given dockey).
+// given DocID).
//
-// The given field value should be the string representation of the dockey of the primary document to be
+// The given field value should be the string representation of the DocID of the primary document to be
// patched.
func (c *collection) patchPrimaryDoc(
ctx context.Context,
txn datastore.Txn,
secondaryCollectionName string,
relationFieldDescription client.FieldDescription,
- docKey string,
+ docID string,
fieldValue string,
) error {
- primaryDockey, err := client.NewDocKeyFromString(fieldValue)
+ primaryDocID, err := client.NewDocIDFromString(fieldValue)
if err != nil {
return err
}
@@ -384,7 +345,7 @@ func (c *collection) patchPrimaryDoc(
doc, err := primaryCol.Get(
ctx,
- primaryDockey,
+ primaryDocID,
false,
)
if err != nil && !errors.Is(err, ds.ErrNotFound) {
@@ -396,16 +357,22 @@ func (c *collection) patchPrimaryDoc(
return nil
}
+ pc := c.db.newCollection(primaryCol.Description(), primarySchema)
+ err = pc.validateOneToOneLinkDoesntAlreadyExist(ctx, txn, primaryDocID.String(), primaryIDField, docID)
+ if err != nil {
+ return err
+ }
+
existingVal, err := doc.GetValue(primaryIDField.Name)
if err != nil && !errors.Is(err, client.ErrFieldNotExist) {
return err
}
- if existingVal != nil && existingVal.Value() != "" && existingVal.Value() != docKey {
- return NewErrOneOneAlreadyLinked(docKey, fieldValue, relationFieldDescription.RelationName)
+ if existingVal != nil && existingVal.Value() != "" && existingVal.Value() != docID {
+ return NewErrOneOneAlreadyLinked(docID, fieldValue, relationFieldDescription.RelationName)
}
- err = doc.Set(primaryIDField.Name, docKey)
+ err = doc.Set(primaryIDField.Name, docID)
if err != nil {
return err
}
@@ -418,135 +385,6 @@ func (c *collection) patchPrimaryDoc(
return nil
}
-// validateFieldSchema takes a given value as an interface,
-// and ensures it matches the supplied field description.
-// It will do any minor parsing, like dates, and return
-// the typed value again as an interface.
-func validateFieldSchema(val *fastjson.Value, field client.FieldDescription) (any, error) {
- switch field.Kind {
- case client.FieldKind_DocKey, client.FieldKind_STRING:
- return getString(val)
-
- case client.FieldKind_STRING_ARRAY:
- return getArray(val, getString)
-
- case client.FieldKind_NILLABLE_STRING_ARRAY:
- return getNillableArray(val, getString)
-
- case client.FieldKind_BOOL:
- return getBool(val)
-
- case client.FieldKind_BOOL_ARRAY:
- return getArray(val, getBool)
-
- case client.FieldKind_NILLABLE_BOOL_ARRAY:
- return getNillableArray(val, getBool)
-
- case client.FieldKind_FLOAT:
- return getFloat64(val)
-
- case client.FieldKind_FLOAT_ARRAY:
- return getArray(val, getFloat64)
-
- case client.FieldKind_NILLABLE_FLOAT_ARRAY:
- return getNillableArray(val, getFloat64)
-
- case client.FieldKind_DATETIME:
- // @TODO: Requires Typed Document refactor
- // to handle this correctly.
- // For now, we will persist DateTime as a
- // RFC3339 string
- // see https://github.com/sourcenetwork/defradb/issues/935
- return getString(val)
-
- case client.FieldKind_INT:
- return getInt64(val)
-
- case client.FieldKind_INT_ARRAY:
- return getArray(val, getInt64)
-
- case client.FieldKind_NILLABLE_INT_ARRAY:
- return getNillableArray(val, getInt64)
-
- case client.FieldKind_FOREIGN_OBJECT, client.FieldKind_FOREIGN_OBJECT_ARRAY:
- return nil, NewErrFieldOrAliasToFieldNotExist(field.Name)
- }
-
- return nil, client.NewErrUnhandledType("FieldKind", field.Kind)
-}
-
-func getString(v *fastjson.Value) (string, error) {
- b, err := v.StringBytes()
- return string(b), err
-}
-
-func getBool(v *fastjson.Value) (bool, error) {
- return v.Bool()
-}
-
-func getFloat64(v *fastjson.Value) (float64, error) {
- return v.Float64()
-}
-
-func getInt64(v *fastjson.Value) (int64, error) {
- return v.Int64()
-}
-
-func getArray[T any](
- val *fastjson.Value,
- typeGetter func(*fastjson.Value) (T, error),
-) ([]T, error) {
- if val.Type() == fastjson.TypeNull {
- return nil, nil
- }
-
- valArray, err := val.Array()
- if err != nil {
- return nil, err
- }
-
- arr := make([]T, len(valArray))
- for i, arrItem := range valArray {
- if arrItem.Type() == fastjson.TypeNull {
- continue
- }
- arr[i], err = typeGetter(arrItem)
- if err != nil {
- return nil, err
- }
- }
-
- return arr, nil
-}
-
-func getNillableArray[T any](
- val *fastjson.Value,
- typeGetter func(*fastjson.Value) (T, error),
-) ([]*T, error) {
- if val.Type() == fastjson.TypeNull {
- return nil, nil
- }
-
- valArray, err := val.Array()
- if err != nil {
- return nil, err
- }
-
- arr := make([]*T, len(valArray))
- for i, arrItem := range valArray {
- if arrItem.Type() == fastjson.TypeNull {
- continue
- }
- v, err := typeGetter(arrItem)
- if err != nil {
- return nil, err
- }
- arr[i] = &v
- }
-
- return arr, nil
-}
-
// makeSelectionPlan constructs a simple read-only plan of the collection using the given filter.
// currently it doesn't support any other operations other than filters.
// (IE: No limit, order, etc)
diff --git a/db/db.go b/db/db.go
index f2f59ecdaf..1046b2db54 100644
--- a/db/db.go
+++ b/db/db.go
@@ -31,7 +31,6 @@ import (
"github.com/sourcenetwork/defradb/events"
"github.com/sourcenetwork/defradb/lens"
"github.com/sourcenetwork/defradb/logging"
- "github.com/sourcenetwork/defradb/merkle/crdt"
"github.com/sourcenetwork/defradb/request/graphql"
)
@@ -56,8 +55,6 @@ type db struct {
rootstore datastore.RootStore
multistore datastore.MultiStore
- crdtFactory *crdt.Factory
-
events events.Events
parser core.Parser
@@ -114,7 +111,6 @@ func NewDB(ctx context.Context, rootstore datastore.RootStore, options ...Option
func newDB(ctx context.Context, rootstore datastore.RootStore, options ...Option) (*implicitTxnDB, error) {
log.Debug(ctx, "Loading: internal datastores")
multistore := datastore.MultiStoreFrom(rootstore)
- crdtFactory := crdt.DefaultFactory.WithStores(multistore)
parser, err := graphql.NewParser()
if err != nil {
@@ -125,8 +121,6 @@ func newDB(ctx context.Context, rootstore datastore.RootStore, options ...Option
rootstore: rootstore,
multistore: multistore,
- crdtFactory: &crdtFactory,
-
parser: parser,
options: options,
}
diff --git a/db/description/schema.go b/db/description/schema.go
index 06b129f3df..c486ee1a59 100644
--- a/db/description/schema.go
+++ b/db/description/schema.go
@@ -283,3 +283,45 @@ func GetSchemaVersionIDs(
return schemaVersions, nil
}
+
+// GetCollectionlessSchemas returns all schema that are not attached to a collection.
+//
+// Typically this means any schema embedded in a View.
+//
+// WARNING: This function does not currently account for multiple versions of collectionless schema,
+// at the moment such a situation is impossible, but that is likely to change, at which point this
+// function will need to account for that.
+func GetCollectionlessSchemas(
+ ctx context.Context,
+ txn datastore.Txn,
+) ([]client.SchemaDescription, error) {
+ cols, err := GetCollections(ctx, txn)
+ if err != nil {
+ return nil, err
+ }
+
+ allSchemas, err := GetAllSchemas(ctx, txn)
+ if err != nil {
+ return nil, err
+ }
+
+ schemaRootsByVersionID := map[string]string{}
+ for _, schema := range allSchemas {
+ schemaRootsByVersionID[schema.VersionID] = schema.Root
+ }
+
+ colSchemaRoots := map[string]struct{}{}
+ for _, col := range cols {
+ schemaRoot := schemaRootsByVersionID[col.SchemaVersionID]
+ colSchemaRoots[schemaRoot] = struct{}{}
+ }
+
+ collectionlessSchema := []client.SchemaDescription{}
+ for _, schema := range allSchemas {
+ if _, hasCollection := colSchemaRoots[schema.Root]; !hasCollection {
+ collectionlessSchema = append(collectionlessSchema, schema)
+ }
+ }
+
+ return collectionlessSchema, nil
+}
diff --git a/db/errors.go b/db/errors.go
index 651bcbe42b..d8c9773926 100644
--- a/db/errors.go
+++ b/db/errors.go
@@ -42,13 +42,12 @@ const (
errDuplicateField string = "duplicate field"
errCannotMutateField string = "mutating an existing field is not supported"
errCannotMoveField string = "moving fields is not currently supported"
- errInvalidCRDTType string = "only default or LWW (last writer wins) CRDT types are supported"
errCannotDeleteField string = "deleting an existing field is not supported"
errFieldKindNotFound string = "no type found for given name"
errFieldKindDoesNotMatchFieldSchema string = "field Kind does not match field Schema"
errSchemaNotFound string = "no schema found for given name"
- errDocumentAlreadyExists string = "a document with the given dockey already exists"
- errDocumentDeleted string = "a document with the given dockey has been deleted"
+ errDocumentAlreadyExists string = "a document with the given ID already exists"
+ errDocumentDeleted string = "a document with the given ID has been deleted"
errIndexMissingFields string = "index missing fields"
errNonZeroIndexIDProvided string = "non-zero index ID provided"
errIndexFieldMissingName string = "index field missing name"
@@ -73,9 +72,10 @@ const (
errIndexDescHasNonExistingField string = "index description has non existing field"
errFieldOrAliasToFieldNotExist string = "The given field or alias to field does not exist"
errCreateFile string = "failed to create file"
+ errRemoveFile string = "failed to remove file"
errOpenFile string = "failed to open file"
errCloseFile string = "failed to close file"
- errRemoveFile string = "failed to remove file"
+ errFailedtoCloseQueryReqAllIDs string = "failed to close query requesting all docIDs"
errFailedToReadByte string = "failed to read byte"
errFailedToWriteString string = "failed to write string"
errJSONDecode string = "failed to decode JSON"
@@ -86,92 +86,30 @@ const (
errExpectedJSONArray string = "expected JSON array"
errOneOneAlreadyLinked string = "target document is already linked to another document"
errIndexDoesNotMatchName string = "the index used does not match the given name"
+ errCanNotIndexNonUniqueField string = "can not index a doc's field that violates unique index"
+ errInvalidViewQuery string = "the query provided is not valid as a View"
)
var (
- ErrFailedToGetHeads = errors.New(errFailedToGetHeads)
- ErrFailedToCreateCollectionQuery = errors.New(errFailedToCreateCollectionQuery)
- ErrFailedToGetCollection = errors.New(errFailedToGetCollection)
- ErrFailedToGetAllCollections = errors.New(errFailedToGetAllCollections)
- // ErrDocVerification occurs when a documents contents fail the verification during a Create()
- // call against the supplied Document Key.
- ErrDocVerification = errors.New(errDocVerification)
- ErrSubscriptionsNotAllowed = errors.New("server does not accept subscriptions")
- ErrDeleteTargetEmpty = errors.New("the doc delete targeter cannot be empty")
- ErrDeleteEmpty = errors.New("the doc delete cannot be empty")
- ErrUpdateTargetEmpty = errors.New("the doc update targeter cannot be empty")
- ErrUpdateEmpty = errors.New("the doc update cannot be empty")
- ErrInvalidMergeValueType = errors.New(
- "the type of value in the merge patch doesn't match the schema",
- )
- ErrMissingDocFieldToUpdate = errors.New("missing document field to update")
- ErrDocMissingKey = errors.New("document is missing key")
- ErrInvalidFilter = errors.New("invalid filter")
- ErrInvalidOpPath = errors.New("invalid patch op path")
- ErrDocumentAlreadyExists = errors.New(errDocumentAlreadyExists)
- ErrDocumentDeleted = errors.New(errDocumentDeleted)
- ErrUnknownCRDTArgument = errors.New("invalid CRDT arguments")
- ErrUnknownCRDT = errors.New("unknown crdt")
- ErrCollectionAlreadyExists = errors.New("collection already exists")
- ErrCollectionNameEmpty = errors.New("collection name can't be empty")
- ErrSchemaNameEmpty = errors.New("schema name can't be empty")
- ErrSchemaRootEmpty = errors.New("schema root can't be empty")
- ErrSchemaVersionIDEmpty = errors.New("schema version ID can't be empty")
- ErrKeyEmpty = errors.New("key cannot be empty")
- ErrAddingP2PCollection = errors.New(errAddingP2PCollection)
- ErrRemovingP2PCollection = errors.New(errRemovingP2PCollection)
- ErrAddCollectionWithPatch = errors.New(errAddCollectionWithPatch)
- ErrCollectionIDDoesntMatch = errors.New(errCollectionIDDoesntMatch)
- ErrSchemaRootDoesntMatch = errors.New(errSchemaRootDoesntMatch)
- ErrCannotModifySchemaName = errors.New(errCannotModifySchemaName)
- ErrCannotSetVersionID = errors.New(errCannotSetVersionID)
- ErrCannotSetFieldID = errors.New(errCannotSetFieldID)
- ErrRelationalFieldMissingSchema = errors.New(errRelationalFieldMissingSchema)
- ErrRelationalFieldInvalidRelationType = errors.New(errRelationalFieldInvalidRelationType)
- ErrRelationalFieldMissingIDField = errors.New(errRelationalFieldMissingIDField)
- ErrRelationalFieldMissingRelationName = errors.New(errRelationalFieldMissingRelationName)
- ErrPrimarySideNotDefined = errors.New(errPrimarySideNotDefined)
- ErrPrimarySideOnMany = errors.New(errPrimarySideOnMany)
- ErrBothSidesPrimary = errors.New(errBothSidesPrimary)
- ErrRelatedFieldKindMismatch = errors.New(errRelatedFieldKindMismatch)
- ErrRelatedFieldRelationTypeMismatch = errors.New(errRelatedFieldRelationTypeMismatch)
- ErrRelationalFieldIDInvalidType = errors.New(errRelationalFieldIDInvalidType)
- ErrDuplicateField = errors.New(errDuplicateField)
- ErrCannotMutateField = errors.New(errCannotMutateField)
- ErrCannotMoveField = errors.New(errCannotMoveField)
- ErrInvalidCRDTType = errors.New(errInvalidCRDTType)
- ErrCannotDeleteField = errors.New(errCannotDeleteField)
- ErrFieldKindNotFound = errors.New(errFieldKindNotFound)
- ErrFieldKindDoesNotMatchFieldSchema = errors.New(errFieldKindDoesNotMatchFieldSchema)
- ErrSchemaNotFound = errors.New(errSchemaNotFound)
- ErrIndexMissingFields = errors.New(errIndexMissingFields)
- ErrIndexFieldMissingName = errors.New(errIndexFieldMissingName)
- ErrIndexFieldMissingDirection = errors.New(errIndexFieldMissingDirection)
- ErrIndexSingleFieldWrongDirection = errors.New(errIndexSingleFieldWrongDirection)
- ErrCorruptedIndex = errors.New(errCorruptedIndex)
- ErrCanNotChangeIndexWithPatch = errors.New(errCanNotChangeIndexWithPatch)
- ErrFieldOrAliasToFieldNotExist = errors.New(errFieldOrAliasToFieldNotExist)
- ErrCreateFile = errors.New(errCreateFile)
- ErrOpenFile = errors.New(errOpenFile)
- ErrCloseFile = errors.New(errCloseFile)
- ErrRemoveFile = errors.New(errRemoveFile)
- ErrFailedToReadByte = errors.New(errFailedToReadByte)
- ErrFailedToWriteString = errors.New(errFailedToWriteString)
- ErrJSONDecode = errors.New(errJSONDecode)
- ErrDocFromMap = errors.New(errDocFromMap)
- ErrDocCreate = errors.New(errDocCreate)
- ErrDocUpdate = errors.New(errDocUpdate)
- ErrExpectedJSONObject = errors.New(errExpectedJSONObject)
- ErrExpectedJSONArray = errors.New(errExpectedJSONArray)
- ErrOneOneAlreadyLinked = errors.New(errOneOneAlreadyLinked)
- ErrIndexDoesNotMatchName = errors.New(errIndexDoesNotMatchName)
+ ErrFailedToGetCollection = errors.New(errFailedToGetCollection)
+ ErrSubscriptionsNotAllowed = errors.New("server does not accept subscriptions")
+ ErrInvalidFilter = errors.New("invalid filter")
+ ErrCollectionAlreadyExists = errors.New("collection already exists")
+ ErrCollectionNameEmpty = errors.New("collection name can't be empty")
+ ErrSchemaNameEmpty = errors.New("schema name can't be empty")
+ ErrSchemaRootEmpty = errors.New("schema root can't be empty")
+ ErrSchemaVersionIDEmpty = errors.New("schema version ID can't be empty")
+ ErrKeyEmpty = errors.New("key cannot be empty")
+ ErrCannotSetVersionID = errors.New(errCannotSetVersionID)
+ ErrIndexMissingFields = errors.New(errIndexMissingFields)
+ ErrIndexFieldMissingName = errors.New(errIndexFieldMissingName)
+ ErrIndexSingleFieldWrongDirection = errors.New(errIndexSingleFieldWrongDirection)
+ ErrCorruptedIndex = errors.New(errCorruptedIndex)
+ ErrExpectedJSONObject = errors.New(errExpectedJSONObject)
+ ErrExpectedJSONArray = errors.New(errExpectedJSONArray)
+ ErrInvalidViewQuery = errors.New(errInvalidViewQuery)
)
-// NewErrFieldOrAliasToFieldNotExist returns an error indicating that the given field or an alias field does not exist.
-func NewErrFieldOrAliasToFieldNotExist(name string) error {
- return errors.New(errFieldOrAliasToFieldNotExist, errors.NewKV("Name", name))
-}
-
// NewErrFailedToGetHeads returns a new error indicating that the heads of a document
// could not be obtained.
func NewErrFailedToGetHeads(inner error) error {
@@ -243,6 +181,9 @@ func NewErrFailedToGetAllCollections(inner error) error {
}
// NewErrDocVerification returns a new error indicating that the document verification failed.
+//
+// This occurs when a documents contents fail the verification during a Create()
+// call against the supplied Document ID (docID).
func NewErrDocVerification(expected string, actual string) error {
return errors.New(
errDocVerification,
@@ -432,14 +373,6 @@ func NewErrCannotMoveField(name string, proposedIndex, existingIndex int) error
)
}
-func NewErrInvalidCRDTType(name string, crdtType client.CType) error {
- return errors.New(
- errInvalidCRDTType,
- errors.NewKV("Name", name),
- errors.NewKV("CRDTType", crdtType),
- )
-}
-
func NewErrCannotDeleteField(name string, id client.FieldID) error {
return errors.New(
errCannotDeleteField,
@@ -448,17 +381,17 @@ func NewErrCannotDeleteField(name string, id client.FieldID) error {
)
}
-func NewErrDocumentAlreadyExists(dockey string) error {
+func NewErrDocumentAlreadyExists(docID string) error {
return errors.New(
errDocumentAlreadyExists,
- errors.NewKV("DocKey", dockey),
+ errors.NewKV("DocID", docID),
)
}
-func NewErrDocumentDeleted(dockey string) error {
+func NewErrDocumentDeleted(docID string) error {
return errors.New(
errDocumentDeleted,
- errors.NewKV("DocKey", dockey),
+ errors.NewKV("DocID", docID),
)
}
@@ -632,3 +565,27 @@ func NewErrIndexDoesNotMatchName(index, name string) error {
errors.NewKV("Name", name),
)
}
+
+func NewErrCanNotIndexNonUniqueField(docID, fieldName string, value any) error {
+ return errors.New(
+ errCanNotIndexNonUniqueField,
+ errors.NewKV("DocID", docID),
+ errors.NewKV("Field name", fieldName),
+ errors.NewKV("Field value", value),
+ )
+}
+
+func NewErrInvalidViewQueryCastFailed(query string) error {
+ return errors.New(
+ errInvalidViewQuery,
+ errors.NewKV("Query", query),
+ errors.NewKV("Reason", "Internal error, cast failed"),
+ )
+}
+
+func NewErrInvalidViewQueryMissingQuery() error {
+ return errors.New(
+ errInvalidViewQuery,
+ errors.NewKV("Reason", "No query provided"),
+ )
+}
diff --git a/db/fetcher/encoded_doc.go b/db/fetcher/encoded_doc.go
index bc22471465..e88ee80f9d 100644
--- a/db/fetcher/encoded_doc.go
+++ b/db/fetcher/encoded_doc.go
@@ -19,16 +19,20 @@ import (
)
type EncodedDocument interface {
- // Key returns the key of the document
- Key() []byte
+ // ID returns the ID of the document
+ ID() []byte
+
SchemaVersionID() string
+
// Status returns the document status.
//
// For example, whether it is deleted or active.
Status() client.DocumentStatus
+
// Properties returns a copy of the decoded property values mapped by their field
// description.
Properties(onlyFilterProps bool) (map[client.FieldDescription]any, error)
+
// Reset re-initializes the EncodedDocument object.
Reset()
}
@@ -61,7 +65,7 @@ func (e encProperty) Decode() (any, error) {
// @todo: Implement Encoded Document type
type encodedDocument struct {
- key []byte
+ id []byte
schemaVersionID string
status client.DocumentStatus
properties map[client.FieldDescription]*encProperty
@@ -78,8 +82,8 @@ type encodedDocument struct {
var _ EncodedDocument = (*encodedDocument)(nil)
-func (encdoc *encodedDocument) Key() []byte {
- return encdoc.key
+func (encdoc *encodedDocument) ID() []byte {
+ return encdoc.id
}
func (encdoc *encodedDocument) SchemaVersionID() string {
@@ -93,7 +97,7 @@ func (encdoc *encodedDocument) Status() client.DocumentStatus {
// Reset re-initializes the EncodedDocument object.
func (encdoc *encodedDocument) Reset() {
encdoc.properties = make(map[client.FieldDescription]*encProperty, 0)
- encdoc.key = nil
+ encdoc.id = nil
encdoc.filterSet = nil
encdoc.selectSet = nil
encdoc.schemaVersionID = ""
@@ -102,27 +106,25 @@ func (encdoc *encodedDocument) Reset() {
}
// Decode returns a properly decoded document object
-func Decode(encdoc EncodedDocument) (*client.Document, error) {
- key, err := client.NewDocKeyFromString(string(encdoc.Key()))
+func Decode(encdoc EncodedDocument, sd client.SchemaDescription) (*client.Document, error) {
+ docID, err := client.NewDocIDFromString(string(encdoc.ID()))
if err != nil {
return nil, err
}
- doc := client.NewDocWithKey(key)
+ doc := client.NewDocWithID(docID, sd)
properties, err := encdoc.Properties(false)
if err != nil {
return nil, err
}
for desc, val := range properties {
- err = doc.SetAs(desc.Name, val, desc.Typ)
+ err = doc.Set(desc.Name, val)
if err != nil {
return nil, err
}
}
- doc.SchemaVersionID = encdoc.SchemaVersionID()
-
// client.Document tracks which fields have been set ('dirtied'), here we
// are simply decoding a clean document and the dirty flag is an artifact
// of the current client.Document interface.
@@ -141,8 +143,8 @@ func (encdoc *encodedDocument) MergeProperties(other EncodedDocument) {
for field, prop := range otherEncDoc.properties {
encdoc.properties[field] = prop
}
- if other.Key() != nil {
- encdoc.key = other.Key()
+ if other.ID() != nil {
+ encdoc.id = other.ID()
}
if other.SchemaVersionID() != "" {
encdoc.schemaVersionID = other.SchemaVersionID()
@@ -153,7 +155,7 @@ func (encdoc *encodedDocument) MergeProperties(other EncodedDocument) {
// map of field/value pairs
func DecodeToDoc(encdoc EncodedDocument, mapping *core.DocumentMapping, filter bool) (core.Doc, error) {
doc := mapping.NewDoc()
- doc.SetKey(string(encdoc.Key()))
+ doc.SetID(string(encdoc.ID()))
properties, err := encdoc.Properties(filter)
if err != nil {
diff --git a/db/fetcher/fetcher.go b/db/fetcher/fetcher.go
index da7a0df1e1..a9cb39d9d5 100644
--- a/db/fetcher/fetcher.go
+++ b/db/fetcher/fetcher.go
@@ -127,7 +127,7 @@ type DocumentFetcher struct {
// Since deleted documents are stored under a different instance type than active documents,
// we use a parallel fetcher to be able to return the documents in the expected order.
- // That being lexicographically ordered dockeys.
+ // That being lexicographically ordered docIDs.
deletedDocFetcher *DocumentFetcher
execInfo ExecInfo
@@ -250,7 +250,7 @@ func (df *DocumentFetcher) start(ctx context.Context, spans core.Spans, withDele
df.deletedDocs = withDeleted
if !spans.HasValue { // no specified spans so create a prefix scan key for the entire collection
- start := base.MakeCollectionKey(df.col.Description())
+ start := base.MakeDataStoreKeyWithCollectionDescription(df.col.Description())
if withDeleted {
start = start.WithDeletedFlag()
} else {
@@ -334,7 +334,7 @@ func (df *DocumentFetcher) nextKey(ctx context.Context, seekNext bool) (spanDone
if seekNext {
curKey := df.kv.Key
- curKey.FieldId = "" // clear field so prefixEnd applies to dockey
+ curKey.FieldId = "" // clear field so prefixEnd applies to docID
seekKey := curKey.PrefixEnd().ToString()
spanDone, df.kv, err = df.seekKV(seekKey)
// handle any internal errors
@@ -370,7 +370,7 @@ func (df *DocumentFetcher) nextKey(ctx context.Context, seekNext bool) (spanDone
}
// check if we've crossed document boundries
- if (df.doc.key != nil && df.kv.Key.DocKey != string(df.doc.key)) || seekNext {
+ if (df.doc.id != nil && df.kv.Key.DocID != string(df.doc.id)) || seekNext {
df.isReadingDocument = false
return false, true, nil
}
@@ -472,10 +472,10 @@ func (df *DocumentFetcher) processKV(kv *keyValue) error {
if df.filterSet != nil {
df.doc.filterSet = bitset.New(df.filterSet.Len())
if df.filterSet.Test(0) {
- df.doc.filterSet.Set(0) // mark dockey as set
+ df.doc.filterSet.Set(0) // mark docID as set
}
}
- df.doc.key = []byte(kv.Key.DocKey)
+ df.doc.id = []byte(kv.Key.DocID)
df.passedFilter = false
df.ranFilter = false
@@ -534,15 +534,15 @@ func (df *DocumentFetcher) FetchNext(ctx context.Context) (EncodedDocument, Exec
var resultExecInfo ExecInfo
// If the deletedDocFetcher isn't nil, this means that the user requested to include the deleted documents
- // in the query. To keep the active and deleted docs in lexicographic order of dockeys, we use the two distinct
- // fetchers and fetch the one that has the next lowest (or highest if requested in reverse order) dockey value.
+ // in the query. To keep the active and deleted docs in lexicographic order of docIDs, we use the two distinct
+ // fetchers and fetch the one that has the next lowest (or highest if requested in reverse order) docID value.
ddf := df.deletedDocFetcher
if ddf != nil {
// If we've reached the end of the deleted docs, we can skip to getting the next active docs.
if !ddf.kvEnd {
if df.kvEnd ||
- (df.reverse && ddf.kv.Key.DocKey > df.kv.Key.DocKey) ||
- (!df.reverse && ddf.kv.Key.DocKey < df.kv.Key.DocKey) {
+ (df.reverse && ddf.kv.Key.DocID > df.kv.Key.DocID) ||
+ (!df.reverse && ddf.kv.Key.DocID < df.kv.Key.DocID) {
encdoc, execInfo, err := ddf.FetchNext(ctx)
if err != nil {
return nil, ExecInfo{}, err
@@ -573,7 +573,7 @@ func (df *DocumentFetcher) fetchNext(ctx context.Context) (EncodedDocument, Exec
if df.kv == nil {
return nil, ExecInfo{}, client.NewErrUninitializeProperty("DocumentFetcher", "kv")
}
- // save the DocKey of the current kv pair so we can track when we cross the doc pair boundries
+ // save the DocID of the current kv pair so we can track when we cross the doc pair boundries
// keyparts := df.kv.Key.List()
// key := keyparts[len(keyparts)-2]
diff --git a/db/fetcher/indexer.go b/db/fetcher/indexer.go
index a0ee94d0b9..b8608e2b7d 100644
--- a/db/fetcher/indexer.go
+++ b/db/fetcher/indexer.go
@@ -32,6 +32,7 @@ type IndexFetcher struct {
mapping *core.DocumentMapping
indexedField client.FieldDescription
docFields []client.FieldDescription
+ indexDesc client.IndexDescription
indexIter indexIterator
indexDataStoreKey core.IndexDataStoreKey
execInfo ExecInfo
@@ -70,6 +71,7 @@ func (f *IndexFetcher) Init(
for _, index := range col.Description().Indexes {
if index.Fields[0].Name == f.indexedField.Name {
+ f.indexDesc = index
f.indexDataStoreKey.IndexID = index.ID
break
}
@@ -84,7 +86,7 @@ func (f *IndexFetcher) Init(
}
}
- iter, err := createIndexIterator(f.indexDataStoreKey, f.indexFilter, &f.execInfo)
+ iter, err := createIndexIterator(f.indexDataStoreKey, f.indexFilter, &f.execInfo, f.indexDesc.Unique)
if err != nil {
return err
}
@@ -112,28 +114,32 @@ func (f *IndexFetcher) FetchNext(ctx context.Context) (EncodedDocument, ExecInfo
for {
f.doc.Reset()
- indexKey, hasValue, err := f.indexIter.Next()
+ res, err := f.indexIter.Next()
if err != nil {
return nil, ExecInfo{}, err
}
- if !hasValue {
+ if !res.foundKey {
return nil, f.execInfo, nil
}
property := &encProperty{
Desc: f.indexedField,
- Raw: indexKey.FieldValues[0],
+ Raw: res.key.FieldValues[0],
}
- f.doc.key = indexKey.FieldValues[1]
+ if f.indexDesc.Unique {
+ f.doc.id = res.value
+ } else {
+ f.doc.id = res.key.FieldValues[1]
+ }
f.doc.properties[f.indexedField] = property
f.execInfo.FieldsFetched++
if f.docFetcher != nil && len(f.docFields) > 0 {
- targetKey := base.MakeDocKey(f.col.Description(), string(f.doc.key))
+ targetKey := base.MakeDataStoreKeyWithCollectionAndDocID(f.col.Description(), string(f.doc.id))
spans := core.NewSpans(core.NewSpan(targetKey, targetKey.PrefixEnd()))
- err = f.docFetcher.Start(ctx, spans)
+ err := f.docFetcher.Start(ctx, spans)
if err != nil {
return nil, ExecInfo{}, err
}
diff --git a/db/fetcher/indexer_iterators.go b/db/fetcher/indexer_iterators.go
index b563c9b3a3..aa24605559 100644
--- a/db/fetcher/indexer_iterators.go
+++ b/db/fetcher/indexer_iterators.go
@@ -17,6 +17,7 @@ import (
"strings"
"github.com/fxamacker/cbor/v2"
+ ds "github.com/ipfs/go-datastore"
"github.com/sourcenetwork/defradb/client"
"github.com/sourcenetwork/defradb/connor"
@@ -45,45 +46,51 @@ const (
// For example, iteration over condition _eq and _gt will have completely different logic.
type indexIterator interface {
Init(context.Context, datastore.DSReaderWriter) error
- Next() (core.IndexDataStoreKey, bool, error)
+ Next() (indexIterResult, error)
Close() error
}
+type indexIterResult struct {
+ key core.IndexDataStoreKey
+ foundKey bool
+ value []byte
+}
+
type queryResultIterator struct {
resultIter query.Results
}
-func (i queryResultIterator) Next() (core.IndexDataStoreKey, bool, error) {
+func (i *queryResultIterator) Next() (indexIterResult, error) {
res, hasVal := i.resultIter.NextSync()
if res.Error != nil {
- return core.IndexDataStoreKey{}, false, res.Error
+ return indexIterResult{}, res.Error
}
if !hasVal {
- return core.IndexDataStoreKey{}, false, nil
+ return indexIterResult{}, nil
}
key, err := core.NewIndexDataStoreKey(res.Key)
if err != nil {
- return core.IndexDataStoreKey{}, false, err
+ return indexIterResult{}, err
}
- return key, true, nil
+ return indexIterResult{key: key, value: res.Value, foundKey: true}, nil
}
-func (i queryResultIterator) Close() error {
+func (i *queryResultIterator) Close() error {
return i.resultIter.Close()
}
-type eqIndexIterator struct {
+type eqPrefixIndexIterator struct {
+ filterValueHolder
+ indexKey core.IndexDataStoreKey
+ execInfo *ExecInfo
+
queryResultIterator
- indexKey core.IndexDataStoreKey
- filterVal []byte
- execInfo *ExecInfo
}
-func (i *eqIndexIterator) Init(ctx context.Context, store datastore.DSReaderWriter) error {
- i.indexKey.FieldValues = [][]byte{i.filterVal}
+func (i *eqPrefixIndexIterator) Init(ctx context.Context, store datastore.DSReaderWriter) error {
+ i.indexKey.FieldValues = [][]byte{i.value}
resultIter, err := store.Query(ctx, query.Query{
- Prefix: i.indexKey.ToString(),
- KeysOnly: true,
+ Prefix: i.indexKey.ToString(),
})
if err != nil {
return err
@@ -92,16 +99,65 @@ func (i *eqIndexIterator) Init(ctx context.Context, store datastore.DSReaderWrit
return nil
}
-func (i *eqIndexIterator) Next() (core.IndexDataStoreKey, bool, error) {
- key, hasValue, err := i.queryResultIterator.Next()
- if hasValue {
+func (i *eqPrefixIndexIterator) Next() (indexIterResult, error) {
+ res, err := i.queryResultIterator.Next()
+ if res.foundKey {
i.execInfo.IndexesFetched++
}
- return key, hasValue, err
+ return res, err
+}
+
+type filterValueIndexIterator interface {
+ indexIterator
+ SetFilterValue([]byte)
+}
+
+type filterValueHolder struct {
+ value []byte
+}
+
+func (h *filterValueHolder) SetFilterValue(value []byte) {
+ h.value = value
+}
+
+type eqSingleIndexIterator struct {
+ filterValueHolder
+ indexKey core.IndexDataStoreKey
+ execInfo *ExecInfo
+
+ ctx context.Context
+ store datastore.DSReaderWriter
+}
+
+func (i *eqSingleIndexIterator) Init(ctx context.Context, store datastore.DSReaderWriter) error {
+ i.ctx = ctx
+ i.store = store
+ return nil
+}
+
+func (i *eqSingleIndexIterator) Next() (indexIterResult, error) {
+ if i.store == nil {
+ return indexIterResult{}, nil
+ }
+ i.indexKey.FieldValues = [][]byte{i.value}
+ val, err := i.store.Get(i.ctx, i.indexKey.ToDS())
+ if err != nil {
+ if errors.Is(err, ds.ErrNotFound) {
+ return indexIterResult{key: i.indexKey}, nil
+ }
+ return indexIterResult{}, err
+ }
+ i.store = nil
+ i.execInfo.IndexesFetched++
+ return indexIterResult{key: i.indexKey, value: val, foundKey: true}, nil
+}
+
+func (i *eqSingleIndexIterator) Close() error {
+ return nil
}
type inIndexIterator struct {
- eqIndexIterator
+ filterValueIndexIterator
filterValues [][]byte
nextValIndex int
ctx context.Context
@@ -110,22 +166,18 @@ type inIndexIterator struct {
}
func newInIndexIterator(
- indexKey core.IndexDataStoreKey,
+ indexIter filterValueIndexIterator,
filterValues [][]byte,
- execInfo *ExecInfo,
) *inIndexIterator {
return &inIndexIterator{
- eqIndexIterator: eqIndexIterator{
- indexKey: indexKey,
- execInfo: execInfo,
- },
- filterValues: filterValues,
+ filterValueIndexIterator: indexIter,
+ filterValues: filterValues,
}
}
func (i *inIndexIterator) nextIterator() (bool, error) {
if i.nextValIndex > 0 {
- err := i.eqIndexIterator.Close()
+ err := i.filterValueIndexIterator.Close()
if err != nil {
return false, err
}
@@ -135,8 +187,8 @@ func (i *inIndexIterator) nextIterator() (bool, error) {
return false, nil
}
- i.filterVal = i.filterValues[i.nextValIndex]
- err := i.eqIndexIterator.Init(i.ctx, i.store)
+ i.SetFilterValue(i.filterValues[i.nextValIndex])
+ err := i.filterValueIndexIterator.Init(i.ctx, i.store)
if err != nil {
return false, err
}
@@ -152,22 +204,22 @@ func (i *inIndexIterator) Init(ctx context.Context, store datastore.DSReaderWrit
return err
}
-func (i *inIndexIterator) Next() (core.IndexDataStoreKey, bool, error) {
+func (i *inIndexIterator) Next() (indexIterResult, error) {
for i.hasIterator {
- key, hasValue, err := i.eqIndexIterator.Next()
+ res, err := i.filterValueIndexIterator.Next()
if err != nil {
- return core.IndexDataStoreKey{}, false, err
+ return indexIterResult{}, err
}
- if !hasValue {
+ if !res.foundKey {
i.hasIterator, err = i.nextIterator()
if err != nil {
- return core.IndexDataStoreKey{}, false, err
+ return indexIterResult{}, err
}
continue
}
- return key, true, nil
+ return res, nil
}
- return core.IndexDataStoreKey{}, false, nil
+ return indexIterResult{}, nil
}
func (i *inIndexIterator) Close() error {
@@ -220,9 +272,8 @@ func (i *scanningIndexIterator) Init(ctx context.Context, store datastore.DSRead
i.filter.matcher = &execInfoIndexMatcherDecorator{matcher: i.matcher, execInfo: i.execInfo}
iter, err := store.Query(ctx, query.Query{
- Prefix: i.indexKey.ToString(),
- KeysOnly: true,
- Filters: []query.Filter{&i.filter},
+ Prefix: i.indexKey.ToString(),
+ Filters: []query.Filter{&i.filter},
})
if err != nil {
return err
@@ -232,12 +283,12 @@ func (i *scanningIndexIterator) Init(ctx context.Context, store datastore.DSRead
return nil
}
-func (i *scanningIndexIterator) Next() (core.IndexDataStoreKey, bool, error) {
- key, hasValue, err := i.queryResultIterator.Next()
+func (i *scanningIndexIterator) Next() (indexIterResult, error) {
+ res, err := i.queryResultIterator.Next()
if i.filter.err != nil {
- return core.IndexDataStoreKey{}, false, i.filter.err
+ return indexIterResult{}, i.filter.err
}
- return key, hasValue, err
+ return res, err
}
// checks if the stored index value satisfies the condition
@@ -348,6 +399,7 @@ func createIndexIterator(
indexDataStoreKey core.IndexDataStoreKey,
indexFilterConditions *mapper.Filter,
execInfo *ExecInfo,
+ isUnique bool,
) (indexIterator, error) {
var op string
var filterVal any
@@ -364,20 +416,32 @@ func createIndexIterator(
switch op {
case opEq, opGt, opGe, opLt, opLe, opNe:
- writableValue := client.NewCBORValue(client.LWW_REGISTER, filterVal)
+ fieldValue := client.NewFieldValue(client.LWW_REGISTER, filterVal)
- valueBytes, err := writableValue.Bytes()
+ valueBytes, err := fieldValue.Bytes()
if err != nil {
return nil, err
}
switch op {
case opEq:
- return &eqIndexIterator{
- indexKey: indexDataStoreKey,
- filterVal: valueBytes,
- execInfo: execInfo,
- }, nil
+ if isUnique {
+ return &eqSingleIndexIterator{
+ indexKey: indexDataStoreKey,
+ filterValueHolder: filterValueHolder{
+ value: valueBytes,
+ },
+ execInfo: execInfo,
+ }, nil
+ } else {
+ return &eqPrefixIndexIterator{
+ indexKey: indexDataStoreKey,
+ filterValueHolder: filterValueHolder{
+ value: valueBytes,
+ },
+ execInfo: execInfo,
+ }, nil
+ }
case opGt:
return &scanningIndexIterator{
indexKey: indexDataStoreKey,
@@ -430,15 +494,27 @@ func createIndexIterator(
}
valArr := make([][]byte, 0, len(inArr))
for _, v := range inArr {
- writableValue := client.NewCBORValue(client.LWW_REGISTER, v)
- valueBytes, err := writableValue.Bytes()
+ fieldValue := client.NewFieldValue(client.LWW_REGISTER, v)
+ valueBytes, err := fieldValue.Bytes()
if err != nil {
return nil, err
}
valArr = append(valArr, valueBytes)
}
if op == opIn {
- return newInIndexIterator(indexDataStoreKey, valArr, execInfo), nil
+ var iter filterValueIndexIterator
+ if isUnique {
+ iter = &eqSingleIndexIterator{
+ indexKey: indexDataStoreKey,
+ execInfo: execInfo,
+ }
+ } else {
+ iter = &eqPrefixIndexIterator{
+ indexKey: indexDataStoreKey,
+ execInfo: execInfo,
+ }
+ }
+ return newInIndexIterator(iter, valArr), nil
} else {
return &scanningIndexIterator{
indexKey: indexDataStoreKey,
diff --git a/db/fetcher/mocks/encoded_document.go b/db/fetcher/mocks/encoded_document.go
index 538d32ff4d..5d9382a14d 100644
--- a/db/fetcher/mocks/encoded_document.go
+++ b/db/fetcher/mocks/encoded_document.go
@@ -21,8 +21,8 @@ func (_m *EncodedDocument) EXPECT() *EncodedDocument_Expecter {
return &EncodedDocument_Expecter{mock: &_m.Mock}
}
-// Key provides a mock function with given fields:
-func (_m *EncodedDocument) Key() []byte {
+// ID provides a mock function with given fields:
+func (_m *EncodedDocument) ID() []byte {
ret := _m.Called()
var r0 []byte
@@ -37,29 +37,29 @@ func (_m *EncodedDocument) Key() []byte {
return r0
}
-// EncodedDocument_Key_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Key'
-type EncodedDocument_Key_Call struct {
+// EncodedDocument_ID_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'ID'
+type EncodedDocument_ID_Call struct {
*mock.Call
}
-// Key is a helper method to define mock.On call
-func (_e *EncodedDocument_Expecter) Key() *EncodedDocument_Key_Call {
- return &EncodedDocument_Key_Call{Call: _e.mock.On("Key")}
+// ID is a helper method to define mock.On call
+func (_e *EncodedDocument_Expecter) ID() *EncodedDocument_ID_Call {
+ return &EncodedDocument_ID_Call{Call: _e.mock.On("ID")}
}
-func (_c *EncodedDocument_Key_Call) Run(run func()) *EncodedDocument_Key_Call {
+func (_c *EncodedDocument_ID_Call) Run(run func()) *EncodedDocument_ID_Call {
_c.Call.Run(func(args mock.Arguments) {
run()
})
return _c
}
-func (_c *EncodedDocument_Key_Call) Return(_a0 []byte) *EncodedDocument_Key_Call {
+func (_c *EncodedDocument_ID_Call) Return(_a0 []byte) *EncodedDocument_ID_Call {
_c.Call.Return(_a0)
return _c
}
-func (_c *EncodedDocument_Key_Call) RunAndReturn(run func() []byte) *EncodedDocument_Key_Call {
+func (_c *EncodedDocument_ID_Call) RunAndReturn(run func() []byte) *EncodedDocument_ID_Call {
_c.Call.Return(run)
return _c
}
diff --git a/db/fetcher/versioned.go b/db/fetcher/versioned.go
index 4ab8ef54a7..c33f1a35da 100644
--- a/db/fetcher/versioned.go
+++ b/db/fetcher/versioned.go
@@ -25,8 +25,7 @@ import (
"github.com/sourcenetwork/defradb/datastore/memory"
"github.com/sourcenetwork/defradb/db/base"
"github.com/sourcenetwork/defradb/errors"
- "github.com/sourcenetwork/defradb/events"
- "github.com/sourcenetwork/defradb/merkle/crdt"
+ merklecrdt "github.com/sourcenetwork/defradb/merkle/crdt"
"github.com/sourcenetwork/defradb/planner/mapper"
)
@@ -87,14 +86,14 @@ type VersionedFetcher struct {
root datastore.RootStore
store datastore.Txn
- key core.DataStoreKey
+ dsKey core.DataStoreKey
version cid.Cid
queuedCids *list.List
col client.Collection
// @todo index *client.IndexDescription
- mCRDTs map[uint32]crdt.MerkleCRDT
+ mCRDTs map[uint32]merklecrdt.MerkleCRDT
}
// Init initializes the VersionedFetcher.
@@ -110,7 +109,7 @@ func (vf *VersionedFetcher) Init(
) error {
vf.col = col
vf.queuedCids = list.New()
- vf.mCRDTs = make(map[uint32]crdt.MerkleCRDT)
+ vf.mCRDTs = make(map[uint32]merklecrdt.MerkleCRDT)
vf.txn = txn
// create store
@@ -145,25 +144,25 @@ func (vf *VersionedFetcher) Start(ctx context.Context, spans core.Spans) error {
}
// For the VersionedFetcher, the spans needs to be in the format
- // Span{Start: DocKey, End: CID}
+ // Span{Start: DocID, End: CID}
dk := spans.Value[0].Start()
cidRaw := spans.Value[0].End()
- if dk.DocKey == "" {
- return client.NewErrUninitializeProperty("Spans", "DocKey")
- } else if cidRaw.DocKey == "" { // todo: dont abuse DataStoreKey/Span like this!
+ if dk.DocID == "" {
+ return client.NewErrUninitializeProperty("Spans", "DocID")
+ } else if cidRaw.DocID == "" { // todo: dont abuse DataStoreKey/Span like this!
return client.NewErrUninitializeProperty("Spans", "CID")
}
// decode cidRaw from core.Key to cid.Cid
// need to remove '/' prefix from the core.Key
- c, err := cid.Decode(cidRaw.DocKey)
+ c, err := cid.Decode(cidRaw.DocID)
if err != nil {
return NewErrFailedToDecodeCIDForVFetcher(err)
}
vf.ctx = ctx
- vf.key = dk
+ vf.dsKey = dk
vf.version = c
if err := vf.seekTo(vf.version); err != nil {
@@ -181,7 +180,7 @@ func (vf *VersionedFetcher) Rootstore() ds.Datastore {
// Start a fetcher with the needed info (cid embedded in a span)
/*
-1. Init with DocKey (VersionedFetched is scoped to a single doc)
+1. Init with DocID (VersionedFetched is scoped to a single doc)
2. - Create transient stores (head, data, block)
3. Start with a given Txn and CID span set (length 1 for now)
4. call traverse with the target cid
@@ -259,8 +258,8 @@ func (vf *VersionedFetcher) seekNext(c cid.Cid, topParent bool) error {
// check if cid block exists in the global store, handle err
// @todo: Find an efficient way to determine if a CID is a member of a
- // DocKey State graph
- // @body: We could possibly append the DocKey to the CID either as a
+ // DocID State graph
+ // @body: We could possibly append the DocID to the CID either as a
// child key, or an instance on the CID key.
hasLocalBlock, err := vf.store.DAGstore().Has(vf.ctx, c)
@@ -340,7 +339,7 @@ func (vf *VersionedFetcher) merge(c cid.Cid) error {
}
// first arg 0 is the index for the composite DAG in the mCRDTs cache
- if err := vf.processNode(0, nd, client.COMPOSITE, ""); err != nil {
+ if err := vf.processNode(0, nd, client.COMPOSITE, client.FieldKind_None, ""); err != nil {
return err
}
@@ -362,9 +361,7 @@ func (vf *VersionedFetcher) merge(c cid.Cid) error {
if !ok {
return client.NewErrFieldNotExist(l.Name)
}
- // @todo: Right now we ONLY handle LWW_REGISTER, need to swith on this and
- // get CType from descriptions
- if err := vf.processNode(uint32(field.ID), subNd, client.LWW_REGISTER, l.Name); err != nil {
+ if err := vf.processNode(uint32(field.ID), subNd, field.Typ, field.Kind, l.Name); err != nil {
return err
}
}
@@ -376,21 +373,22 @@ func (vf *VersionedFetcher) processNode(
crdtIndex uint32,
nd format.Node,
ctype client.CType,
+ kind client.FieldKind,
fieldName string,
) (err error) {
// handle CompositeDAG
mcrdt, exists := vf.mCRDTs[crdtIndex]
if !exists {
- key, err := base.MakePrimaryIndexKeyForCRDT(vf.col.Description(), vf.col.Schema(), ctype, vf.key, fieldName)
+ dsKey, err := base.MakePrimaryIndexKeyForCRDT(vf.col.Description(), vf.col.Schema(), ctype, vf.dsKey, fieldName)
if err != nil {
return err
}
- mcrdt, err = crdt.DefaultFactory.InstanceWithStores(
+ mcrdt, err = merklecrdt.InstanceWithStore(
vf.store,
core.CollectionSchemaVersionKey{},
- events.EmptyUpdateChannel,
ctype,
- key,
+ kind,
+ dsKey,
fieldName,
)
if err != nil {
@@ -431,7 +429,7 @@ func (vf *VersionedFetcher) Close() error {
}
// NewVersionedSpan creates a new VersionedSpan from a DataStoreKey and a version CID.
-func NewVersionedSpan(dockey core.DataStoreKey, version cid.Cid) core.Spans {
+func NewVersionedSpan(dsKey core.DataStoreKey, version cid.Cid) core.Spans {
// Todo: Dont abuse DataStoreKey for version cid!
- return core.NewSpans(core.NewSpan(dockey, core.DataStoreKey{DocKey: version.String()}))
+ return core.NewSpans(core.NewSpan(dsKey, core.DataStoreKey{DocID: version.String()}))
}
diff --git a/db/index.go b/db/index.go
index ce9e55f519..59fd25eaa9 100644
--- a/db/index.go
+++ b/db/index.go
@@ -18,6 +18,7 @@ import (
"github.com/sourcenetwork/defradb/core"
"github.com/sourcenetwork/defradb/datastore"
"github.com/sourcenetwork/defradb/errors"
+ "github.com/sourcenetwork/defradb/request/graphql/schema/types"
)
// CollectionIndex is an interface for collection indexes
@@ -51,6 +52,14 @@ func getValidateIndexFieldFunc(kind client.FieldKind) func(any) bool {
return canConvertIndexFieldValue[float64]
case client.FieldKind_BOOL:
return canConvertIndexFieldValue[bool]
+ case client.FieldKind_BLOB:
+ return func(val any) bool {
+ blobStrVal, ok := val.(string)
+ if !ok {
+ return false
+ }
+ return types.BlobPattern.MatchString(blobStrVal)
+ }
case client.FieldKind_DATETIME:
return func(val any) bool {
timeStrVal, ok := val.(string)
@@ -81,29 +90,50 @@ func NewCollectionIndex(
if len(desc.Fields) == 0 {
return nil, NewErrIndexDescHasNoFields(desc)
}
- index := &collectionSimpleIndex{collection: collection, desc: desc}
field, foundField := collection.Schema().GetField(desc.Fields[0].Name)
if !foundField {
return nil, NewErrIndexDescHasNonExistingField(desc, desc.Fields[0].Name)
}
- var e error
- index.fieldDesc = field
- index.validateFieldFunc, e = getFieldValidateFunc(field.Kind)
- return index, e
+ base := collectionBaseIndex{collection: collection, desc: desc}
+ base.fieldDesc = field
+ var err error
+ base.validateFieldFunc, err = getFieldValidateFunc(field.Kind)
+ if err != nil {
+ return nil, err
+ }
+ if desc.Unique {
+ return &collectionUniqueIndex{collectionBaseIndex: base}, nil
+ } else {
+ return &collectionSimpleIndex{collectionBaseIndex: base}, nil
+ }
}
-// collectionSimpleIndex is an non-unique index that indexes documents by a single field.
-// Single-field indexes store values only in ascending order.
-type collectionSimpleIndex struct {
+type collectionBaseIndex struct {
collection client.Collection
desc client.IndexDescription
validateFieldFunc func(any) bool
fieldDesc client.FieldDescription
}
-var _ CollectionIndex = (*collectionSimpleIndex)(nil)
+func (i *collectionBaseIndex) getDocFieldValue(doc *client.Document) ([]byte, error) {
+ // collectionSimpleIndex only supports single field indexes, that's why we
+ // can safely access the first field
+ indexedFieldName := i.desc.Fields[0].Name
+ fieldVal, err := doc.GetValue(indexedFieldName)
+ if err != nil {
+ if errors.Is(err, client.ErrFieldNotExist) {
+ return client.NewFieldValue(client.LWW_REGISTER, nil).Bytes()
+ } else {
+ return nil, err
+ }
+ }
+ if !i.validateFieldFunc(fieldVal.Value()) {
+ return nil, NewErrInvalidFieldValue(i.fieldDesc.Kind, fieldVal)
+ }
+ return fieldVal.Bytes()
+}
-func (i *collectionSimpleIndex) getDocumentsIndexKey(
+func (i *collectionBaseIndex) getDocumentsIndexKey(
doc *client.Document,
) (core.IndexDataStoreKey, error) {
fieldValue, err := i.getDocFieldValue(doc)
@@ -114,27 +144,75 @@ func (i *collectionSimpleIndex) getDocumentsIndexKey(
indexDataStoreKey := core.IndexDataStoreKey{}
indexDataStoreKey.CollectionID = i.collection.ID()
indexDataStoreKey.IndexID = i.desc.ID
- indexDataStoreKey.FieldValues = [][]byte{fieldValue, []byte(doc.Key().String())}
+ indexDataStoreKey.FieldValues = [][]byte{fieldValue}
return indexDataStoreKey, nil
}
-func (i *collectionSimpleIndex) getDocFieldValue(doc *client.Document) ([]byte, error) {
- // collectionSimpleIndex only supports single field indexes, that's why we
- // can safely access the first field
- indexedFieldName := i.desc.Fields[0].Name
- fieldVal, err := doc.GetValue(indexedFieldName)
+func (i *collectionBaseIndex) deleteIndexKey(
+ ctx context.Context,
+ txn datastore.Txn,
+ key core.IndexDataStoreKey,
+) error {
+ exists, err := txn.Datastore().Has(ctx, key.ToDS())
if err != nil {
- if errors.Is(err, client.ErrFieldNotExist) {
- return client.NewCBORValue(client.LWW_REGISTER, nil).Bytes()
- } else {
- return nil, err
+ return err
+ }
+ if !exists {
+ return NewErrCorruptedIndex(i.desc.Name)
+ }
+ return txn.Datastore().Delete(ctx, key.ToDS())
+}
+
+// RemoveAll remove all artifacts of the index from the storage, i.e. all index
+// field values for all documents.
+func (i *collectionBaseIndex) RemoveAll(ctx context.Context, txn datastore.Txn) error {
+ prefixKey := core.IndexDataStoreKey{}
+ prefixKey.CollectionID = i.collection.ID()
+ prefixKey.IndexID = i.desc.ID
+
+ keys, err := datastore.FetchKeysForPrefix(ctx, prefixKey.ToString(), txn.Datastore())
+ if err != nil {
+ return err
+ }
+
+ for _, key := range keys {
+ err := txn.Datastore().Delete(ctx, key)
+ if err != nil {
+ return NewCanNotDeleteIndexedField(err)
}
}
- writeableVal, ok := fieldVal.(client.WriteableValue)
- if !ok || !i.validateFieldFunc(fieldVal.Value()) {
- return nil, NewErrInvalidFieldValue(i.fieldDesc.Kind, writeableVal)
+
+ return nil
+}
+
+// Name returns the name of the index
+func (i *collectionBaseIndex) Name() string {
+ return i.desc.Name
+}
+
+// Description returns the description of the index
+func (i *collectionBaseIndex) Description() client.IndexDescription {
+ return i.desc
+}
+
+// collectionSimpleIndex is an non-unique index that indexes documents by a single field.
+// Single-field indexes store values only in ascending order.
+type collectionSimpleIndex struct {
+ collectionBaseIndex
+}
+
+var _ CollectionIndex = (*collectionSimpleIndex)(nil)
+
+func (i *collectionSimpleIndex) getDocumentsIndexKey(
+ doc *client.Document,
+) (core.IndexDataStoreKey, error) {
+ key, err := i.collectionBaseIndex.getDocumentsIndexKey(doc)
+ if err != nil {
+ return core.IndexDataStoreKey{}, err
}
- return writeableVal.Bytes()
+
+ key.FieldValues = append(key.FieldValues, []byte(doc.ID().String()))
+ return key, nil
}
// Save indexes a document by storing the indexed field value.
@@ -154,60 +232,110 @@ func (i *collectionSimpleIndex) Save(
return nil
}
-// Update updates indexed field values of an existing document.
-// It removes the old document from the index and adds the new one.
func (i *collectionSimpleIndex) Update(
ctx context.Context,
txn datastore.Txn,
oldDoc *client.Document,
newDoc *client.Document,
) error {
- key, err := i.getDocumentsIndexKey(oldDoc)
- if err != nil {
- return err
- }
- exists, err := txn.Datastore().Has(ctx, key.ToDS())
+ err := i.deleteDocIndex(ctx, txn, oldDoc)
if err != nil {
return err
}
- if !exists {
- return NewErrCorruptedIndex(i.desc.Name)
- }
- err = txn.Datastore().Delete(ctx, key.ToDS())
+ return i.Save(ctx, txn, newDoc)
+}
+
+func (i *collectionSimpleIndex) deleteDocIndex(
+ ctx context.Context,
+ txn datastore.Txn,
+ doc *client.Document,
+) error {
+ key, err := i.getDocumentsIndexKey(doc)
if err != nil {
return err
}
- return i.Save(ctx, txn, newDoc)
+ return i.deleteIndexKey(ctx, txn, key)
}
-// RemoveAll remove all artifacts of the index from the storage, i.e. all index
-// field values for all documents.
-func (i *collectionSimpleIndex) RemoveAll(ctx context.Context, txn datastore.Txn) error {
- prefixKey := core.IndexDataStoreKey{}
- prefixKey.CollectionID = i.collection.ID()
- prefixKey.IndexID = i.desc.ID
+type collectionUniqueIndex struct {
+ collectionBaseIndex
+}
- keys, err := datastore.FetchKeysForPrefix(ctx, prefixKey.ToString(), txn.Datastore())
+var _ CollectionIndex = (*collectionUniqueIndex)(nil)
+
+func (i *collectionUniqueIndex) Save(
+ ctx context.Context,
+ txn datastore.Txn,
+ doc *client.Document,
+) error {
+ key, err := i.getDocumentsIndexKey(doc)
+ if err != nil {
+ return err
+ }
+ exists, err := txn.Datastore().Has(ctx, key.ToDS())
if err != nil {
return err
}
+ if exists {
+ return i.newUniqueIndexError(doc)
+ }
+ err = txn.Datastore().Put(ctx, key.ToDS(), []byte(doc.ID().String()))
+ if err != nil {
+ return NewErrFailedToStoreIndexedField(key.ToDS().String(), err)
+ }
+ return nil
+}
- for _, key := range keys {
- err := txn.Datastore().Delete(ctx, key)
- if err != nil {
- return NewCanNotDeleteIndexedField(err)
+func (i *collectionUniqueIndex) newUniqueIndexError(
+ doc *client.Document,
+) error {
+ fieldVal, err := doc.GetValue(i.fieldDesc.Name)
+ var val any
+ if err != nil {
+ // If the error is ErrFieldNotExist, we leave `val` as is (e.g. nil)
+ // otherwise we return the error
+ if !errors.Is(err, client.ErrFieldNotExist) {
+ return err
}
+ } else {
+ val = fieldVal.Value()
}
- return nil
+ return NewErrCanNotIndexNonUniqueField(doc.ID().String(), i.fieldDesc.Name, val)
}
-// Name returns the name of the index
-func (i *collectionSimpleIndex) Name() string {
- return i.desc.Name
+func (i *collectionUniqueIndex) Update(
+ ctx context.Context,
+ txn datastore.Txn,
+ oldDoc *client.Document,
+ newDoc *client.Document,
+) error {
+ newKey, err := i.getDocumentsIndexKey(newDoc)
+ if err != nil {
+ return err
+ }
+ exists, err := txn.Datastore().Has(ctx, newKey.ToDS())
+ if err != nil {
+ return err
+ }
+ if exists {
+ return i.newUniqueIndexError(newDoc)
+ }
+ err = i.deleteDocIndex(ctx, txn, oldDoc)
+ if err != nil {
+ return err
+ }
+ return i.Save(ctx, txn, newDoc)
}
-// Description returns the description of the index
-func (i *collectionSimpleIndex) Description() client.IndexDescription {
- return i.desc
+func (i *collectionUniqueIndex) deleteDocIndex(
+ ctx context.Context,
+ txn datastore.Txn,
+ doc *client.Document,
+) error {
+ key, err := i.getDocumentsIndexKey(doc)
+ if err != nil {
+ return err
+ }
+ return i.deleteIndexKey(ctx, txn, key)
}
diff --git a/db/index_test.go b/db/index_test.go
index e5682b551c..911228e649 100644
--- a/db/index_test.go
+++ b/db/index_test.go
@@ -58,7 +58,7 @@ type indexTestFixture struct {
t *testing.T
}
-func (f *indexTestFixture) getUsersCollectionDesc() client.Collection {
+func (f *indexTestFixture) addUsersCollection() client.Collection {
_, err := f.db.AddSchema(
f.ctx,
fmt.Sprintf(
@@ -129,7 +129,7 @@ func newIndexTestFixtureBare(t *testing.T) *indexTestFixture {
func newIndexTestFixture(t *testing.T) *indexTestFixture {
f := newIndexTestFixtureBare(t)
- f.users = f.getUsersCollectionDesc()
+ f.users = f.addUsersCollection()
return f
}
@@ -178,14 +178,24 @@ func getProductsIndexDescOnCategory() client.IndexDescription {
func (f *indexTestFixture) createUserCollectionIndexOnName() client.IndexDescription {
newDesc, err := f.createCollectionIndexFor(f.users.Name(), getUsersIndexDescOnName())
require.NoError(f.t, err)
- f.commitTxn()
+ return newDesc
+}
+
+func makeUnique(indexDesc client.IndexDescription) client.IndexDescription {
+ indexDesc.Unique = true
+ return indexDesc
+}
+
+func (f *indexTestFixture) createUserCollectionUniqueIndexOnName() client.IndexDescription {
+ indexDesc := makeUnique(getUsersIndexDescOnName())
+ newDesc, err := f.createCollectionIndexFor(f.users.Name(), indexDesc)
+ require.NoError(f.t, err)
return newDesc
}
func (f *indexTestFixture) createUserCollectionIndexOnAge() client.IndexDescription {
newDesc, err := f.createCollectionIndexFor(f.users.Name(), getUsersIndexDescOnAge())
require.NoError(f.t, err)
- f.commitTxn()
return newDesc
}
@@ -226,7 +236,11 @@ func (f *indexTestFixture) createCollectionIndexFor(
collectionName string,
desc client.IndexDescription,
) (client.IndexDescription, error) {
- return f.db.createCollectionIndex(f.ctx, f.txn, collectionName, desc)
+ index, err := f.db.createCollectionIndex(f.ctx, f.txn, collectionName, desc)
+ if err == nil {
+ f.commitTxn()
+ }
+ return index, err
}
func (f *indexTestFixture) getAllIndexes() (map[client.CollectionName][]client.IndexDescription, error) {
@@ -239,6 +253,7 @@ func (f *indexTestFixture) getCollectionIndexes(colName string) ([]client.IndexD
func TestCreateIndex_IfFieldsIsEmpty_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
_, err := f.createCollectionIndex(client.IndexDescription{
Name: "some_index_name",
@@ -248,6 +263,7 @@ func TestCreateIndex_IfFieldsIsEmpty_ReturnError(t *testing.T) {
func TestCreateIndex_IfIndexDescriptionIDIsNotZero_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
for _, id := range []uint32{1, 20, 999} {
desc := client.IndexDescription{
@@ -264,6 +280,7 @@ func TestCreateIndex_IfIndexDescriptionIDIsNotZero_ReturnError(t *testing.T) {
func TestCreateIndex_IfValidInput_CreateIndex(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
desc := client.IndexDescription{
Name: "some_index_name",
@@ -275,10 +292,12 @@ func TestCreateIndex_IfValidInput_CreateIndex(t *testing.T) {
assert.NoError(t, err)
assert.Equal(t, desc.Name, resultDesc.Name)
assert.Equal(t, desc.Fields, resultDesc.Fields)
+ assert.Equal(t, desc.Unique, resultDesc.Unique)
}
func TestCreateIndex_IfFieldNameIsEmpty_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
desc := client.IndexDescription{
Name: "some_index_name",
@@ -292,6 +311,7 @@ func TestCreateIndex_IfFieldNameIsEmpty_ReturnError(t *testing.T) {
func TestCreateIndex_IfFieldHasNoDirection_DefaultToAsc(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
desc := client.IndexDescription{
Name: "some_index_name",
@@ -304,6 +324,7 @@ func TestCreateIndex_IfFieldHasNoDirection_DefaultToAsc(t *testing.T) {
func TestCreateIndex_IfSingleFieldInDescOrder_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
desc := client.IndexDescription{
Fields: []client.IndexedFieldDescription{
@@ -316,6 +337,7 @@ func TestCreateIndex_IfSingleFieldInDescOrder_ReturnError(t *testing.T) {
func TestCreateIndex_IfIndexWithNameAlreadyExists_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
name := "some_index_name"
desc1 := client.IndexDescription{
@@ -334,6 +356,7 @@ func TestCreateIndex_IfIndexWithNameAlreadyExists_ReturnError(t *testing.T) {
func TestCreateIndex_IfGeneratedNameMatchesExisting_AddIncrement(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
name := usersColName + "_" + usersAgeFieldName + "_ASC"
desc1 := client.IndexDescription{
@@ -359,6 +382,7 @@ func TestCreateIndex_IfGeneratedNameMatchesExisting_AddIncrement(t *testing.T) {
func TestCreateIndex_ShouldSaveToSystemStorage(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
name := "users_age_ASC"
desc := client.IndexDescription{
@@ -380,6 +404,7 @@ func TestCreateIndex_ShouldSaveToSystemStorage(t *testing.T) {
func TestCreateIndex_IfCollectionDoesntExist_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
desc := client.IndexDescription{
Fields: []client.IndexedFieldDescription{{Name: productsPriceFieldName}},
@@ -391,6 +416,7 @@ func TestCreateIndex_IfCollectionDoesntExist_ReturnError(t *testing.T) {
func TestCreateIndex_IfPropertyDoesntExist_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
const field = "non_existing_field"
desc := client.IndexDescription{
@@ -403,7 +429,7 @@ func TestCreateIndex_IfPropertyDoesntExist_ReturnError(t *testing.T) {
func TestCreateIndex_WithMultipleCollectionsAndIndexes_AssignIncrementedIDPerCollection(t *testing.T) {
f := newIndexTestFixtureBare(t)
- users := f.getUsersCollectionDesc()
+ users := f.addUsersCollection()
products := f.getProductsCollectionDesc()
makeIndex := func(fieldName string) client.IndexDescription {
@@ -433,10 +459,13 @@ func TestCreateIndex_WithMultipleCollectionsAndIndexes_AssignIncrementedIDPerCol
func TestCreateIndex_IfFailsToCreateTxn_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
testErr := errors.New("test error")
mockedRootStore := mocks.NewRootStore(t)
+ mockedRootStore.On("Close").Return(nil)
+
mockedRootStore.EXPECT().NewTransaction(mock.Anything, mock.Anything).Return(nil, testErr)
f.db.rootstore = mockedRootStore
@@ -446,6 +475,7 @@ func TestCreateIndex_IfFailsToCreateTxn_ReturnError(t *testing.T) {
func TestCreateIndex_IfProvideInvalidIndexName_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
indexDesc := getUsersIndexDescOnName()
indexDesc.Name = "!"
@@ -455,6 +485,7 @@ func TestCreateIndex_IfProvideInvalidIndexName_ReturnError(t *testing.T) {
func TestCreateIndex_ShouldUpdateCollectionsDescription(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
indOnName, err := f.users.CreateIndex(f.ctx, getUsersIndexDescOnName())
require.NoError(t, err)
@@ -495,11 +526,11 @@ func TestCreateIndex_IfAttemptToIndexOnUnsupportedType_ReturnError(t *testing.T)
_, err = f.createCollectionIndexFor(collection.Name(), indexDesc)
require.ErrorIs(f.t, err, NewErrUnsupportedIndexFieldType(unsupportedKind))
- f.commitTxn()
}
func TestGetIndexes_ShouldReturnListOfAllExistingIndexes(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
usersIndexDesc := client.IndexDescription{
Name: "users_name_index",
@@ -508,8 +539,6 @@ func TestGetIndexes_ShouldReturnListOfAllExistingIndexes(t *testing.T) {
_, err := f.createCollectionIndexFor(usersColName, usersIndexDesc)
assert.NoError(t, err)
- f.commitTxn()
-
f.getProductsCollectionDesc()
productsIndexDesc := client.IndexDescription{
Name: "products_description_index",
@@ -531,6 +560,7 @@ func TestGetIndexes_ShouldReturnListOfAllExistingIndexes(t *testing.T) {
func TestGetIndexes_IfInvalidIndexIsStored_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
indexKey := core.NewCollectionIndexKey(usersColName, "users_name_index")
err := f.txn.Systemstore().Put(f.ctx, indexKey.ToDS(), []byte("invalid"))
@@ -542,6 +572,7 @@ func TestGetIndexes_IfInvalidIndexIsStored_ReturnError(t *testing.T) {
func TestGetIndexes_IfInvalidIndexKeyIsStored_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
indexKey := core.NewCollectionIndexKey(usersColName, "users_name_index")
key := ds.NewKey(indexKey.ToString() + "/invalid")
@@ -561,6 +592,7 @@ func TestGetIndexes_IfInvalidIndexKeyIsStored_ReturnError(t *testing.T) {
func TestGetIndexes_IfSystemStoreFails_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
testErr := errors.New("test error")
@@ -575,6 +607,7 @@ func TestGetIndexes_IfSystemStoreFails_ReturnError(t *testing.T) {
func TestGetIndexes_IfSystemStoreFails_ShouldCloseIterator(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
mockedTxn := f.mockTxn()
mockedTxn.MockSystemstore.EXPECT().Query(mock.Anything, mock.Anything).Unset()
@@ -587,6 +620,7 @@ func TestGetIndexes_IfSystemStoreFails_ShouldCloseIterator(t *testing.T) {
func TestGetIndexes_IfSystemStoreQueryIteratorFails_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
testErr := errors.New("test error")
@@ -604,6 +638,7 @@ func TestGetIndexes_IfSystemStoreQueryIteratorFails_ReturnError(t *testing.T) {
func TestGetIndexes_IfSystemStoreHasInvalidData_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
mockedTxn := f.mockTxn()
@@ -619,6 +654,7 @@ func TestGetIndexes_IfSystemStoreHasInvalidData_ReturnError(t *testing.T) {
func TestGetCollectionIndexes_ShouldReturnListOfCollectionIndexes(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
usersIndexDesc := client.IndexDescription{
Name: "users_name_index",
@@ -627,8 +663,6 @@ func TestGetCollectionIndexes_ShouldReturnListOfCollectionIndexes(t *testing.T)
_, err := f.createCollectionIndexFor(usersColName, usersIndexDesc)
assert.NoError(t, err)
- f.commitTxn()
-
f.getProductsCollectionDesc()
productsIndexDesc := client.IndexDescription{
Name: "products_description_index",
@@ -656,6 +690,7 @@ func TestGetCollectionIndexes_ShouldReturnListOfCollectionIndexes(t *testing.T)
func TestGetCollectionIndexes_IfSystemStoreFails_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
testErr := errors.New("test error")
@@ -671,6 +706,7 @@ func TestGetCollectionIndexes_IfSystemStoreFails_ReturnError(t *testing.T) {
func TestGetCollectionIndexes_IfSystemStoreFails_ShouldCloseIterator(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
mockedTxn := f.mockTxn()
mockedTxn.MockSystemstore = mocks.NewDSReaderWriter(t)
@@ -685,6 +721,7 @@ func TestGetCollectionIndexes_IfSystemStoreFails_ShouldCloseIterator(t *testing.
func TestGetCollectionIndexes_IfSystemStoreQueryIteratorFails_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
testErr := errors.New("test error")
@@ -701,6 +738,7 @@ func TestGetCollectionIndexes_IfSystemStoreQueryIteratorFails_ReturnError(t *tes
func TestGetCollectionIndexes_IfInvalidIndexIsStored_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
indexKey := core.NewCollectionIndexKey(usersColName, "users_name_index")
err := f.txn.Systemstore().Put(f.ctx, indexKey.ToDS(), []byte("invalid"))
@@ -712,6 +750,7 @@ func TestGetCollectionIndexes_IfInvalidIndexIsStored_ReturnError(t *testing.T) {
func TestCollectionGetIndexes_ShouldReturnIndexes(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
@@ -724,6 +763,7 @@ func TestCollectionGetIndexes_ShouldReturnIndexes(t *testing.T) {
func TestCollectionGetIndexes_ShouldCloseQueryIterator(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
@@ -784,6 +824,7 @@ func TestCollectionGetIndexes_IfSystemStoreFails_ReturnError(t *testing.T) {
for _, testCase := range testCases {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
@@ -800,6 +841,7 @@ func TestCollectionGetIndexes_IfSystemStoreFails_ReturnError(t *testing.T) {
func TestCollectionGetIndexes_IfFailsToCreateTxn_ShouldNotCache(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
@@ -824,7 +866,7 @@ func TestCollectionGetIndexes_IfFailsToCreateTxn_ShouldNotCache(t *testing.T) {
func TestCollectionGetIndexes_IfStoredIndexWithUnsupportedType_ReturnError(t *testing.T) {
f := newIndexTestFixtureBare(t)
- f.getUsersCollectionDesc()
+ f.addUsersCollection()
const unsupportedKind = client.FieldKind_BOOL_ARRAY
_, err := f.db.AddSchema(
@@ -861,6 +903,7 @@ func TestCollectionGetIndexes_IfStoredIndexWithUnsupportedType_ReturnError(t *te
func TestCollectionGetIndexes_IfInvalidIndexIsStored_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
f.createUserCollectionIndexOnAge()
@@ -877,6 +920,7 @@ func TestCollectionGetIndexes_IfInvalidIndexIsStored_ReturnError(t *testing.T) {
func TestCollectionGetIndexes_IfIndexIsCreated_ReturnUpdateIndexes(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
@@ -894,6 +938,7 @@ func TestCollectionGetIndexes_IfIndexIsCreated_ReturnUpdateIndexes(t *testing.T)
func TestCollectionGetIndexes_IfIndexIsDropped_ReturnUpdateIndexes(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
f.createUserCollectionIndexOnAge()
@@ -969,7 +1014,6 @@ func TestCollectionGetIndexes_ShouldReturnIndexesInOrderedByName(t *testing.T) {
_, err := f.createCollectionIndexFor(collection.Name(), indexDesc)
require.NoError(t, err)
}
- f.commitTxn()
indexes, err := collection.GetIndexes(f.ctx)
require.NoError(t, err)
@@ -982,6 +1026,7 @@ func TestCollectionGetIndexes_ShouldReturnIndexesInOrderedByName(t *testing.T) {
func TestDropIndex_ShouldDeleteIndex(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
desc := f.createUserCollectionIndexOnName()
err := f.dropIndex(usersColName, desc.Name)
@@ -994,6 +1039,7 @@ func TestDropIndex_ShouldDeleteIndex(t *testing.T) {
func TestDropIndex_IfStorageFails_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
desc := f.createUserCollectionIndexOnName()
f.db.Close()
@@ -1003,6 +1049,7 @@ func TestDropIndex_IfStorageFails_ReturnError(t *testing.T) {
func TestDropIndex_IfCollectionDoesntExist_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
err := f.dropIndex(productsColName, "any_name")
assert.ErrorIs(t, err, NewErrCanNotReadCollection(usersColName, nil))
@@ -1010,12 +1057,15 @@ func TestDropIndex_IfCollectionDoesntExist_ReturnError(t *testing.T) {
func TestDropIndex_IfFailsToCreateTxn_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
testErr := errors.New("test error")
mockedRootStore := mocks.NewRootStore(t)
+ mockedRootStore.On("Close").Return(nil)
+
mockedRootStore.EXPECT().NewTransaction(mock.Anything, mock.Anything).Return(nil, testErr)
f.db.rootstore = mockedRootStore
@@ -1025,6 +1075,7 @@ func TestDropIndex_IfFailsToCreateTxn_ReturnError(t *testing.T) {
func TestDropIndex_IfFailsToDeleteFromStorage_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
@@ -1043,6 +1094,7 @@ func TestDropIndex_IfFailsToDeleteFromStorage_ReturnError(t *testing.T) {
func TestDropIndex_ShouldUpdateCollectionsDescription(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
col := f.users.WithTxn(f.txn)
_, err := col.CreateIndex(f.ctx, getUsersIndexDescOnName())
require.NoError(t, err)
@@ -1064,6 +1116,7 @@ func TestDropIndex_ShouldUpdateCollectionsDescription(t *testing.T) {
func TestDropIndex_IfIndexWithNameDoesNotExist_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
const name = "not_existing_index"
err := f.users.DropIndex(f.ctx, name)
@@ -1074,6 +1127,7 @@ func TestDropIndex_IfSystemStoreFails_ReturnError(t *testing.T) {
testErr := errors.New("test error")
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
@@ -1091,6 +1145,7 @@ func TestDropIndex_IfSystemStoreFails_ReturnError(t *testing.T) {
func TestDropAllIndexes_ShouldDeleteAllIndexes(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
_, err := f.createCollectionIndexFor(usersColName, client.IndexDescription{
Fields: []client.IndexedFieldDescription{
{Name: usersNameFieldName, Direction: client.Ascending},
@@ -1115,6 +1170,7 @@ func TestDropAllIndexes_ShouldDeleteAllIndexes(t *testing.T) {
func TestDropAllIndexes_IfStorageFails_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
f.db.Close()
@@ -1165,6 +1221,7 @@ func TestDropAllIndexes_IfSystemStorageFails_ReturnError(t *testing.T) {
for _, testCase := range testCases {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
mockedTxn := f.mockTxn()
@@ -1180,6 +1237,7 @@ func TestDropAllIndexes_IfSystemStorageFails_ReturnError(t *testing.T) {
func TestDropAllIndexes_ShouldCloseQueryIterator(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
mockedTxn := f.mockTxn()
@@ -1198,6 +1256,7 @@ func TestDropAllIndexes_ShouldCloseQueryIterator(t *testing.T) {
func TestNewCollectionIndex_IfDescriptionHasNoFields_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
desc := getUsersIndexDescOnName()
desc.Fields = nil
_, err := NewCollectionIndex(f.users, desc)
@@ -1206,6 +1265,7 @@ func TestNewCollectionIndex_IfDescriptionHasNoFields_ReturnError(t *testing.T) {
func TestNewCollectionIndex_IfDescriptionHasNonExistingField_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
desc := getUsersIndexDescOnName()
desc.Fields[0].Name = "non_existing_field"
_, err := NewCollectionIndex(f.users, desc)
diff --git a/db/indexed_docs_test.go b/db/indexed_docs_test.go
index 6503429c96..b7c7abbf9d 100644
--- a/db/indexed_docs_test.go
+++ b/db/indexed_docs_test.go
@@ -15,9 +15,7 @@ import (
"encoding/json"
"errors"
"fmt"
- "strconv"
"testing"
- "time"
ipfsDatastore "github.com/ipfs/go-datastore"
"github.com/ipfs/go-datastore/query"
@@ -50,26 +48,27 @@ type productDoc struct {
func (f *indexTestFixture) saveDocToCollection(doc *client.Document, col client.Collection) {
err := col.Create(f.ctx, doc)
require.NoError(f.t, err)
+ f.commitTxn()
f.txn, err = f.db.NewTxn(f.ctx, false)
require.NoError(f.t, err)
}
-func (f *indexTestFixture) newUserDoc(name string, age int) *client.Document {
+func (f *indexTestFixture) newUserDoc(name string, age int, col client.Collection) *client.Document {
d := userDoc{Name: name, Age: age, Weight: 154.1}
data, err := json.Marshal(d)
require.NoError(f.t, err)
- doc, err := client.NewDocFromJSON(data)
+ doc, err := client.NewDocFromJSON(data, col.Schema())
require.NoError(f.t, err)
return doc
}
-func (f *indexTestFixture) newProdDoc(id int, price float64, cat string) *client.Document {
+func (f *indexTestFixture) newProdDoc(id int, price float64, cat string, col client.Collection) *client.Document {
d := productDoc{ID: id, Price: price, Category: cat}
data, err := json.Marshal(d)
require.NoError(f.t, err)
- doc, err := client.NewDocFromJSON(data)
+ doc, err := client.NewDocFromJSON(data, col.Schema())
require.NoError(f.t, err)
return doc
}
@@ -160,20 +159,21 @@ func (b *indexKeyBuilder) Build() core.IndexDataStoreKey {
if b.doc != nil {
var fieldBytesVal []byte
- var writeableVal client.WriteableValue
+ var fieldValue *client.FieldValue
+ var err error
if len(b.values) == 0 {
- fieldVal, err := b.doc.GetValue(b.fieldName)
+ fieldValue, err = b.doc.GetValue(b.fieldName)
require.NoError(b.f.t, err)
- var ok bool
- writeableVal, ok = fieldVal.(client.WriteableValue)
- require.True(b.f.t, ok)
} else {
- writeableVal = client.NewCBORValue(client.LWW_REGISTER, b.values[0])
+ fieldValue = client.NewFieldValue(client.LWW_REGISTER, b.values[0])
}
- fieldBytesVal, err = writeableVal.Bytes()
+ fieldBytesVal, err = fieldValue.Bytes()
require.NoError(b.f.t, err)
- key.FieldValues = [][]byte{fieldBytesVal, []byte(b.doc.Key().String())}
+ key.FieldValues = [][]byte{fieldBytesVal}
+ if !b.isUnique {
+ key.FieldValues = append(key.FieldValues, []byte(b.doc.ID().String()))
+ }
} else if len(b.values) > 0 {
key.FieldValues = b.values
}
@@ -249,9 +249,10 @@ func (f *indexTestFixture) stubSystemStore(systemStoreOn *mocks.DSReaderWriter_E
func TestNonUnique_IfDocIsAdded_ShouldBeIndexed(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
- doc := f.newUserDoc("John", 21)
+ doc := f.newUserDoc("John", 21, f.users)
f.saveDocToCollection(doc, f.users)
key := newIndexKeyBuilder(f).Col(usersColName).Field(usersNameFieldName).Doc(doc).Build()
@@ -263,9 +264,10 @@ func TestNonUnique_IfDocIsAdded_ShouldBeIndexed(t *testing.T) {
func TestNonUnique_IfFailsToStoredIndexedDoc_Error(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
- doc := f.newUserDoc("John", 21)
+ doc := f.newUserDoc("John", 21, f.users)
key := newIndexKeyBuilder(f).Col(usersColName).Field(usersNameFieldName).Doc(doc).Build()
mockTxn := f.mockTxn()
@@ -281,6 +283,7 @@ func TestNonUnique_IfFailsToStoredIndexedDoc_Error(t *testing.T) {
func TestNonUnique_IfDocDoesNotHaveIndexedField_SkipIndex(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
data, err := json.Marshal(struct {
@@ -289,7 +292,7 @@ func TestNonUnique_IfDocDoesNotHaveIndexedField_SkipIndex(t *testing.T) {
}{Age: 21, Weight: 154.1})
require.NoError(f.t, err)
- doc, err := client.NewDocFromJSON(data)
+ doc, err := client.NewDocFromJSON(data, f.users.Schema())
require.NoError(f.t, err)
err = f.users.Create(f.ctx, doc)
@@ -302,9 +305,10 @@ func TestNonUnique_IfDocDoesNotHaveIndexedField_SkipIndex(t *testing.T) {
func TestNonUnique_IfSystemStorageHasInvalidIndexDescription_Error(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
- doc := f.newUserDoc("John", 21)
+ doc := f.newUserDoc("John", 21, f.users)
mockTxn := f.mockTxn().ClearSystemStore()
systemStoreOn := mockTxn.MockSystemstore.EXPECT()
@@ -317,9 +321,10 @@ func TestNonUnique_IfSystemStorageHasInvalidIndexDescription_Error(t *testing.T)
func TestNonUnique_IfSystemStorageFailsToReadIndexDesc_Error(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
- doc := f.newUserDoc("John", 21)
+ doc := f.newUserDoc("John", 21, f.users)
testErr := errors.New("test error")
@@ -334,9 +339,10 @@ func TestNonUnique_IfSystemStorageFailsToReadIndexDesc_Error(t *testing.T) {
func TestNonUnique_IfIndexIntField_StoreIt(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnAge()
- doc := f.newUserDoc("John", 21)
+ doc := f.newUserDoc("John", 21, f.users)
f.saveDocToCollection(doc, f.users)
key := newIndexKeyBuilder(f).Col(usersColName).Field(usersAgeFieldName).Doc(doc).Build()
@@ -348,7 +354,7 @@ func TestNonUnique_IfIndexIntField_StoreIt(t *testing.T) {
func TestNonUnique_IfMultipleCollectionsWithIndexes_StoreIndexWithCollectionID(t *testing.T) {
f := newIndexTestFixtureBare(t)
- users := f.getUsersCollectionDesc()
+ users := f.addUsersCollection()
products := f.getProductsCollectionDesc()
_, err := f.createCollectionIndexFor(users.Name(), getUsersIndexDescOnName())
@@ -357,8 +363,8 @@ func TestNonUnique_IfMultipleCollectionsWithIndexes_StoreIndexWithCollectionID(t
require.NoError(f.t, err)
f.commitTxn()
- userDoc := f.newUserDoc("John", 21)
- prodDoc := f.newProdDoc(1, 3, "games")
+ userDoc := f.newUserDoc("John", 21, users)
+ prodDoc := f.newProdDoc(1, 3, "games", products)
err = users.Create(f.ctx, userDoc)
require.NoError(f.t, err)
@@ -366,23 +372,24 @@ func TestNonUnique_IfMultipleCollectionsWithIndexes_StoreIndexWithCollectionID(t
require.NoError(f.t, err)
f.commitTxn()
- userDocKey := newIndexKeyBuilder(f).Col(usersColName).Field(usersNameFieldName).Doc(userDoc).Build()
- prodDocKey := newIndexKeyBuilder(f).Col(productsColName).Field(productsCategoryFieldName).Doc(prodDoc).Build()
+ userDocID := newIndexKeyBuilder(f).Col(usersColName).Field(usersNameFieldName).Doc(userDoc).Build()
+ prodDocID := newIndexKeyBuilder(f).Col(productsColName).Field(productsCategoryFieldName).Doc(prodDoc).Build()
- data, err := f.txn.Datastore().Get(f.ctx, userDocKey.ToDS())
+ data, err := f.txn.Datastore().Get(f.ctx, userDocID.ToDS())
require.NoError(t, err)
assert.Len(t, data, 0)
- data, err = f.txn.Datastore().Get(f.ctx, prodDocKey.ToDS())
+ data, err = f.txn.Datastore().Get(f.ctx, prodDocID.ToDS())
require.NoError(t, err)
assert.Len(t, data, 0)
}
func TestNonUnique_IfMultipleIndexes_StoreIndexWithIndexID(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
f.createUserCollectionIndexOnAge()
- doc := f.newUserDoc("John", 21)
+ doc := f.newUserDoc("John", 21, f.users)
f.saveDocToCollection(doc, f.users)
nameKey := newIndexKeyBuilder(f).Col(usersColName).Field(usersNameFieldName).Doc(doc).Build()
@@ -396,92 +403,96 @@ func TestNonUnique_IfMultipleIndexes_StoreIndexWithIndexID(t *testing.T) {
assert.Len(t, data, 0)
}
-func TestNonUnique_StoringIndexedFieldValueOfDifferentTypes(t *testing.T) {
- f := newIndexTestFixtureBare(t)
-
- now := time.Now()
- nowStr := now.Format(time.RFC3339)
-
- testCase := []struct {
- Name string
- FieldKind client.FieldKind
- // FieldVal is the value the index will receive for serialization
- FieldVal any
- ShouldFail bool
- }{
- {Name: "invalid int", FieldKind: client.FieldKind_INT, FieldVal: "invalid", ShouldFail: true},
- {Name: "invalid float", FieldKind: client.FieldKind_FLOAT, FieldVal: "invalid", ShouldFail: true},
- {Name: "invalid bool", FieldKind: client.FieldKind_BOOL, FieldVal: "invalid", ShouldFail: true},
- {Name: "invalid datetime", FieldKind: client.FieldKind_DATETIME, FieldVal: nowStr[1:], ShouldFail: true},
- {Name: "invalid datetime type", FieldKind: client.FieldKind_DATETIME, FieldVal: 1, ShouldFail: true},
-
- {Name: "valid int", FieldKind: client.FieldKind_INT, FieldVal: 12},
- {Name: "valid float", FieldKind: client.FieldKind_FLOAT, FieldVal: 36.654},
- {Name: "valid bool true", FieldKind: client.FieldKind_BOOL, FieldVal: true},
- {Name: "valid bool false", FieldKind: client.FieldKind_BOOL, FieldVal: false},
- {Name: "valid datetime string", FieldKind: client.FieldKind_DATETIME, FieldVal: nowStr},
- {Name: "valid empty string", FieldKind: client.FieldKind_STRING, FieldVal: ""},
- }
-
- for i, tc := range testCase {
- _, err := f.db.AddSchema(
- f.ctx,
- fmt.Sprintf(
- `type %s {
- field: %s
- }`,
- "testTypeCol"+strconv.Itoa(i),
- tc.FieldKind.String(),
- ),
- )
- require.NoError(f.t, err)
-
- collection, err := f.db.GetCollectionByName(f.ctx, "testTypeCol"+strconv.Itoa(i))
- require.NoError(f.t, err)
-
- f.txn, err = f.db.NewTxn(f.ctx, false)
- require.NoError(f.t, err)
-
- indexDesc := client.IndexDescription{
- Fields: []client.IndexedFieldDescription{
- {Name: "field", Direction: client.Ascending},
- },
- }
-
- _, err = f.createCollectionIndexFor(collection.Name(), indexDesc)
- require.NoError(f.t, err)
- f.commitTxn()
-
- d := struct {
- Field any `json:"field"`
- }{Field: tc.FieldVal}
- data, err := json.Marshal(d)
- require.NoError(f.t, err)
- doc, err := client.NewDocFromJSON(data)
- require.NoError(f.t, err)
-
- err = collection.Create(f.ctx, doc)
- f.commitTxn()
- if tc.ShouldFail {
- require.ErrorIs(f.t, err,
- NewErrInvalidFieldValue(tc.FieldKind, tc.FieldVal), "test case: %s", tc.Name)
- } else {
- assertMsg := fmt.Sprintf("test case: %s", tc.Name)
- require.NoError(f.t, err, assertMsg)
-
- keyBuilder := newIndexKeyBuilder(f).Col(collection.Name()).Field("field").Doc(doc)
- key := keyBuilder.Build()
-
- keyStr := key.ToDS()
- data, err := f.txn.Datastore().Get(f.ctx, keyStr)
- require.NoError(t, err, assertMsg)
- assert.Len(t, data, 0, assertMsg)
- }
- }
-}
+// func TestNonUnique_StoringIndexedFieldValueOfDifferentTypes(t *testing.T) {
+// f := newIndexTestFixtureBare(t)
+
+// now := time.Now()
+// nowStr := now.Format(time.RFC3339)
+
+// testCase := []struct {
+// Name string
+// FieldKind client.FieldKind
+// // FieldVal is the value the index will receive for serialization
+// FieldVal any
+// ShouldFail bool
+// }{
+// {Name: "invalid int", FieldKind: client.FieldKind_INT, FieldVal: "invalid", ShouldFail: true},
+// {Name: "invalid float", FieldKind: client.FieldKind_FLOAT, FieldVal: "invalid", ShouldFail: true},
+// {Name: "invalid bool", FieldKind: client.FieldKind_BOOL, FieldVal: "invalid", ShouldFail: true},
+// {Name: "invalid datetime", FieldKind: client.FieldKind_DATETIME, FieldVal: nowStr[1:], ShouldFail: true},
+// {Name: "invalid datetime type", FieldKind: client.FieldKind_DATETIME, FieldVal: 1, ShouldFail: true},
+// {Name: "invalid blob", FieldKind: client.FieldKind_BLOB, FieldVal: "invalid", ShouldFail: true},
+// {Name: "invalid blob type", FieldKind: client.FieldKind_BLOB, FieldVal: 1, ShouldFail: true},
+
+// {Name: "valid int", FieldKind: client.FieldKind_INT, FieldVal: 12},
+// {Name: "valid float", FieldKind: client.FieldKind_FLOAT, FieldVal: 36.654},
+// {Name: "valid bool true", FieldKind: client.FieldKind_BOOL, FieldVal: true},
+// {Name: "valid bool false", FieldKind: client.FieldKind_BOOL, FieldVal: false},
+// {Name: "valid datetime string", FieldKind: client.FieldKind_DATETIME, FieldVal: nowStr},
+// {Name: "valid empty string", FieldKind: client.FieldKind_STRING, FieldVal: ""},
+// {Name: "valid blob type", FieldKind: client.FieldKind_BLOB, FieldVal: "00ff"},
+// }
+
+// for i, tc := range testCase {
+// _, err := f.db.AddSchema(
+// f.ctx,
+// fmt.Sprintf(
+// `type %s {
+// field: %s
+// }`,
+// "testTypeCol"+strconv.Itoa(i),
+// tc.FieldKind.String(),
+// ),
+// )
+// require.NoError(f.t, err)
+
+// collection, err := f.db.GetCollectionByName(f.ctx, "testTypeCol"+strconv.Itoa(i))
+// require.NoError(f.t, err)
+
+// f.txn, err = f.db.NewTxn(f.ctx, false)
+// require.NoError(f.t, err)
+
+// indexDesc := client.IndexDescription{
+// Fields: []client.IndexedFieldDescription{
+// {Name: "field", Direction: client.Ascending},
+// },
+// }
+
+// _, err = f.createCollectionIndexFor(collection.Name(), indexDesc)
+// require.NoError(f.t, err)
+// f.commitTxn()
+
+// d := struct {
+// Field any `json:"field"`
+// }{Field: tc.FieldVal}
+// data, err := json.Marshal(d)
+// require.NoError(f.t, err)
+// doc, err := client.NewDocFromJSON(data, collection.Schema())
+// require.NoError(f.t, err)
+
+// err = collection.Create(f.ctx, doc)
+// f.commitTxn()
+// if tc.ShouldFail {
+// require.ErrorIs(f.t, err,
+// NewErrInvalidFieldValue(tc.FieldKind, tc.FieldVal), "test case: %s", tc.Name)
+// } else {
+// assertMsg := fmt.Sprintf("test case: %s", tc.Name)
+// require.NoError(f.t, err, assertMsg)
+
+// keyBuilder := newIndexKeyBuilder(f).Col(collection.Name()).Field("field").Doc(doc)
+// key := keyBuilder.Build()
+
+// keyStr := key.ToDS()
+// data, err := f.txn.Datastore().Get(f.ctx, keyStr)
+// require.NoError(t, err, assertMsg)
+// assert.Len(t, data, 0, assertMsg)
+// }
+// }
+// }
func TestNonUnique_IfIndexedFieldIsNil_StoreItAsNil(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
docJSON, err := json.Marshal(struct {
@@ -489,7 +500,7 @@ func TestNonUnique_IfIndexedFieldIsNil_StoreItAsNil(t *testing.T) {
}{Age: 44})
require.NoError(f.t, err)
- doc, err := client.NewDocFromJSON(docJSON)
+ doc, err := client.NewDocFromJSON(docJSON, f.users.Schema())
require.NoError(f.t, err)
f.saveDocToCollection(doc, f.users)
@@ -504,10 +515,11 @@ func TestNonUnique_IfIndexedFieldIsNil_StoreItAsNil(t *testing.T) {
func TestNonUniqueCreate_ShouldIndexExistingDocs(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
- doc1 := f.newUserDoc("John", 21)
+ doc1 := f.newUserDoc("John", 21, f.users)
f.saveDocToCollection(doc1, f.users)
- doc2 := f.newUserDoc("Islam", 18)
+ doc2 := f.newUserDoc("Islam", 18, f.users)
f.saveDocToCollection(doc2, f.users)
f.createUserCollectionIndexOnName()
@@ -578,8 +590,9 @@ func TestNonUniqueCreate_IfUponIndexingExistingDocsFetcherFails_ReturnError(t *t
for _, tc := range cases {
f := newIndexTestFixture(t)
+ defer f.db.Close()
- doc := f.newUserDoc("John", 21)
+ doc := f.newUserDoc("John", 21, f.users)
f.saveDocToCollection(doc, f.users)
f.users.(*collection).fetcherFactory = tc.PrepareFetcher
@@ -595,13 +608,14 @@ func TestNonUniqueCreate_IfUponIndexingExistingDocsFetcherFails_ReturnError(t *t
func TestNonUniqueCreate_IfDatastoreFailsToStoreIndex_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
- doc := f.newUserDoc("John", 21)
+ doc := f.newUserDoc("John", 21, f.users)
f.saveDocToCollection(doc, f.users)
fieldKeyString := core.DataStoreKey{
CollectionID: f.users.Description().IDString(),
- }.WithDocKey(doc.Key().String()).
+ }.WithDocID(doc.ID().String()).
WithFieldId("1").
WithValueFlag().
ToString()
@@ -609,15 +623,16 @@ func TestNonUniqueCreate_IfDatastoreFailsToStoreIndex_ReturnError(t *testing.T)
invalidKeyString := fieldKeyString + "/doesn't matter/"
// Insert an invalid key within the document prefix, this will generate an error within the fetcher.
- f.db.multistore.Datastore().Put(f.ctx, ipfsDatastore.NewKey(invalidKeyString), []byte("doesn't matter"))
+ err := f.db.multistore.Datastore().Put(f.ctx, ipfsDatastore.NewKey(invalidKeyString), []byte("doesn't matter"))
+ require.NoError(f.t, err)
- _, err := f.users.CreateIndex(f.ctx, getUsersIndexDescOnName())
+ _, err = f.users.CreateIndex(f.ctx, getUsersIndexDescOnName())
require.ErrorIs(f.t, err, core.ErrInvalidKey)
}
func TestNonUniqueDrop_ShouldDeleteStoredIndexedFields(t *testing.T) {
f := newIndexTestFixtureBare(t)
- users := f.getUsersCollectionDesc()
+ users := f.addUsersCollection()
_, err := f.createCollectionIndexFor(users.Name(), getUsersIndexDescOnName())
require.NoError(f.t, err)
_, err = f.createCollectionIndexFor(users.Name(), getUsersIndexDescOnAge())
@@ -626,15 +641,15 @@ func TestNonUniqueDrop_ShouldDeleteStoredIndexedFields(t *testing.T) {
require.NoError(f.t, err)
f.commitTxn()
- f.saveDocToCollection(f.newUserDoc("John", 21), users)
- f.saveDocToCollection(f.newUserDoc("Islam", 23), users)
+ f.saveDocToCollection(f.newUserDoc("John", 21, users), users)
+ f.saveDocToCollection(f.newUserDoc("Islam", 23, users), users)
products := f.getProductsCollectionDesc()
_, err = f.createCollectionIndexFor(products.Name(), getProductsIndexDescOnCategory())
require.NoError(f.t, err)
f.commitTxn()
- f.saveDocToCollection(f.newProdDoc(1, 55, "games"), products)
+ f.saveDocToCollection(f.newProdDoc(1, 55, "games", products), products)
userNameKey := newIndexKeyBuilder(f).Col(usersColName).Field(usersNameFieldName).Build()
userAgeKey := newIndexKeyBuilder(f).Col(usersColName).Field(usersAgeFieldName).Build()
@@ -652,6 +667,7 @@ func TestNonUniqueDrop_ShouldDeleteStoredIndexedFields(t *testing.T) {
func TestNonUniqueUpdate_ShouldDeleteOldValueAndStoreNewOne(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
cases := []struct {
@@ -675,7 +691,7 @@ func TestNonUniqueUpdate_ShouldDeleteOldValueAndStoreNewOne(t *testing.T) {
},
}
- doc := f.newUserDoc("John", 21)
+ doc := f.newUserDoc("John", 21, f.users)
f.saveDocToCollection(doc, f.users)
for _, tc := range cases {
@@ -698,9 +714,10 @@ func TestNonUniqueUpdate_ShouldDeleteOldValueAndStoreNewOne(t *testing.T) {
func TestNonUniqueUpdate_IfFailsToReadIndexDescription_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
- doc := f.newUserDoc("John", 21)
+ doc := f.newUserDoc("John", 21, f.users)
f.saveDocToCollection(doc, f.users)
err := doc.Set(usersNameFieldName, "Islam")
@@ -786,9 +803,10 @@ func TestNonUniqueUpdate_IfFetcherFails_ReturnError(t *testing.T) {
t.Log(tc.Name)
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
- doc := f.newUserDoc("John", 21)
+ doc := f.newUserDoc("John", 21, f.users)
f.saveDocToCollection(doc, f.users)
f.users.(*collection).fetcherFactory = tc.PrepareFetcher
@@ -810,9 +828,10 @@ func TestNonUniqueUpdate_IfFetcherFails_ReturnError(t *testing.T) {
func TestNonUniqueUpdate_IfFailsToUpdateIndex_ReturnError(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnAge()
- doc := f.newUserDoc("John", 21)
+ doc := f.newUserDoc("John", 21, f.users)
f.saveDocToCollection(doc, f.users)
f.commitTxn()
@@ -829,6 +848,7 @@ func TestNonUniqueUpdate_IfFailsToUpdateIndex_ReturnError(t *testing.T) {
func TestNonUniqueUpdate_ShouldPassToFetcherOnlyRelevantFields(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
f.createUserCollectionIndexOnAge()
@@ -853,7 +873,7 @@ func TestNonUniqueUpdate_ShouldPassToFetcherOnlyRelevantFields(t *testing.T) {
})
return f
}
- doc := f.newUserDoc("John", 21)
+ doc := f.newUserDoc("John", 21, f.users)
f.saveDocToCollection(doc, f.users)
err := doc.Set(usersNameFieldName, "Islam")
@@ -891,14 +911,15 @@ func TestNonUniqueUpdate_IfDatastoreFails_ReturnError(t *testing.T) {
t.Log(tc.Name)
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
- doc := f.newUserDoc("John", 21)
+ doc := f.newUserDoc("John", 21, f.users)
err := doc.Set(usersNameFieldName, "Islam")
require.NoError(t, err)
encodedDoc := shimEncodedDocument{
- key: []byte(doc.Key().String()),
+ key: []byte(doc.ID().String()),
schemaVersionID: f.users.Schema().VersionID,
}
@@ -922,6 +943,7 @@ func TestNonUniqueUpdate_IfDatastoreFails_ReturnError(t *testing.T) {
func TestNonUpdate_IfIndexedFieldWasNil_ShouldDeleteIt(t *testing.T) {
f := newIndexTestFixture(t)
+ defer f.db.Close()
f.createUserCollectionIndexOnName()
docJSON, err := json.Marshal(struct {
@@ -929,7 +951,7 @@ func TestNonUpdate_IfIndexedFieldWasNil_ShouldDeleteIt(t *testing.T) {
}{Age: 44})
require.NoError(f.t, err)
- doc, err := client.NewDocFromJSON(docJSON)
+ doc, err := client.NewDocFromJSON(docJSON, f.users.Schema())
require.NoError(f.t, err)
f.saveDocToCollection(doc, f.users)
@@ -961,7 +983,7 @@ type shimEncodedDocument struct {
var _ fetcher.EncodedDocument = (*shimEncodedDocument)(nil)
-func (encdoc *shimEncodedDocument) Key() []byte {
+func (encdoc *shimEncodedDocument) ID() []byte {
return encdoc.key
}
@@ -983,3 +1005,117 @@ func (encdoc *shimEncodedDocument) Reset() {
encdoc.status = 0
encdoc.properties = map[client.FieldDescription]any{}
}
+
+func TestUniqueCreate_ShouldIndexExistingDocs(t *testing.T) {
+ f := newIndexTestFixture(t)
+ defer f.db.Close()
+
+ doc1 := f.newUserDoc("John", 21, f.users)
+ f.saveDocToCollection(doc1, f.users)
+ doc2 := f.newUserDoc("Islam", 18, f.users)
+ f.saveDocToCollection(doc2, f.users)
+
+ f.createUserCollectionUniqueIndexOnName()
+
+ key1 := newIndexKeyBuilder(f).Col(usersColName).Field(usersNameFieldName).Unique().Doc(doc1).Build()
+ key2 := newIndexKeyBuilder(f).Col(usersColName).Field(usersNameFieldName).Unique().Doc(doc2).Build()
+
+ data, err := f.txn.Datastore().Get(f.ctx, key1.ToDS())
+ require.NoError(t, err, key1.ToString())
+ assert.Equal(t, data, []byte(doc1.ID().String()))
+ data, err = f.txn.Datastore().Get(f.ctx, key2.ToDS())
+ require.NoError(t, err)
+ assert.Equal(t, data, []byte(doc2.ID().String()))
+}
+
+func TestUnique_IfIndexedFieldIsNil_StoreItAsNil(t *testing.T) {
+ f := newIndexTestFixture(t)
+ defer f.db.Close()
+ f.createUserCollectionUniqueIndexOnName()
+
+ docJSON, err := json.Marshal(struct {
+ Age int `json:"age"`
+ }{Age: 44})
+ require.NoError(f.t, err)
+
+ doc, err := client.NewDocFromJSON(docJSON, f.users.Schema())
+ require.NoError(f.t, err)
+
+ f.saveDocToCollection(doc, f.users)
+
+ key := newIndexKeyBuilder(f).Col(usersColName).Field(usersNameFieldName).Unique().Doc(doc).
+ Values([]byte(nil)).Build()
+
+ data, err := f.txn.Datastore().Get(f.ctx, key.ToDS())
+ require.NoError(t, err)
+ assert.Equal(t, data, []byte(doc.ID().String()))
+}
+
+func TestUniqueDrop_ShouldDeleteStoredIndexedFields(t *testing.T) {
+ f := newIndexTestFixtureBare(t)
+ users := f.addUsersCollection()
+ _, err := f.createCollectionIndexFor(users.Name(), makeUnique(getUsersIndexDescOnName()))
+ require.NoError(f.t, err)
+ _, err = f.createCollectionIndexFor(users.Name(), makeUnique(getUsersIndexDescOnAge()))
+ require.NoError(f.t, err)
+ f.commitTxn()
+
+ f.saveDocToCollection(f.newUserDoc("John", 21, users), users)
+ f.saveDocToCollection(f.newUserDoc("Islam", 23, users), users)
+
+ userNameKey := newIndexKeyBuilder(f).Col(usersColName).Field(usersNameFieldName).Build()
+ userAgeKey := newIndexKeyBuilder(f).Col(usersColName).Field(usersAgeFieldName).Build()
+
+ err = f.dropIndex(usersColName, testUsersColIndexAge)
+ require.NoError(f.t, err)
+
+ assert.Len(t, f.getPrefixFromDataStore(userNameKey.ToString()), 2)
+ assert.Len(t, f.getPrefixFromDataStore(userAgeKey.ToString()), 0)
+}
+
+func TestUniqueUpdate_ShouldDeleteOldValueAndStoreNewOne(t *testing.T) {
+ f := newIndexTestFixture(t)
+ defer f.db.Close()
+ f.createUserCollectionUniqueIndexOnName()
+
+ cases := []struct {
+ Name string
+ NewValue string
+ Exec func(doc *client.Document) error
+ }{
+ {
+ Name: "update",
+ NewValue: "Islam",
+ Exec: func(doc *client.Document) error {
+ return f.users.Update(f.ctx, doc)
+ },
+ },
+ {
+ Name: "save",
+ NewValue: "Andy",
+ Exec: func(doc *client.Document) error {
+ return f.users.Save(f.ctx, doc)
+ },
+ },
+ }
+
+ doc := f.newUserDoc("John", 21, f.users)
+ f.saveDocToCollection(doc, f.users)
+
+ for _, tc := range cases {
+ oldKey := newIndexKeyBuilder(f).Col(usersColName).Field(usersNameFieldName).Unique().Doc(doc).Build()
+
+ err := doc.Set(usersNameFieldName, tc.NewValue)
+ require.NoError(t, err)
+ err = tc.Exec(doc)
+ require.NoError(t, err)
+ f.commitTxn()
+
+ newKey := newIndexKeyBuilder(f).Col(usersColName).Field(usersNameFieldName).Unique().Doc(doc).Build()
+
+ _, err = f.txn.Datastore().Get(f.ctx, oldKey.ToDS())
+ require.Error(t, err)
+ _, err = f.txn.Datastore().Get(f.ctx, newKey.ToDS())
+ require.NoError(t, err)
+ }
+}
diff --git a/db/schema.go b/db/schema.go
index df95df60e2..988aea5e17 100644
--- a/db/schema.go
+++ b/db/schema.go
@@ -54,11 +54,6 @@ func (db *db) addSchema(
return nil, err
}
- err = db.parser.SetSchema(ctx, txn, append(existingDefinitions, newDefinitions...))
- if err != nil {
- return nil, err
- }
-
returnDescriptions := make([]client.CollectionDescription, len(newDefinitions))
for i, definition := range newDefinitions {
col, err := db.createCollection(ctx, txn, definition)
@@ -68,20 +63,20 @@ func (db *db) addSchema(
returnDescriptions[i] = col.Description()
}
+ err = db.loadSchema(ctx, txn)
+ if err != nil {
+ return nil, err
+ }
+
return returnDescriptions, nil
}
func (db *db) loadSchema(ctx context.Context, txn datastore.Txn) error {
- collections, err := db.getAllCollections(ctx, txn)
+ definitions, err := db.getAllActiveDefinitions(ctx, txn)
if err != nil {
return err
}
- definitions := make([]client.CollectionDefinition, len(collections))
- for i := range collections {
- definitions[i] = collections[i].Definition()
- }
-
return db.parser.SetSchema(ctx, txn, definitions)
}
@@ -150,17 +145,7 @@ func (db *db) patchSchema(ctx context.Context, txn datastore.Txn, patchString st
}
}
- newCollections, err := db.getAllCollections(ctx, txn)
- if err != nil {
- return err
- }
-
- definitions := make([]client.CollectionDefinition, len(newCollections))
- for i, col := range newCollections {
- definitions[i] = col.Definition()
- }
-
- return db.parser.SetSchema(ctx, txn, definitions)
+ return db.loadSchema(ctx, txn)
}
// substituteSchemaPatch handles any substitution of values that may be required before
diff --git a/db/subscriptions.go b/db/subscriptions.go
index af981ad95f..2e7d2d4123 100644
--- a/db/subscriptions.go
+++ b/db/subscriptions.go
@@ -74,7 +74,7 @@ func (db *db) handleEvent(
) {
p := planner.New(ctx, db.WithTxn(txn), txn)
- s := r.ToSelect(evt.DocKey, evt.Cid.String())
+ s := r.ToSelect(evt.DocID, evt.Cid.String())
result, err := p.RunSubscriptionRequest(ctx, s)
if err != nil {
diff --git a/db/txn_db.go b/db/txn_db.go
index 380cfeed34..92f9cde6c1 100644
--- a/db/txn_db.go
+++ b/db/txn_db.go
@@ -378,6 +378,30 @@ func (db *explicitTxnDB) SetMigration(ctx context.Context, cfg client.LensConfig
return db.lensRegistry.SetMigration(ctx, cfg)
}
+func (db *implicitTxnDB) AddView(ctx context.Context, query string, sdl string) ([]client.CollectionDefinition, error) {
+ txn, err := db.NewTxn(ctx, false)
+ if err != nil {
+ return nil, err
+ }
+ defer txn.Discard(ctx)
+
+ defs, err := db.addView(ctx, txn, query, sdl)
+ if err != nil {
+ return nil, err
+ }
+
+ err = txn.Commit(ctx)
+ if err != nil {
+ return nil, err
+ }
+
+ return defs, nil
+}
+
+func (db *explicitTxnDB) AddView(ctx context.Context, query string, sdl string) ([]client.CollectionDefinition, error) {
+ return db.addView(ctx, db.txn, query, sdl)
+}
+
// BasicImport imports a json dataset.
// filepath must be accessible to the node.
func (db *implicitTxnDB) BasicImport(ctx context.Context, filepath string) error {
diff --git a/db/view.go b/db/view.go
new file mode 100644
index 0000000000..2b4666df22
--- /dev/null
+++ b/db/view.go
@@ -0,0 +1,89 @@
+// Copyright 2023 Democratized Data Foundation
+//
+// Use of this software is governed by the Business Source License
+// included in the file licenses/BSL.txt.
+//
+// As of the Change Date specified in that file, in accordance with
+// the Business Source License, use of this software will be governed
+// by the Apache License, Version 2.0, included in the file
+// licenses/APL.txt.
+
+package db
+
+import (
+ "context"
+ "errors"
+ "fmt"
+
+ "github.com/sourcenetwork/defradb/client"
+ "github.com/sourcenetwork/defradb/client/request"
+ "github.com/sourcenetwork/defradb/datastore"
+ "github.com/sourcenetwork/defradb/db/description"
+)
+
+func (db *db) addView(
+ ctx context.Context,
+ txn datastore.Txn,
+ inputQuery string,
+ sdl string,
+) ([]client.CollectionDefinition, error) {
+ // Wrap the given query as part of the GQL query object - this simplifies the syntax for users
+ // and ensures that we can't be given mutations. In the future this line should disappear along
+ // with the all calls to the parser appart from `ParseSDL` when we implement the DQL stuff.
+ query := fmt.Sprintf(`query { %s }`, inputQuery)
+
+ newDefinitions, err := db.parser.ParseSDL(ctx, sdl)
+ if err != nil {
+ return nil, err
+ }
+
+ ast, err := db.parser.BuildRequestAST(query)
+ if err != nil {
+ return nil, err
+ }
+
+ req, errs := db.parser.Parse(ast)
+ if len(errs) > 0 {
+ return nil, errors.Join(errs...)
+ }
+
+ if len(req.Queries) == 0 || len(req.Queries[0].Selections) == 0 {
+ return nil, NewErrInvalidViewQueryMissingQuery()
+ }
+
+ baseQuery, ok := req.Queries[0].Selections[0].(*request.Select)
+ if !ok {
+ return nil, NewErrInvalidViewQueryCastFailed(inputQuery)
+ }
+
+ for i := range newDefinitions {
+ newDefinitions[i].Description.BaseQuery = baseQuery
+ }
+
+ returnDescriptions := make([]client.CollectionDefinition, len(newDefinitions))
+ for i, definition := range newDefinitions {
+ if definition.Description.Name == "" {
+ schema, err := description.CreateSchemaVersion(ctx, txn, definition.Schema)
+ if err != nil {
+ return nil, err
+ }
+ returnDescriptions[i] = client.CollectionDefinition{
+ // `Collection` is left as default for embedded types
+ Schema: schema,
+ }
+ } else {
+ col, err := db.createCollection(ctx, txn, definition)
+ if err != nil {
+ return nil, err
+ }
+ returnDescriptions[i] = col.Definition()
+ }
+ }
+
+ err = db.loadSchema(ctx, txn)
+ if err != nil {
+ return nil, err
+ }
+
+ return returnDescriptions, nil
+}
diff --git a/docs/cli/defradb_client.md b/docs/cli/defradb_client.md
index a52fce09f3..c3bd354bc7 100644
--- a/docs/cli/defradb_client.md
+++ b/docs/cli/defradb_client.md
@@ -38,4 +38,5 @@ Execute queries, add schema types, obtain node info, etc.
* [defradb client query](defradb_client_query.md) - Send a DefraDB GraphQL query request
* [defradb client schema](defradb_client_schema.md) - Interact with the schema system of a DefraDB node
* [defradb client tx](defradb_client_tx.md) - Create, commit, and discard DefraDB transactions
+* [defradb client view](defradb_client_view.md) - Manage views within a running DefraDB instance
diff --git a/docs/cli/defradb_client_collection.md b/docs/cli/defradb_client_collection.md
index 7807f49503..d164902b97 100644
--- a/docs/cli/defradb_client_collection.md
+++ b/docs/cli/defradb_client_collection.md
@@ -33,9 +33,9 @@ Create, read, update, and delete documents within a collection.
* [defradb client](defradb_client.md) - Interact with a DefraDB node
* [defradb client collection create](defradb_client_collection_create.md) - Create a new document.
-* [defradb client collection delete](defradb_client_collection_delete.md) - Delete documents by key or filter.
+* [defradb client collection delete](defradb_client_collection_delete.md) - Delete documents by docID or filter.
* [defradb client collection describe](defradb_client_collection_describe.md) - View collection description.
+* [defradb client collection docIDs](defradb_client_collection_docIDs.md) - List all document IDs (docIDs).
* [defradb client collection get](defradb_client_collection_get.md) - View document fields.
-* [defradb client collection keys](defradb_client_collection_keys.md) - List all document keys.
-* [defradb client collection update](defradb_client_collection_update.md) - Update documents by key or filter.
+* [defradb client collection update](defradb_client_collection_update.md) - Update documents by docID or filter.
diff --git a/docs/cli/defradb_client_collection_delete.md b/docs/cli/defradb_client_collection_delete.md
index fea6c6ccc7..30676654d5 100644
--- a/docs/cli/defradb_client_collection_delete.md
+++ b/docs/cli/defradb_client_collection_delete.md
@@ -1,28 +1,28 @@
## defradb client collection delete
-Delete documents by key or filter.
+Delete documents by docID or filter.
### Synopsis
-Delete documents by key or filter and lists the number of documents deleted.
+Delete documents by docID or filter and lists the number of documents deleted.
-Example: delete by key(s)
- defradb client collection delete --name User --key bae-123,bae-456
+Example: delete by docID(s)
+ defradb client collection delete --name User --docID bae-123,bae-456
Example: delete by filter
defradb client collection delete --name User --filter '{ "_gte": { "points": 100 } }'
```
-defradb client collection delete [--filter --key ] [flags]
+defradb client collection delete [--filter --docID ] [flags]
```
### Options
```
+ --docID strings Document ID
--filter string Document filter
-h, --help help for delete
- --key strings Document key
```
### Options inherited from parent commands
diff --git a/docs/cli/defradb_client_collection_keys.md b/docs/cli/defradb_client_collection_docIDs.md
similarity index 83%
rename from docs/cli/defradb_client_collection_keys.md
rename to docs/cli/defradb_client_collection_docIDs.md
index 234d8e051f..5ca8444e2e 100644
--- a/docs/cli/defradb_client_collection_keys.md
+++ b/docs/cli/defradb_client_collection_docIDs.md
@@ -1,23 +1,23 @@
-## defradb client collection keys
+## defradb client collection docIDs
-List all document keys.
+List all document IDs (docIDs).
### Synopsis
-List all document keys.
+List all document IDs (docIDs).
Example:
- defradb client collection keys --name User
+ defradb client collection docIDs --name User
```
-defradb client collection keys [flags]
+defradb client collection docIDs [flags]
```
### Options
```
- -h, --help help for keys
+ -h, --help help for docIDs
```
### Options inherited from parent commands
diff --git a/docs/cli/defradb_client_collection_get.md b/docs/cli/defradb_client_collection_get.md
index 675988c487..3f60490272 100644
--- a/docs/cli/defradb_client_collection_get.md
+++ b/docs/cli/defradb_client_collection_get.md
@@ -11,7 +11,7 @@ Example:
```
-defradb client collection get [--show-deleted] [flags]
+defradb client collection get [--show-deleted] [flags]
```
### Options
diff --git a/docs/cli/defradb_client_collection_update.md b/docs/cli/defradb_client_collection_update.md
index c081614cce..4ba111f025 100644
--- a/docs/cli/defradb_client_collection_update.md
+++ b/docs/cli/defradb_client_collection_update.md
@@ -1,33 +1,33 @@
## defradb client collection update
-Update documents by key or filter.
+Update documents by docID or filter.
### Synopsis
-Update documents by key or filter.
+Update documents by docID or filter.
Example: update from string
- defradb client collection update --name User --key bae-123 '{ "name": "Bob" }'
+ defradb client collection update --name User --docID bae-123 '{ "name": "Bob" }'
Example: update by filter
defradb client collection update --name User \
--filter '{ "_gte": { "points": 100 } }' --updater '{ "verified": true }'
-Example: update by keys
+Example: update by docIDs
defradb client collection update --name User \
- --key bae-123,bae-456 --updater '{ "verified": true }'
+ --docID bae-123,bae-456 --updater '{ "verified": true }'
```
-defradb client collection update [--filter --key --updater ] [flags]
+defradb client collection update [--filter --docID --updater ] [flags]
```
### Options
```
+ --docID strings Document ID
--filter string Document filter
-h, --help help for update
- --key strings Document key
--updater string Document updater
```
diff --git a/docs/cli/defradb_client_document.md b/docs/cli/defradb_client_document.md
deleted file mode 100644
index bc527357e7..0000000000
--- a/docs/cli/defradb_client_document.md
+++ /dev/null
@@ -1,38 +0,0 @@
-## defradb client document
-
-Create, read, update, and delete documents.
-
-### Synopsis
-
-Create, read, update, and delete documents.
-
-### Options
-
-```
- -h, --help help for document
-```
-
-### Options inherited from parent commands
-
-```
- --logformat string Log format to use. Options are csv, json (default "csv")
- --logger stringArray Override logger parameters. Usage: --logger ,level=,output=