dev-unstable-12-05-2023

THIS IS A UNSTABLE DEV BUILD. DO NOT USE UNLESS SPECIFICALLY DIRECTED TO.

Built using commit: 8aa317a
Built on date: 12-05-2023.

Unstable pre-release of upcoming 0.5.1

v0.5.0 Release

🎉 Introducing DefraDB v0.5: Open Source Release 🎉

We are excited to announce the release of DefraDB v0.5, a major pre-production milestone for our user-centric database that prioritizes data ownership, personal privacy, and information security. This release marks the first open-source release for DefraDB and includes significant quality of life improvements, bug fixes, and overall cleanliness of the repo, making it ready for public testing and use.

Key Features:

• First iteration of our schema update system, enabling developers to add new fields to schemas using our JSON Patch based DDL
• New DAG based delete system that persists "soft-delete" ops into the CRDT Merkle DAG
• Early prototype for collection-level peer-to-peer synchronization

Important Notice:

This release includes a Breaking Change for existing v0.4.x databases. If you need assistance migrating an existing deployment, please contact us at [email protected] or join our Discord at https://discord.source.network/.

Explore the Release:

For a complete outline of the changes and updates, check out the official changelog above. We encourage you to try DefraDB v0.5 and provide feedback as we continue to develop this innovative solution for decentralized data and access-control management, user-centric apps, data trustworthiness, and more. Your input is invaluable in helping us shape the future of DefraDB and the Source technologies!

Get started with DefraDB v0.5 now!

You can find the full changelog below:

Features

• Add document delete mechanics (#1263)
• Ability to explain an executed request (#1188)
• Add SchemaPatch CLI command (#1250)
• Add support for one-one mutation from sec. side (#1247)
• Store only key in DAG instead of dockey path (#1245)
• Add collectionId field to commit field (#1235)
• Add field kind substitution for PatchSchema (#1223)
• Add dockey field for commit field (#1216)
• Allow new fields to be added locally to schema (#1139)
• Add like sub-string filter (#1091)
• Add ability for P2P to wait for pushlog by peer (#1098)
• Add P2P collection topic subscription (#1086)
• Add support for schema version id in queries (#1067)
• Add schema version id to commit queries (#1061)
• Persist schema version at time of commit (#1055)
• Add ability to input simple explain type arg (#1039)

Fixes

• API address parameter validation (#1311)
• Improve error message for NonNull GQL types (#1333)
• Handle panics in the rpc server (#1330)
• Handle returned error in select.go (#1329)
• Resolve handful of CLI issues (#1318)
• Only check for events queue on subscription request (#1326)
• Remove client Create/UpdateCollection (#1309)
• CLI to display specific command usage help (#1314)
• Fix P2P collection CLI commands (#1295)
• Dont double up badger file path (#1299)
• Update immutable package (#1290)
• Fix panic on success of Add/RemoveP2PCollections (#1297)
• Fix deadlock on memory-datastore Close (#1273)
• Determine if query is introspection query (#1255)
• Allow newly added fields to sync via p2p (#1226)
• Expose ExplainEnum in the GQL schema (#1204)
• Resolve aggregates' mapping with deep nested subtypes (#1175)
• Make sort stable and handle nil comparison (#1094)
• Change successful schema add status to 200 (#1106)
• Add delay in P2P test util execution (#1093)
• Ensure errors test don't hard expect folder name (#1072)
• Remove potential P2P deadlock (#1056)
• Rework the P2P integration tests (#989)
• Improve DAG sync with highly concurrent updates (#1031)

Documentation

• Update docs for the v0.5 release (#1320)
• Document client interfaces in client/db.go (#1305)
• Document client Description types (#1307)
• Improve security policy (#1240)
• Add security disclosure policy (#1194)
• Correct commits query example in readme (#1172)

Refactoring

• Improve p2p collection operations on peer (#1286)
• Migrate gql introspection tests to new framework (#1211)
• Reorganise client transaction related interfaces (#1180)
• Config-local viper, rootdir, and logger parsing (#1132)
• Migrate mutation-relation tests to new framework (#1109)
• Rework integration test framework (#1089)
• Generate gql types using col. desc (#1080)
• Extract config errors to dedicated file (#1107)
• Change terminology from query to request (#1054)
• Allow db keys to handle multiple schema versions (#1026)
• Extract query schema errors to dedicated file (#1037)
• Extract planner errors to dedicated file (#1034)
• Extract query parser errors to dedicated file (#1035)

Testing

• Remove test reference to DEFRA_ROOTDIR env var (#1328)
• Expand tests for Peer subscribe actions (#1287)
• Fix flaky TestCloseThroughContext test (#1265)
• Add gql introspection tests for patch schema (#1219)
• Explicitly state change detector split for test (#1228)
• Add test for successful one-one create mutation (#1215)
• Ensure that all databases are always closed on exit (#1187)
• Add P2P tests for Schema Update adding field (#1182)
• Migrate P2P/state tests to new framework (#1160)
• Remove sleep from subscription tests (#1156)
• Fetch documents on test execution start (#1163)
• Introduce basic testing for the version module (#1111)
• Boost test coverage for collection_update (#1050)
• Wait between P2P update retry attempts (#1052)
• Exclude auto-generated protobuf files from codecov (#1048)
• Add P2P tests for relational docs (#1042)

Continuous integration

• Add workflow that builds DefraDB AMI upon tag push (#1304)
• Allow PR title to end with a capital letter (#1291)
• Chan...

dev-unstable-21-03-2023

THIS IS A UNSTABLE DEV BUILD. DO NOT USE UNLESS SPECIFICALLY DIRECTED TO.

Built using commit: 8cf6855
Built on date: 21-03-2023.

Unstable pre-release of upcoming 0.5.

The following is a very short summary of the nature of this release:

• Several refactorings have been done to extract errors to dedicated files and to allow database keys to handle multiple schema versions.
• The DAG sync has been improved to handle highly concurrent updates and potential P2P deadlocks have been removed.
• New features have been added to the system, such as the ability to input a simple explain type argument, persist schema version at the time of commit, and add like sub-string filter.
• The system has been updated to include a new workflow that ensures dependencies build properly.
• There have been changes to the test suites, including adding P2P tests for relational docs, boosting test coverage for collection updates, and introducing basic testing for the version module.
• Several bug fixes have been implemented, including reworking P2P integration tests, changing terminology from query to request, and making sort stable and handling nil comparison.

v0.4.0 Release

DefraDB v0.4 is a major pre-production release. Until the stable version 1.0 is reached, the SemVer minor patch number will denote notable releases, which will give the project freedom to experiment and explore potentially breaking changes.

There are various new features in this release - some of which are breaking - and we invite you to review the official changelog below. Some highlights are persistence of replicators, DateTime scalars, TLS support, and GQL subscriptions.

This release does include a Breaking Change to existing v0.3.x databases. If you need help migrating an existing deployment, reach out at [email protected] or join our Discord at https://discord.source.network/.

Features

• Add basic metric functionality (#971)
• Add thread safe transactional in-memory datastore (#947)
• Persist p2p replicators (#960)
• Add DateTime custom scalars (#931)
• Add GraphQL subscriptions (#934)
• Add support for tls (#885)
• Add group by support for commits (#887)
• Add depth support for commits (#889)
• Make dockey optional for allCommits queries (#847)
• Add WithStack to the errors package (#870)
• Add event system (#834)

Fixes

• Correct errors.WithStack behaviour (#984)
• Correctly handle nested one to one joins (#964)
• Do not assume parent record exists when joining (#963)
• Change time format for HTTP API log (#910)
• Error if group select contains non-group-by fields (#898)
• Add inspection of values for ENV flags (#900)
• Remove panics from document (#881)
• Add __typename support (#871)
• Handle subscriber close (#877)
• Publish update events post commit (#866)

Refactoring

• Make rootstore require Batching and TxnDatastore (#940)
• Conceptually clarify schema vs query-language (#924)
• Decouple db.db from gql (#912)
• Merkle clock heads cleanup (#918)
• Simplify dag fetcher (#913)
• Cleanup parsing logic (#909)
• Move planner outside the gql directory (#907)
• Refactor commit nodes (#892)
• Make latest commits syntax sugar (#890)
• Remove commit query (#841)

Testing

• Add event tests (#965)
• Add new setup for testing explain functionality (#949)
• Add txn relation-type delete and create tests (#875)
• Skip change detection for tests that assert panic (#883)

Continuous integration

• Bump all gh-action versions to support node16 (#990)
• Bump ssh-agent action to v0.7.0 (#978)
• Add error message format check (#901)

Chore

• Extract (events, merkle) errors to errors.go (#973)
• Extract (datastore, db) errors to errors.go (#969)
• Extract (connor, crdt, core) errors to errors.go (#968)
• Extract inline (http and client) errors to errors.go (#967)
• Update badger version (#966)
• Move Option and Enumerable to immutables (#939)
• Add configuration of external loggers (#942)
• Strip DSKey prefixes and simplify NewDataStoreKey (#944)
• Include version metadata in cross-building (#930)
• Update to v0.23.2 the libP2P package (#908)
• Remove ipfslite dependency (#739)

v0.3.1 Release

DefraDB v0.3.1 is a minor release, primarily focusing on additional/extended features and fixes of items added in the v0.3.0 release.

Features

• Add cid support for allCommits (#857)
• Add offset support to allCommits (#859)
• Add limit support to allCommits query (#856)
• Add order support to allCommits (#845)
• Display CLI usage on user error (#819)
• Add support for dockey filters in child joins (#806)
• Add sort support for numeric aggregates (#786)
• Allow filtering by nil (#789)
• Add aggregate offset support (#778)
• Remove filter depth limit (#777)
• Add support for and-or inline array aggregate filters (#779)
• Add limit support for aggregates (#771)
• Add support for inline arrays of nillable types (#759)
• Create errors package (#548)
• Add ability to display peer id (#719)
• Add a config option to set the vlog max file size (#743)
• Explain topLevelNode like a MultiNode plan (#749)
• Make topLevelNode explainable (#737)

Fixes

• Order subtype without selecting the join child (#810)
• Correctly handles nil one-one joins (#837)
• Reset scan node for each join (#828)
• Handle filter input field argument being nil (#787)
• Ensure CLI outputs JSON to stdout when directed to pipe (#804)
• Error if given the wrong side of a one-one relationship (#795)
• Add object marker to enable return of empty docs (#800)
• Resolve the extra typeIndexJoins for _avg aggregate (#774)
• Remove _like filter operator (#797)
• Remove having gql types (#785)
• Error if child _group selected without parent groupBy (#781)
• Error nicely on missing field specifier (#782)
• Handle order input field argument being nil (#701)
• Change output to outputpath in config file template for logger (#716)
• Delete mutations not correct persisting all keys (#731)

Tooling

• Ban the usage of ioutil package (#747)
• Migrate from CircleCi to GitHub Actions (#679)

Documentation

• Clarify meaning of url param, update in-repo CLI docs (#814)
• Disclaimer of exposed to network and not encrypted (#793)
• Update logo to respect theme (#728)

Refactoring

• Replace all interface{} with any alias (#805)
• Use fastjson to parse mutation data string (#772)
• Rework limit node flow (#767)
• Make Option immutable (#769)
• Rework sum and count nodes to make use of generics (#757)
• Remove some possible panics from codebase (#732)
• Change logging calls to use feedback in CLI package (#714)

Testing

• Add tests for aggs with nil filters (#813)
• Add not equals filter tests (#798)
• Fix cli/peerid_test to not clash addresses (#766)
• Add change detector summary to test readme (#754)
• Add tests for inline array grouping (#752)

Continuous integration

• Reduce test resource usage and test with file db (#791)
• Add makefile target to verify the local module cache (#775)
• Allow PR titles to end with a number (#745)
• Add a workflow to validate pull request titles (#734)
• Fix the linter version to v1.47 (#726)

Chore

• Remove file system paths from resulting executable (#831)
• Add goimports linter for consistent imports ordering (#816)
• Improve UX by providing more information (#802)
• Change to defra errors and handle errors stacktrace (#794)
• Clean up go.mod with pruned module graphs (#756)
• Update to v0.20.3 of libp2p (#740)
• Bump to GoLang v1.18 (#721)

v0.3.0 Release

DefraDB v0.3 is a major pre-production release. Until the stable version 1.0 is reached, the SemVer minor patch number will denote notable releases, which will give the project freedom to experiment and explore potentially breaking changes.

There are several new features in this release, and we invite you to review the official changelog below. Some highlights are various new features for Grouping & Aggregation for the query system, like top-level aggregation and group filtering. Moreover, a brand new Query Explain system was added to introspect the execution plans created by DefraDB. Lastly we introduced a revamped CLI configuration system.

This release does include a Breaking Change to existing v0.2.x databases. If you need help migrating an existing deployment, reach out at [email protected] or join our Discord at https://discord.source.network/.

Features

• Add named config overrides (#659)
• Expose color and caller log options, add validation (#652)
• Add ability to explain groupNode and it's attribute(s). (#641)
• Add primary directive for schema definitions (@primary) (#650)
• Add support for aggregate filters on inline arrays (#622)
• Add explainable renderLimitNode & hardLimitNode attributes. (#614)
• Add support for top level aggregates (#594)
• Update countNode explanation to be consistent. (#600)
• Add support for stdin as input in CLI (#608)
• Explain cid & field attributes for dagScanNode (#598)
• Add ability to explain dagScanNode attribute(s). (#560)
• Add the ability to send user feedback to the console even when logging to file. (#568)
• Add ability to explain sortNode attribute(s). (#558)
• Add ability to explain sumNode attribute(s). (#559)
• Introduce top-level config package (#389)
• Add ability to explain updateNode attributes. (#514)
• Add typeIndexJoin explainable attributes. (#499)
• Add support to explain countNode attributes. (#504)
• Add CORS capability to HTTP API (#467)
• Add explaination of spans for scanNode. (#492)
• Add ability to Explain the response plan. (#385)
• Add aggregate filter support for groups only (#426)
• Configurable caller option in logger (#416)
• Add Average aggregate support (#383)
• Allow summation of aggregates (#341)
• Add ability to check DefraDB CLI version. (#339)

Fixes

• Add a check to ensure limit is not 0 when evaluating query limit and offset (#706)
• Support multiple --logger flags (#704)
• Return without an error if relation is finalized (#698)
• Logger not correctly applying named config (#696)
• Add content-type media type parsing (#678)
• Remove portSyncLock deadlock condition (#671)
• Silence cobra default errors and usage printing (#668)
• Add stdout validation when setting logging output path (#666)
• Consider --logoutput CLI flag properly (#645)
• Handle errors and responses in CLI client commands (#579)
• Rename aggregate gql types (#638)
• Error when attempting to insert value into relationship field (#632)
• Allow adding of new schema to database (#635)
• Correctly parse dockey in broadcast log event. (#631)
• Increase system's open files limit in integration tests (#627)
• Avoid populating order.ordering with empties. (#618)
• Change to supporting of non-null inline arrays (#609)
• Assert fields exist in collection before saving to them (#604)
• CLI init command to reinitialize only config file (#603)
• Add config and registry clearing to TestLogWritesMessagesToFeedbackLog (#596)
• Change $eq to _eq in the failing test. (#576)
• Resolve failing HTTP API tests via cleanup (#557)
• Ensure Makefile compatibility with macOS (#527)
• Separate out iotas in their own blocks. (#464)
• Use x/cases for titling instead of strings to handle deprecation (#457)
• Handle limit and offset in sub groups (#440)
• Issue preventing DB from restarting with no records (#437)
• log serving HTTP API before goroutine blocks (#358)

Testing

• Add integration testing for P2P. (#655)
• Fix formatting of tests with no extra brackets (#643)
• Add tests for averageNode explain. (#639)
• Add schema integration tests (#628)
• Add tests for default properties (#611)
• Specify which collection to update in test framework (#601)
• Add tests for grouping by undefined value (#543)
• Add test for querying undefined field (#544)
• Expand commit query tests (#541)
• Add cid (time-travel) query tests (#539)
• Restructure and expand filter tests (#512)
• Basic unit testing of node package (#503)
• Test filter in filter tests (#473)
• Add test for deletion of records in a relationship (#329)
• Benchmark transaction iteration (#289)

Refactoring

• Improve CLI error handling and fix small issues (#649)
• Add top-level version package (#583)
• Remove extra log levels (#634)
• Change sortNode to orderNode. (#591)
• Rework update and delete node to remove secondary planner (#571)
• Trim imported connor package (#530)
• Internal doc restructure (#471)
• Copy-paste connor fork into repo (#567)
• Add safety to the tests, add ability to catch stderr logs and add output path validation (#552)
• Change handler functions implementation and response formatting (#498)
• Improve the HTTP API implementation (#382)
• Use new logger in net/api (#420)
• Rename NewCidV1_SHA2_256 to mixedCaps (#415)
• Remove utils package ([#397](https://gi...

v0.2.1 Release

This is a minor release that resolves several bugs and implements some minor features that are only extensions of features released in v0.2.

Features

• Add ability to delete multiple documents using filter (#206)
• Add ability to delete multiple documents, using multiple ids (#196)

Fixes

• Concurrency control of Document using RWMutex (#213)
• Only log errors and above when benchmarking (#261)
• Handle proper type conversion on sort nodes (#228)
• Return empty array if no values found (#223)
• Close fetcher on error (#210)
• Installing binary using defradb name (#190)

Tooling

• Add short benchmark runner option (#263)

Documentation

• Add data format changes documentation folder (#89)
• Correcting typos (#143)
• Update generated CLI docs (#208)
• Updated readme with P2P section (#220)
• Update old or missing license headers (#205)
• Update git-chglog config and template (#195)

Refactoring

• Introduction of logging system (#67)
• Restructure db/txn/multistore structures (#199)
• Initialize database in constructor (#211)
• Purge all println and ban it (#253)

Testing

• Detect and force breaking filesystem changes to be documented (#89)
• Boost collection test coverage (#183)

Continuous integration

• Combine the Lint and Benchmark workflows so that the benchmark job depends on the lint job in one workflow (#209)
• Add rule to only run benchmark if other check are successful (#194)
• Increase linter timeout (#230)

Chore

• Remove commented out code (#238)
• Remove dead code from multi node (#186)

v0.2 Release

DefraDB v0.2 is a major pre-production release. Until the stable version 1.0 is reached, the SemVer minor patch number will denote notable releases, which will give the project freedom to experiment and explore potentially breaking changes.

This release is jam-packed with new features and a small number of breaking changes. Read the full changelog for a detailed description. Most notable features include a new Peer-to-Peer (P2P) data synchronization system, an expanded query system to support GroupBy & Aggregate operations, and lastly TimeTraveling queries allowing to query previous states of a document.

Much more than just that has been added to ensure we're building reliable software expected of any database, such as expanded test & benchmark suites, automated bug detection, performance gains, and more.

This release does include a Breaking Change to existing v0.1 databases regarding the internal data model, which affects the "Content Identifiers" we use to generate DocKeys and VersionIDs. If you need help migrating an existing deployment, reach out at [email protected] or join our Discord at https://discord.source.network.

Features

• Added Peer-to-Peer networking data synchronization (#177)
• TimeTraveling (History Traversing) query engine and doc fetcher (#59)
• Add Document Deletion with a Key (#150)
• Add support for sum aggregate (#121)
• Add support for lwwr scalar arrays (full replace on update) (#115)
• Add count aggregate support (#102)
• Add support for named relationships (#108)
• Add multi doc key lookup support (#76)
• Add basic group by functionality (#43)
• Update datastore packages to allow use of context (#48)

Bug fixes

• Only add join if aggregating child object collection (#188)
• Handle errors generated during input object thunks (#123)
• Remove new types from in-memory cache on generate error (#122)
• Support relationships where both fields have the same name (#109)
• Handle errors generated in fields thunk (#66)
• Ensure OperationDefinition case has at least one selection(#24)
• Close datastore iterator on scan close (#56) (resulted in a panic when using limit)
• Close superseded iterators before orphaning (#56) (fixes a panic in the join code)
• Move discard to after error check (#88) (did result in panic if transaction creation fails)
• Check for nil iterator before closing document fetcher (#108)

Tooling

• Added benchmark suite (#160)

Documentation

• Correcting comment typos (#142)
• Correcting README typos (#140)

Testing

• Add transaction integration tests (#175)
• Allow running of tests using badger-file as well as IM options (#128)
• Add test datastore selection support (#88)

Refactoring

• Datatype modification protection (#138)
• Cleanup Linter Complaints and Setup Makefile (#63)
• Rework document rendering to avoid data duplication and mutation (#68)
• Remove dependency on concrete datastore implementations from db package (#51)
• Remove all errors.Wrap and update them with fmt.Errorf. (#41)
• Restructure integration tests to provide better visibility (#15)
• Remove schemaless code branches (#23)

Performance

• Add badger multi scan support (#85)
• Add support for range spans (#86)

Continous integration

• Use more accurate test coverage. (#134)
• Disable Codecov's Patch Check
• Make codcov less strict for now to unblock development (#125)
• Add codecov config file. (#118)
• Add workflow that runs a job on AWS EC2 instance. (#110)
• Add Code Test Coverage with CodeCov (#116)
• Integrate GitHub Action for golangci-lint Annotations (#106)
• Add Linter Check to CircleCi (#92)

Chore

• Remove the S1038 rule of the gosimple linter. (#129)
• Update to badger v3, and use badger as default in memory store (#56)
• Make Cid versions consistent (#57)

v0.1 Alpha Build

early access