From 04ada13b61553ce2dc2cfe4c88f6b5c2ef983958 Mon Sep 17 00:00:00 2001 From: Muralidhar Basani Date: Fri, 14 Feb 2025 14:56:16 +0100 Subject: [PATCH 1/3] Updated installation notes --- README.rst | 70 +++++++++++++++++++++++++++------ rest-proxy-karapace.config.json | 8 ++++ website/source/about.rst | 6 ++- website/source/install.rst | 25 ++++++++++++ website/source/quickstart.rst | 12 +++--- 5 files changed, 103 insertions(+), 18 deletions(-) create mode 100644 rest-proxy-karapace.config.json diff --git a/README.rst b/README.rst index 73919ec34..89947247c 100644 --- a/README.rst +++ b/README.rst @@ -83,15 +83,60 @@ what you need to change. .. _`Here`: https://github.com/Aiven-Open/karapace/blob/master/karapace.config.json -Source install --------------- +Using Sources +------------- + +Install +^^^^^^^ -Alternatively you can do a source install using:: +You can do a source install using:: pip install . -Quickstart -========== +Trouble shooting notes : +- An updated version of wheel (https://pypi.org/project/wheel/) is required. +- Create and activate virtual environment (venv) to manage dependencies + +Run +^^^ +- Make sure kafka is running. + +Start Karapace. This shout start karapace on http://localhost:8081 :: + + $ karapace karapace.config.json + +Verify in browser http://localhost:8081/subjects should return an array of subjects if exist or an empty array. +or with curl :: + + $ curl -X GET http://localhost:8081/subjects + +Start Karapace rest proxy. This shout start karapace on http://localhost:8082 :: + + karapace rest-proxy-karapace.config.json + +To enable authorization & authentication on the rest proxy, configure 'sasl_mechanism' in the config with values like PLAIN/OAUTHBEARER :: + + sasl_mechanism = "OAUTHBEARER", + sasl_oauth_token_provider = token_provider, + security_protocol="SASL_SSL", + ssl_cafile="ca.pem", + +If 'sasl_mechanism' is configured to PLAIN:: + + sasl_mechanism = "PLAIN", + security_protocol = "SASL_PLAIN", + sasl_plain_username = "your_username", + sasl_plain_password = "your_password" + +There is a detailed section about OAuth2 authentication below. + +Verify with list topics:: + + $ curl "http://localhost:8082/topics" + + +Schema Registry Api reference +============================= To register the first version of a schema under the subject "test" using Avro schema:: @@ -171,6 +216,9 @@ Change compatibility requirement to FULL for the test-key subject:: --data '{"compatibility": "FULL"}' http://localhost:8081/config/test-key {"compatibility":"FULL"} +Schema Registry Rest proxy Api reference +======================================== + List topics:: $ curl "http://localhost:8082/topics" @@ -185,18 +233,18 @@ Produce a message backed up by schema registry:: '{"value_schema": "{\"namespace\": \"example.avro\", \"type\": \"record\", \"name\": \"simple\", \"fields\": \ [{\"name\": \"name\", \"type\": \"string\"}]}", "records": [{"value": {"name": "name0"}}]}' http://localhost:8082/topics/my_topic -Create a consumer:: +Create a consumer with consumer group 'avro_consumers' and consumer instance 'my_consumer' :: $ curl -X POST -H "Content-Type: application/vnd.kafka.v2+json" -H "Accept: application/vnd.kafka.v2+json" \ --data '{"name": "my_consumer", "format": "avro", "auto.offset.reset": "earliest"}' \ http://localhost:8082/consumers/avro_consumers -Subscribe to the topic we previously published to:: +Subscribe to the topic we previously created :: $ curl -X POST -H "Content-Type: application/vnd.kafka.v2+json" --data '{"topics":["my_topic"]}' \ http://localhost:8082/consumers/avro_consumers/instances/my_consumer/subscription -Consume previously published message:: +Consume previously produced message:: $ curl -X GET -H "Accept: application/vnd.kafka.avro.v2+json" \ http://localhost:8082/consumers/avro_consumers/instances/my_consumer/records?timeout=1000 @@ -596,10 +644,10 @@ Example of complete authorization file ] } -Karapace Schema Registry access to the schemas topic +Karapace Schema Registry access to the _schemas topic ==================================================== -The principal used by the Karapace Schema Registry has to have adequate access to the schemas topic (see the ``topic_name`` configuration option above). +The principal used by the Karapace Schema Registry has to have adequate access to the _schemas topic (see the ``topic_name`` configuration option above). In addition to what is required to access the topic, as described in the Confluent Schema Registry documentation_, the unique, single-member consumer group used by consumers in the schema registry needs ``Describe`` and ``Read`` permissions_ on the group. These unique (per instance of the schema registry) consumer group names are prefixed by ``karapace-autogenerated-``, followed by a random string. @@ -644,7 +692,7 @@ In that view the future extension of the normalization process isn't considered Uninstall ========= -To unistall Karapace from the system you can follow the instructions described below. We would love to hear your reasons for uninstalling though. Please file an issue if you experience any problems or email us_ with feedback +To uninstall Karapace from the system, you can follow the instructions described below. We would love to hear your reasons for uninstalling though. Please file an issue if you experience any problems or email us_ with feedback .. _`us`: mailto:opensource@aiven.io diff --git a/rest-proxy-karapace.config.json b/rest-proxy-karapace.config.json new file mode 100644 index 000000000..a4a4ddded --- /dev/null +++ b/rest-proxy-karapace.config.json @@ -0,0 +1,8 @@ +{ + "host": "127.0.0.1", + "port": 8082, + "advertised_hostname": "localhost", + "log_level": "INFO", + "kafka_bootstrap_servers": "localhost:9092", + "karapace_rest": true +} diff --git a/website/source/about.rst b/website/source/about.rst index ab491583a..026975101 100644 --- a/website/source/about.rst +++ b/website/source/about.rst @@ -1,4 +1,8 @@ About Karapace ============== -There are lots of reasons to use schemas alongside your Kafka payloads; Karapace gives you an open source solution for handling your schemas. +There are lots of reasons to use schemas alongside your Kafka payloads; Karapace gives you an open source solution for handling your schemas for the following data formats + +- Json +- Avro +- Protobuf diff --git a/website/source/install.rst b/website/source/install.rst index ac6d5459f..b23e0077c 100644 --- a/website/source/install.rst +++ b/website/source/install.rst @@ -38,3 +38,28 @@ Source install Alternatively you can do a source install using:: pip install . + +Trouble shooting notes : +- An updated version of wheel (https://pypi.org/project/wheel/) is required. +- Create and activate virtual environment (venv) to manage dependencies + +Run +^^^ +- Make sure kafka is running. + +Start Karapace. This shout start karapace on http://localhost:8081 :: + + $ karapace karapace.config.json + +Verify in browser http://localhost:8081/subjects should return an array of subjects if exist or an empty array. +or with curl :: + + $ curl -X GET http://localhost:8081/subjects + +Start Karapace rest proxy. This shout start karapace on http://localhost:8082 :: + + karapace rest-proxy-karapace.config.json + +Verify with list topics:: + + $ curl "http://localhost:8082/topics" diff --git a/website/source/quickstart.rst b/website/source/quickstart.rst index f640e68d2..18c091ce6 100644 --- a/website/source/quickstart.rst +++ b/website/source/quickstart.rst @@ -1,8 +1,8 @@ -Quickstart examples -=================== +Api References +============== -Schema Hub ----------- +Schema Registry +--------------- To register the first version of a schema under the subject "test" using Avro schema:: @@ -83,8 +83,8 @@ Change compatibility requirement to FULL for the test-key subject:: {"compatibility":"FULL"} -REST API ---------- +REST Proxy +---------- List topics:: From c86c37e60a7ae09492d06b9928bd47c801940c19 Mon Sep 17 00:00:00 2001 From: Muralidhar Basani Date: Fri, 14 Feb 2025 15:12:41 +0100 Subject: [PATCH 2/3] Updated installation notes --- README.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/README.rst b/README.rst index 89947247c..b7276f0a6 100644 --- a/README.rst +++ b/README.rst @@ -95,6 +95,7 @@ You can do a source install using:: Trouble shooting notes : - An updated version of wheel (https://pypi.org/project/wheel/) is required. +- An updated version of 'go' and 'rust' is required - Create and activate virtual environment (venv) to manage dependencies Run From 67e6e50aa226754d43ce4e01a789a7e06accf485 Mon Sep 17 00:00:00 2001 From: Muralidhar Basani Date: Wed, 19 Feb 2025 08:49:20 +0100 Subject: [PATCH 3/3] Spelling fix --- README.rst | 2 +- website/source/install.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.rst b/README.rst index b7276f0a6..e06643756 100644 --- a/README.rst +++ b/README.rst @@ -93,7 +93,7 @@ You can do a source install using:: pip install . -Trouble shooting notes : +Troubleshooting notes : - An updated version of wheel (https://pypi.org/project/wheel/) is required. - An updated version of 'go' and 'rust' is required - Create and activate virtual environment (venv) to manage dependencies diff --git a/website/source/install.rst b/website/source/install.rst index b23e0077c..d3b3eef73 100644 --- a/website/source/install.rst +++ b/website/source/install.rst @@ -39,7 +39,7 @@ Alternatively you can do a source install using:: pip install . -Trouble shooting notes : +Troubleshooting notes : - An updated version of wheel (https://pypi.org/project/wheel/) is required. - Create and activate virtual environment (venv) to manage dependencies