Skip to content

Commit

Permalink
Add prometheus support (Fixes #3407) (#3453)
Browse files Browse the repository at this point in the history 
* Introduce the Prometheus lib

* Start plugin

* Unit testing

* Adapt usage of unique parameter to groups

* Skip test when no prometheus installed

* Add test for events

* Enable Prometheus in functional test

* Add section about Prometheus

* Move plugins section above monitoring

* Fix link format in docs

* Remove TODO

* nit docs
  • Loading branch information
@leplatrem
leplatrem authored Oct 16, 2024
1 parent b79907e commit 797fb75
Show file tree
Hide file tree
Showing 10 changed files with 399 additions and 96 deletions.
1 change: 1 addition & 0 deletions constraints.in
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ psycopg2
zope.sqlalchemy
# monitoring
newrelic
prometheus-client
sentry-sdk[sqlalchemy]
statsd
werkzeug
Expand Down
16 changes: 4 additions & 12 deletions constraints.txt
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#
# This file is autogenerated by pip-compile with Python 3.10
# This file is autogenerated by pip-compile with Python 3.12
# by the following command:
#
# pip-compile --output-file=constraints.txt --strip-extras constraints.in
Expand Down Expand Up @@ -40,16 +40,12 @@ coverage==7.4.0
# via pytest-cov
dockerflow==2024.4.2
# via -r constraints.in
exceptiongroup==1.2.0
# via pytest
execnet==2.0.2
# via pytest-cache
fqdn==1.5.1
# via jsonschema
greenlet==3.0.3
# via
# playwright
# sqlalchemy
# via playwright
hupper==1.12
# via pyramid
idna==3.7
Expand Down Expand Up @@ -102,6 +98,8 @@ playwright==1.47.0
# via -r constraints.in
pluggy==1.5.0
# via pytest
prometheus-client==0.21.0
# via -r constraints.in
psycopg2==2.9.9
# via -r constraints.in
pyee==12.0.0
Expand Down Expand Up @@ -186,12 +184,6 @@ statsd==4.0.1
# via -r constraints.in
swagger-spec-validator==3.0.3
# via bravado-core
tomli==2.0.1
# via
# build
# coverage
# pyproject-hooks
# pytest
transaction==5.0
# via
# -r constraints.in
Expand Down
9 changes: 2 additions & 7 deletions docs/configuration/production.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -220,19 +220,14 @@ In the configuration of the CDN service, you should also:
Monitoring
----------

In order to enable monitoring features like *statsd*, install
In order to enable monitoring features (eg. *Prometheus* or *StatsD*), install
extra requirements:

::

make install-monitoring

And configure its URL:

.. code-block :: ini
# StatsD
kinto.statsd_url = udp://carbon.server:8125
See :ref:`settings section <monitoring-with-prometheus>` for the configuration aspects.

Counters
::::::::
Expand Down
171 changes: 97 additions & 74 deletions docs/configuration/settings.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -301,6 +301,83 @@ hello view <api-utilities>`.
# kinto.http_host = production.server.com:7777
.. _configuration-plugins:

Plugins
=======

It is possible to extend the default Kinto behaviors by using "plugins".

The list of plugins to load at startup can be specified in the settings, as a
list of Python modules:

.. code-block:: ini
kinto.includes = kinto.plugins.default_bucket
kinto.plugins.history
kinto.plugins.admin
kinto-attachment
custom-myplugin
+---------------------------------------+--------------------------------------------------------------------------+
| Built-in plugins | What does it do? |
+=======================================+==========================================================================+
| ``kinto.plugins.accounts`` | It allows users to sign-up and authenticate using username and password |
| | (:ref:`more details <api-accounts>`). |
+---------------------------------------+--------------------------------------------------------------------------+
| ``kinto.plugins.admin`` | It is a Web admin UI to manage data from a Kinto server. |
| | (:ref:`more details <kinto-admin>`). |
+---------------------------------------+--------------------------------------------------------------------------+
| ``kinto.plugins.default_bucket`` | It enables a personal bucket ``default``, where collections are created |
| | implicitly (:ref:`more details <buckets-default-id>`). |
+---------------------------------------+--------------------------------------------------------------------------+
| ``kinto.plugins.flush`` | Adds an endpoint to completely remove all data from the database backend |
| | for testing/staging purposes. (:ref:`more details <api-flush>`). |
+---------------------------------------+--------------------------------------------------------------------------+
| ``kinto.plugins.history`` | It tracks every action performed on objects within a bucket |
| | (:ref:`more details <api-history>`). |
+---------------------------------------+--------------------------------------------------------------------------+
| ``kinto.plugins.openid`` | It allows to authenticate users using OpenID Connect from Google, |
| | Microsoft, Auth0, etc. (:ref:`more details <api-openid>`). |
+---------------------------------------+--------------------------------------------------------------------------+
| ``kinto.plugins.quotas`` | It allows to limit storage per collection size, number of records, etc. |
| | (:ref:`more details <api-quotas>`). |
+---------------------------------------+--------------------------------------------------------------------------+
| ``kinto.plugins.prometheus`` | Send metrics about backend duration, authentication, endpoints hits, .. |
| | (:ref:`more details <monitoring-with-prometheus>`). |
+---------------------------------------+--------------------------------------------------------------------------+
| ``kinto.plugins.statsd`` | Send metrics about backend duration, authentication, endpoints hits, .. |
| | (:ref:`more details <monitoring-with-statsd>`). |
+---------------------------------------+--------------------------------------------------------------------------+


There are `many available packages`_ in the Pyramid ecosystem, and it is straightforward to build one,
since the specified module must just define an ``includeme(config)`` function.

.. _many available packages: https://github.com/ITCase/awesome-pyramid

See `our list of community plugins <https://github.com/Kinto/kinto/wiki/Plugins>`_.

See also: :ref:`tutorial-write-plugin` for more in-depth informations on how
to create your own plugin.


Pluggable components
::::::::::::::::::::

:term:`Pluggable <pluggable>` components can be substituted from configuration files,
as long as the replacement follows the original component API.

.. code-block:: ini
kinto.logging_renderer = your_log_renderer.CustomRenderer
This is the simplest way to extend *Kinto*, but will be limited to its
existing components (cache, storage, log renderer, ...).

In order to add extra features, including external packages is the way to go!


Logging and Monitoring
======================

Expand Down Expand Up @@ -413,13 +490,33 @@ Or the equivalent environment variables:
The application sends an event on startup (mainly for setup check).


.. _monitoring-with-prometheus:

Monitoring with Prometheus
::::::::::::::::::::::::::

Requires the ``prometheus-client`` package (installed with ``kinto[monitoring]``).

Prometheus metrics can be enabled with (disabled by default):

.. code-block:: ini
kinto.includes = kinto.plugins.prometheus
Metrics can then be crawled from the ``/__metrics__`` endpoint.


.. _monitoring-with-statsd:

Monitoring with StatsD
::::::::::::::::::::::

Requires the ``statsd`` package.

.. note::

Only one of *Prometheus* and *StatsD* can be enabled. It will take precedence and the other one will be ignored.

+------------------------+----------------------------------------+--------------------------------------------------------------------------+
| Setting name | Default | What does it do? |
+========================+========================================+==========================================================================+
Expand Down Expand Up @@ -475,80 +572,6 @@ New Relic can be enabled (disabled by default):
kinto.newrelic_env = prod
.. _configuration-plugins:

Plugins
=======

It is possible to extend the default Kinto behaviors by using "plugins".

The list of plugins to load at startup can be specified in the settings, as a
list of Python modules:

.. code-block:: ini
kinto.includes = kinto.plugins.default_bucket
kinto.plugins.history
kinto.plugins.admin
kinto-attachment
custom-myplugin
+---------------------------------------+--------------------------------------------------------------------------+
| Built-in plugins | What does it do? |
+=======================================+==========================================================================+
| ``kinto.plugins.accounts`` | It allows users to sign-up and authenticate using username and password |
| | (:ref:`more details <api-accounts>`). |
+---------------------------------------+--------------------------------------------------------------------------+
| ``kinto.plugins.admin`` | It is a Web admin UI to manage data from a Kinto server. |
| | (:ref:`more details <kinto-admin>`). |
+---------------------------------------+--------------------------------------------------------------------------+
| ``kinto.plugins.default_bucket`` | It enables a personal bucket ``default``, where collections are created |
| | implicitly (:ref:`more details <buckets-default-id>`). |
+---------------------------------------+--------------------------------------------------------------------------+
| ``kinto.plugins.flush`` | Adds an endpoint to completely remove all data from the database backend |
| | for testing/staging purposes. (:ref:`more details <api-flush>`). |
+---------------------------------------+--------------------------------------------------------------------------+
| ``kinto.plugins.history`` | It tracks every action performed on objects within a bucket |
| | (:ref:`more details <api-history>`). |
+---------------------------------------+--------------------------------------------------------------------------+
| ``kinto.plugins.openid`` | It allows to authenticate users using OpenID Connect from Google, |
| | Microsoft, Auth0, etc. (:ref:`more details <api-openid>`). |
+---------------------------------------+--------------------------------------------------------------------------+
| ``kinto.plugins.quotas`` | It allows to limit storage per collection size, number of records, etc. |
| | (:ref:`more details <api-quotas>`). |
+---------------------------------------+--------------------------------------------------------------------------+
| ``kinto.plugins.statsd`` | Send metrics about backend duration, authentication, endpoints hits, .. |
| | (:ref:`more details <monitoring-with-statsd>`). |
+---------------------------------------+--------------------------------------------------------------------------+


There are `many available packages`_ in Pyramid ecosystem, and it is straightforward to build one,
since the specified module must just define an ``includeme(config)`` function.

.. _many available packages: https://github.com/ITCase/awesome-pyramid

See `our list of community plugins <https://github.com/Kinto/kinto/wiki/Plugins>`_.

See also: :ref:`tutorial-write-plugin` for more in-depth informations on how
to create your own plugin.


Pluggable components
::::::::::::::::::::

:term:`Pluggable <pluggable>` components can be substituted from configuration files,
as long as the replacement follows the original component API.

.. code-block:: ini
kinto.logging_renderer = your_log_renderer.CustomRenderer
This is the simplest way to extend *Kinto*, but will be limited to its
existing components (cache, storage, log renderer, ...).

In order to add extra features, including external packages is the way to go!


.. _configuration-authentication:

Authentication
Expand Down
5 changes: 4 additions & 1 deletion kinto/core/testing.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,16 @@
from kinto.core import DEFAULT_SETTINGS
from kinto.core.storage import generators
from kinto.core.utils import encode64, follow_subrequest, memcache, sqlalchemy
from kinto.plugins import statsd
from kinto.plugins import prometheus, statsd


skip_if_ci = unittest.skipIf("CI" in os.environ, "ci")
skip_if_no_postgresql = unittest.skipIf(sqlalchemy is None, "postgresql is not installed.")
skip_if_no_memcached = unittest.skipIf(memcache is None, "memcached is not installed.")
skip_if_no_statsd = unittest.skipIf(not statsd.statsd_module, "statsd is not installed.")
skip_if_no_prometheus = unittest.skipIf(
not prometheus.prometheus_module, "prometheus is not installed."
)


class DummyRequest(mock.MagicMock):
Expand Down
Loading

0 comments on commit 797fb75

Please sign in to comment.