Passionate Development

Serving Grafana Private Plugin

The journey of installing a private plugin in an isolated Grafana installation (isolated as in there is no egress internet connectivity) is not for the faint hearted. Fear not, I have gone through the journey so you don’t have to.

What’s the issue?

TBD

TL;DR

The post will be long-ish as I am capturing my process in working this out. But here is the TL;DR - see I am giving you value straight up :) (no money asked!):

  • If you install your Grafana using Grafana helm chart - you can put the plugin zip URL directly under the plugins section. For example:
plugins:
  - http://plugin-repo.plugin-demo.svc.cluster.local/hello-world/versions/1.0.0/hellworld-1.0.0.zip;hello-world
  • The next big question is how/where to host your plugin? Two requirements:
    • Obviously it is a URL that your Grafana has access to.
    • Secondly it should not be under any authentication mechanism - Grafana can’t handle authenticated download.

Ok that’s it!

Now, if you want to read my deep dive into the Grafana private plugin or if you are interested in installing your plugin using grafana cli keep on reading.

Environment Setup

If you want to check out the code I used in this post, check out this repo.

I am using minikube for my testing cluster and I install Grafana locally using Grafana helm chart as mentioned above, this is the command:

helm install grafana grafana/grafana -f grafana/values.yaml --namespace grafana --version 6.59.4

values.yaml is from the repo above, but you will see what it contains further down the post.

Create a Grafana Plugin

Follow the Grafana plugin documentation - should give you some idea on how to create one. Although to be honest, the documentation can be made much clearer.

Grafana plugin is mostly a “frontend” plugin (and an optional backend component). You will need a NodeJS environment setup for this as this is what the plugin will be based on. The backend component is a go app.

This is how I created a plugin:

> npx @grafana/create-plugin@latest
? What is going to be the name of your plugin? hello-world
? What is the organization name of your plugin? passionate-development
? How would you describe your plugin? an app displaying hello-world
? What type of plugin would you like? app
? Do you want a backend part of your plugin? No
? Do you want to add Github CI and Release workflows? No
? Do you want to add a Github workflow for automatically checking "Grafana API compatibility" on PRs? No

The usual install & build for a Node app.

yarn install
yarn build

yarn build will create a /dist directory and we are meant to package the plugin as a .zipfile like so:

mv dist/ hello-world && zip hellworld-1.0.0.zip hello-world -r && rm -rf hello-world/

Signing a plugin

Signing a plugin is out of scope for my experimentation. It needs an account on Grafana cloud, which I couldn’t be bothered of creating at the moment.

To load an unsigned plugin in Grafana, this has to be white listed in the Grafana config (more on this later).

Loading a private plugin to Grafana

So you have built a plugin now you need to load it to your Grafana, how would you do this?

You can install your plugin at any time using the grafana cli command. This command is included in the Grafana installation. As mentioned earlier, I’ve deployed Grafana in Kubernetes hence I would kubectl exec -it into the Grafana pod and run something like grafana cli plugins install my-plugin

The downside of this approach is, you’d need to restart your Grafana after the installation. I’d imagine it’s more useful to the plugin loaded when Grafana started.

To achieve this you need to pass in the GF_INSTALL_PLUGINS environment variables to Grafana. If you are using Grafana helm charts, the helm chart provides plugins directive that will set the env var for you:

plugins:
  - http://plugin-repo.plugin-demo.svc.cluster.local/hello-world/versions/1.0.0/hellworld-1.0.0.zip;my-little-pony

Note the GF_INSTALL_PLUGINS in the background will just issue grafana cli plugins install {plugin} when Grafana starts as stated in the doc - in the doc it describes 3 options for a private plugin you really want to use option 3 - which is the full URL to the plugin.

YOU DO NOT NEED TO SET UP A PRIVATE HOSTING REPO

After this line is work in progress….

Create a private hosting repo

You only need a static page hosting basically - but remember it cannot be an authenticated endpoint as grafana doesn’t have the ability to pass in any auth information.

For this example - let’s configure an nginx. But you could configure any static hosting of your choice, like S3 for example:

kubectl apply -f plugin-repo.yaml

Repo structure

This is the bit that is missing the most.

Let’s go to your grafana pod and test connectivity to the local

k exec -it grafana-88484db8c-5kcbg -- /bin/bash
curl -v http://plugin-repo.plugin-demo.svc.cluster.local:8080/index.html

Let check list the plugins on the repo:

I have no name!@grafana-88484db8c-5kcbg:/opt/bitnami/grafana$
grafana-cli --repo http://plugin-repo.plugin-demo.svc.cluster.local:8080 plugins list-remote
Failed to send requesterror404 not found errorError: ✗ Failed to send request: 404 not found error

It is a good idea to tail the nginx log:

..
2023/09/16 04:07:32 [error] 30#30: *17 open() "/usr/share/nginx/html/repo" failed (2: No such file or directory), client: 10.244.0.6, server: localhost, request: "GET /repo HTTP/1.1", host: "plugin-repo.plugin-demo.svc.cluster.local:8080"
..

So it needs to find /repo - now you might think repo is a folder, wrong it is a file! Here’s an examle:

{
  "plugins": [
    {
      "id": "hello-world",
      "type": "app",
      "url": "https://github.com/cemeng/grafana-private-plugin-demo",
      "versions": [
        {
          "version": "1.0.0"
        }
      ]
    }
  ]
}

Back to the thing

grafana-5777b59fff-zfwkk:/usr/share/grafana$ grafana cli --repo http://plugin-repo.plugin-demo.svc.cluster.local:8080 plugins list-remote
id: hello-world version: 1.0.0


grafana-5777b59fff-zfwkk:/usr/share/grafana$ grafana cli --repo http://plugin-repo.plugin-demo.svc.cluster.local:8080 plugins list-versions hello-world
Error: ✗ Failed to find requested plugin, check if the plugin_id (hello-world) is correct: 404 not found error

Looking at nginx log 2023/09/16 04:15:07 [error] 31#31: *22 open() “/usr/share/nginx/html/repo/hello-world” failed (20: Not a directory), client: 10.244.0.6, server: localhost, request: “GET /repo/hello-world HTTP/1.1”, host: “plugin-repo.plugin-demo.svc.cluster.local:8080”

Ok let’s add that hello-world directory with an index file (in the form of json like this)

mkdir repo/hello-world
cat > repo/hello-world/index.html
{
  "versions": [
    {
      "arch": {
        "any": {}
      },
      "version": "1.0.0"
    }
  ]
}

Let’s try again:

grafana-5777b59fff-zfwkk:/usr/share/grafana$ grafana cli --repo http://plugin-repo.plugin-demo.svc.cluster.local plugins list-versions hello-world
1.0.0

Ok that works! now let’s try installing it!

grafana cli --repo http://plugin-repo.plugin-demo.svc.cluster.local plugins install hello-world 1.0.0
Error: ✗ failed to download plugin archive: 404: <html>
<head><title>404 Not Found</title></head>

Nginx log:

10.244.0.6 - - [16/Sep/2023:04:47:43 +0000] "GET /hello-world/versions/1.0.0/download HTTP/1.1" 404 153 "-" "grafana 10.1.1" "-"

Ok let’s do this:

pwd
mkdir /hello-world
mkdir /hello-world/versions
mkdir /hello-world/versions/1.0.0

Copy the zip as a file named download without any extension, yes you read that right. This is the bit that I don’t understand, seems really odd convention. But if you see the code - it is what it is. https://github.com/grafana/grafana/blob/40c12c17bffcec8c3512c597e7f228b296b0c218/pkg/plugins/repo/service.go#L93

As I go through the grafana code and docs https://grafana.com/docs/grafana/latest/cli/#override-default-plugin-zip-url A much simpler option would be the specify --pluginURL flag rather than --repo. But anyway.

k cp hellworld-1.0.0.zip plugin-repo:/usr/share/nginx/html/hello-world/versions/1.0.0/download -n plugin-demo

grafana-5777b59fff-zfwkk:/usr/share/grafana$ grafana cli --repo http://plugin-repo.plugin-demo.svc.cluster.local plugins install hello-world 1.0.0
✔ Downloaded and extracted passionatedevelopment-helloworld-app v1.0.0 zip successfully to /var/lib/grafana/plugins/hello-world

Please restart Grafana after installing or removing plugins. Refer to Grafana documentation for instructions if necessary.

k cp hellworld-1.0.0.zip plugin-repo:/usr/share/nginx/html/hello-world/versions/1.0.0/ -n plugin-demo

grafana cli --pluginUrl http://plugin-repo.plugin-demo.svc.cluster.local/hello-world/versions/1.0.0/hellworld-1.0.0.zip plugins install hello-world 1.0.0
✔ Downloaded and extracted passionatedevelopment-helloworld-app v1.0.0 zip successfully to /var/lib/grafana/plugins/hello-world

Please restart Grafana after installing or removing plugins. Refer to Grafana documentation for instructions if necessary.

https://volkovlabs.io/blog/installing-grafana-plugins-from-a-private-repository-805b54a1add3/

k exec -it plugin-repo -- /bin/bash
cd

Install via helm

https://github.com/grafana/helm-charts/blob/main/charts/grafana/values.yaml#L541

  ## You can also use other plugin download URL, as long as they are valid zip files,
  ## and specify the name of the plugin after the semicolon. Like this:
  # - https://grafana.com/api/plugins/marcusolsson-json-datasource/versions/1.3.2/download;marcusolsson-json-datasource

plugins:
  - http://plugin-repo.plugin-demo.svc.cluster.local/hello-world/versions/1.0.0/hellworld-1.0.0.zip;my-little-pony

grafana.ini:
  plugins:
    allow_loading_unsigned_plugins: passionatedevelopment-helloworld-app

Tidbits

Also stupidly if you configure it on other port than 80 (and presumably 443) - you will be in a world of pain:

grafana-5777b59fff-zfwkk:/usr/share/grafana$ grafana cli --repo http://plugin-repo.plugin-demo.svc.cluster.local:8080/ plugins list-remote
Failed to send requesterrorGet "http://plugin-repo.plugin-demo.svc.cluster.local/repo/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)Error: ✗ Failed to send request: Get "http://plugin-repo.plugin-demo.svc.cluster.local/repo/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)