Serving Grafana Private Plugin

The journey of installing a private plugin in an isolated Grafana installation (isolated as in there is no egress internet connectivity) is not for the faint hearted. Fear not, I have gone through the journey so you don’t have to.

What’s the issue?

TBD

TL;DR

The post will be long-ish as I am capturing my process in working this out. But here is the TL;DR - see I am giving you value straight up :) (no money asked!):

• If you install your Grafana using Grafana helm chart - you can put the plugin zip URL directly under the plugins section. For example:

plugins:
  - http://plugin-repo.plugin-demo.svc.cluster.local/hello-world/versions/1.0.0/hellworld-1.0.0.zip;hello-world

• The next big question is how/where to host your plugin? Two requirements:
  • Obviously it is a URL that your Grafana has access to.
  • Secondly it should not be under any authentication mechanism - Grafana can’t handle authenticated download.

Ok that’s it!

Now, if you want to read my deep dive into the Grafana private plugin or if you are interested in installing your plugin using grafana cli keep on reading.

Environment Setup

If you want to check out the code I used in this post, check out this repo.

I am using minikube for my testing cluster and I install Grafana locally using Grafana helm chart as mentioned above, this is the command:

helm install grafana grafana/grafana -f grafana/values.yaml --namespace grafana --version 6.59.4

values.yaml is from the repo above, but you will see what it contains further down the post.

Create a Grafana Plugin

Follow the Grafana plugin documentation - should give you some idea on how to create one. Although to be honest, the documentation can be made much clearer.

Grafana plugin is mostly a “frontend” plugin (and an optional backend component). You will need a NodeJS environment setup for this as this is what the plugin will be based on. The backend component is a go app.

This is how I created a plugin:

> npx @grafana/create-plugin@latest
? What is going to be the name of your plugin? hello-world
? What is the organization name of your plugin? passionate-development
? How would you describe your plugin? an app displaying hello-world
? What type of plugin would you like? app
? Do you want a backend part of your plugin? No
? Do you want to add Github CI and Release workflows? No
? Do you want to add a Github workflow for automatically checking "Grafana API compatibility" on PRs? No

The usual install & build for a Node app.

yarn install
yarn build

yarn build will create a /dist directory and we are meant to package the plugin as a .zipfile like so:

mv dist/ hello-world && zip hellworld-1.0.0.zip hello-world -r && rm -rf hello-world/

Signing a plugin

Signing a plugin is out of scope for my experimentation. It needs an account on Grafana cloud, which I couldn’t be bothered of creating at the moment.

To load an unsigned plugin in Grafana, this has to be white listed in the Grafana config (more on this later).

Loading a private plugin to Grafana

So you have built a plugin now you need to load it to your Grafana, how would you do this?

You can install your plugin at any time using the grafana cli command. This command is included in the Grafana installation. As mentioned earlier, I’ve deployed Grafana in Kubernetes hence I would kubectl exec -it into the Grafana pod and run something like grafana cli plugins install my-plugin

The downside of this approach is, you’d need to restart your Grafana after the installation. I’d imagine it’s more useful to the plugin loaded when Grafana started.

To achieve this you need to pass in the GF_INSTALL_PLUGINS environment variables to Grafana. If you are using Grafana helm charts, the helm chart provides plugins directive that will set the env var for you. There are 2 options on passing through the plugin URL.

The easiest way and the one that I’d highly recommend is to pass in the full URL for the plugin URL like this:

plugins:
  - http://plugin-repo.plugin-demo.svc.cluster.local/hello-world/versions/1.0.0/hellworld-1.0.0.zip;my-little-pony

Note the GF_INSTALL_PLUGINS in the background will just issue grafana cli plugins install {plugin} when Grafana starts as stated in the doc - in the doc it describes 3 options for a private plugin you really want to use option 3 - which is the full URL to the plugin.

The more convoluted way, which I have described on the next session is to setup your own private plugin hosting repo. I don’t see much reason on doing this unless you want to manage your plugin using grafana cli. The private plugin hosting way:

plugins:
  - http://plugin-repo.plugin-demo.svc.cluster.local/plugins/hello-world 1.0.0;my-little-pony

Have I not persuade you enough not to go through the craziness of setting up a private plugin hosting? Ok, your choice continue reading.

Create a private hosting repo

You only need a static page hosting basically - but remember it cannot be an authenticated endpoint as Grafana doesn’t have the ability to pass in any auth information.

For this example - let’s configure an nginx server to serve our plugin. But you could configure any static hosting of your choice, like AWS S3 for example.

You can use this manifest to install an nginx pod with a service.

kubectl apply -f plugin-repo.yaml

Grafana plugin repo structure

This is the bit that is the hardest to work out, but the TL;DR here is a plugin repo structure:

root@plugin-repo:/usr/share/nginx/html# tree
.
|-- 50x.html
|-- index.html
`-- plugins
    |-- hello-world
    |   `-- versions
    |       `-- 1.0.0
    |           `-- download # this a zip file 🤷🏻‍♂️
    `-- repo
        |-- hello-world
        |   `-- index.html # this is a JSON file 🤷🏻‍♂️
        `-- index.html     # this is a JSON file 🤷🏻‍♂️

Shout out to Volkov Labs which wrote installing grafana plugins from a private repo article - of which I based a lot of my experiment of.

Let’s go to your Grafana pod and test connectivity to the local

$ k exec -it grafana-d55f9bc77-6k5kl -- /bin/bash

grafana-d55f9bc77-6k5kl:/usr/share/grafana$ curl -v http://plugin-repo.plugin-demo.svc.cluster.local:8080/index.html

Let check list the plugins on the repo:

$ k exec -it grafana-d55f9bc77-6k5kl -- /bin/bash

grafana-d55f9bc77-6k5kl:/usr/share/grafana$ grafana cli --repo http://plugin-repo.plugin-demo.svc.cluster.local/plugins plugins list-remote
Failed to send requesterror404 not found errorError: ✗ Failed to send request: 404 not found error

It is a good idea to tail the nginx log:

2023/09/23 10:35:10 [error] 30#30: *2 open() "/usr/share/nginx/html/plugins/repo" failed (2: No such file or directory), client: 10.244.0.19, server: localhost, request: "GET /plugins/repo HTTP/1.1", host: "plugin-repo.plugin-demo.svc.cluster.local"

So the cli is looking for /plugins/repo - the tricky part is the repo needs to return a JSON file, yes /repo. It would clearer if the cli was to look for a file with an extension like /repo/index.json - but it is what it is.

To be technically correct, I should create an index.json file and make it as a default file for the directory - this needs to be configured in nginx. But since it is a demo, I can’t be bothered - let’s create an index.html - which an actually a JSON file like this one below:

{
  "plugins": [
    {
      "id": "hello-world",
      "type": "app",
      "url": "https://github.com/cemeng/grafana-private-plugin-demo",
      "versions": [
        {
          "version": "1.0.0"
        }
      ]
    }
  ]
}

Back to the Grafana pod:

grafana-d55f9bc77-6k5kl:/usr/share/grafana$ grafana cli --repo http://plugin-repo.plugin-demo.svc.cluster.local/plugins plugins list-remote
id: hello-world version: 1.0.0


grafana-d55f9bc77-6k5kl:/usr/share/grafana$ grafana cli --repo http://plugin-repo.plugin-demo.svc.cluster.local/plugins plugins list-versions hello-world
Error: ✗ Failed to find requested plugin, check if the plugin_id (hello-world) is correct: 404 not found error

Looking at Nginx log:

2023/09/23 10:42:37 [error] 32#32: *4 open() "/usr/share/nginx/html/plugins/repo/hello-world" failed (2: No such file or directory), client: 10.244.0.19, server: localhost, request: "GET /plugins/repo/hello-world HTTP/1.1", host: "plugin-repo.plugin-demo.svc.cluster.local"

So the cli is expecting /plugins/repo/hello-world to return a JSON file like the one below, we will create the the file but naming it index.html for the same reason as previous.

In the Nginx pod:

root@plugin-repo:/usr/share/nginx/html/plugins# mkdir repo/hello-world
root@plugin-repo:/usr/share/nginx/html/plugins# cat > repo/hello-world/index.html
{
  "versions": [
    {
      "arch": {
        "any": {}
      },
      "version": "1.0.0"
    }
  ]
}

Let’s try again from Grafana pod:

grafana-d55f9bc77-6k5kl:/usr/share/grafana$ grafana cli --repo http://plugin-repo.plugin-demo.svc.cluster.local/plugins plugins list-versions hello-world
1.0.0

Ok that works! now let’s try installing it!

grafana-d55f9bc77-6k5kl:/usr/share/grafana$ grafana cli --repo http://plugin-repo.plugin-demo.svc.cluster.local/plugins plugins install hello-world 1.0.0
Error: ✗ failed to download plugin archive: 404: <html>
<head><title>404 Not Found</title></head>

Looking at Nginx log:

10.244.0.19 - - [23/Sep/2023:10:46:53 +0000] "GET /plugins/hello-world/versions/1.0.0/download HTTP/1.1" 404 153 "-" "grafana 10.1.1" "-"
2023/09/23 10:46:53 [error] 29#29: *7 open() "/usr/share/nginx/html/plugins/hello-world/versions/1.0.0/download" failed (2: No such file or directory), client: 10.244.0.19, server: localhost, request: "GET /plugins/hello-world/versions/1.0.0/download HTTP/1.1", host: "plugin-repo.plugin-demo.svc.cluster.local"

Let’s setup the plugin directory in Nginx pod:

root@plugin-repo:/usr/share/nginx/html/plugins# mkdir hello-world
root@plugin-repo:/usr/share/nginx/html/plugins# mkdir hello-world/versions
root@plugin-repo:/usr/share/nginx/html/plugins# mkdir hello-world/versions/1.0.0

Copy the plugin into the pod as a file named download without any extension, yes you read that right!

You can use the sample plugin that I have created here - need to zip it yourselves and copy it to the Nginx pod like so:

$ cd passionatedevelopment-helloworld-app
$ npm run build # this will create /dist folder which contains the plugin
$ zip -r helloworld-1.0.0.zip  dist/
$ kubectl cp hellworld-1.0.0.zip plugin-repo:/usr/share/nginx/html/plugins/hello-world/versions/1.0.0/download -n plugin-demo

This is the /download being a zip file is the bit that still bothers me, seems really odd convention. But if you read the code - it does expect /download to be the zip file. 🤷🏻‍♂️

func (m *Manager) downloadURL(pluginID, version string) string {
	return fmt.Sprintf("%s/%s/versions/%s/download", m.baseURL, pluginID, version)
}

But anyway let’s push on - let’s install the plugin:

grafana-d55f9bc77-6k5kl:/usr/share/grafana$ grafana cli --repo http://plugin-repo.plugin-demo.svc.cluster.local/plugins plugins install hello-world 1.0.0
✔ Downloaded and extracted passionatedevelopment-helloworld-app v1.0.0 zip successfully to /var/lib/grafana/plugins/hello-world

Please restart Grafana after installing or removing plugins. Refer to Grafana documentation for instructions if necessary.

So to conclude if you want to serve your private plugin from a repo this is how you’d structure it:

root@plugin-repo:/usr/share/nginx/html# tree
.
|-- 50x.html
|-- index.html
`-- plugins
    |-- hello-world
    |   `-- versions
    |       `-- 1.0.0
    |           `-- download # this a zip file 🤷🏻‍♂️
    `-- repo
        |-- hello-world
        |   `-- index.html # this is a JSON file 🤷🏻‍♂️
        `-- index.html # this is a JSON file 🤷🏻‍♂️

As I go through the grafana code and docs https://grafana.com/docs/grafana/latest/cli/#override-default-plugin-zip-url https://github.com/grafana/grafana/blob/40c12c17bffcec8c3512c597e7f228b296b0c218/pkg/plugins/repo/service.go#L93

grafana cli --pluginUrl http://plugin-repo.plugin-demo.svc.cluster.local/hello-world/versions/1.0.0/hellworld-1.0.0.zip plugins install hello-world 1.0.0
✔ Downloaded and extracted passionatedevelopment-helloworld-app v1.0.0 zip successfully to /var/lib/grafana/plugins/hello-world

Please restart Grafana after installing or removing plugins. Refer to Grafana documentation for instructions if necessary.

k exec -it plugin-repo -- /bin/bash
cd

Install via helm

https://github.com/grafana/helm-charts/blob/main/charts/grafana/values.yaml#L541

  ## You can also use other plugin download URL, as long as they are valid zip files,
  ## and specify the name of the plugin after the semicolon. Like this:
  # - https://grafana.com/api/plugins/marcusolsson-json-datasource/versions/1.3.2/download;marcusolsson-json-datasource

plugins:
  - http://plugin-repo.plugin-demo.svc.cluster.local/hello-world/versions/1.0.0/hellworld-1.0.0.zip;my-little-pony

grafana.ini:
  plugins:
    allow_loading_unsigned_plugins: passionatedevelopment-helloworld-app