Nico Lippmann

Nowadays, networks, systems and automation simply need monitoring. It should protect against failures, alert the administrators or make general misbehavior more recognizable. There are many applications for monitoring and Grafana is one of the most widely used - and it's free! If you search on shodan.io for active Grafana instances: There are about 95.000 online. 

And I would argue that almost every major company in the S&P500 is running a Grafana instance (whether knowingly or not - unflattering self-promotion for a project where I also participate, and which can help to evaluate your attack surface: Fischfang.

But what many developers and operators forget is that if someone has access to Grafana, there can occur many problems. Since I see these problems more and more often in systems, I would like to explain to you what kind of problems can occur with Grafana and what you should think about before exposing your monitoring data with Grafana. 

The Nightmare of Default Credentials – a low hanging fruit 

Many will not believe it, but on quite few Grafana installations the default combination of username "admin” and password “admin” is still active. Also, if the instance is actively used or other users have already been created. The admin is even prompted to change the password when he logs in with the default password - but you can skip that for convenience … or laziness. So just try it when you see a Grafana interface during a penetration test. I still don't know why such a modern project like Grafana comes with default credentials. 

Standard credentials are also interesting for other applications. It is always worth a shot during a pentest to simply search for it on Google and try it out.

If you want to be even more secure, you can of course add further restrictions such as filtering on IP addresses and only allowing certain network areas or adding certificate authentication on a reverse proxy. 

You should also consider turning off registration and anonymous access to your Grafana instance. I will explain later why it is important to give as little access as possible. 

How Grafana access your data – an introduction to datasources

Grafana supports a variety of database products that can be used to retrieve monitoring data. Just to name a few famous representatives: Prometheus, Elasticsearch, MySQL, InfluxDB and even offers the possibility through the open source approach that more can be added at any time. The term under which Grafana summarizes the products is called “datasources”.  

Grafana also tries to protect these datasources against attacks - but this does not always work. For example, it is not necessary to expose a MySQL database to the outside world because Grafana can act as a proxy here. This way a datasource is protected from direct access of an attacker but indirectly the attacker can simply go the way over Grafana and attack that way with other methods. And because of the already configured user credentials inside the Grafana settings he doesn’t even need to bruteforce them. 

Grafana cannot distinguish whether it is a valid query of monitoring data or whether it is an attack. There are rudimentary restrictions, e.g. in the MySQL datasource. Statements like "show grants;", “session_user()”, “current_user()” (and other similar functions) are blocked, but this is not sufficient to thwart an attack. An attacker has simply too many possibilities to bypass these restrictions. Remember: Whitelisting is better than Blacklisting – but not possible for Grafana. 

This is just an example using the MySQL database ... other datasources behave similarly or have other problems. 

What an attacker can do with access to your Grafana instance 

For this scenario, I will assume that either the attacker 

has access to the admin account because Grafana was still using the default password 

was able to take over another account by phishing or something similar 

has anonymous access to the dashboards and visualizations 

and I assume that a datasource (in this case a badly configured MySQL database) is already configured. 

Now the complete host is vulnerable again although it should only store monitoring data inside a MySQL database and present it via Grafana. 

And this is even expected behavior from the way Grafana works. Grafana must have direct access to the data in order to be able to read it and/or to analyze it. But technically the disaster is that the queries are not stored in Grafana itself and that the attacker can control what is executed by his requests. This behavior coupled with the direct access to the datasources leads to big problems. 

It gets even worse when an attacker has anonymous access to the data because a dashboard is included on a public page, for example. 

This scenario now deals only with the MySQL datasource but other scenarios with other datasources can also take place. For example, it is also possible to perform SSRF with Prometheus datasources. If you want to know more about it, there are already public scripts you can have a look at or just contact me and we can exchange ideas. 

Conclusion – is Grafana bad for my security? 

No. Not if configured properly. 

As I had already written, you should be extremely careful about who has access and with which permissions the datasources are connected. If you use the root user of your insecure MySQL database, Grafana can't protect you from anything. But if you have an extra user with very limited permissions (but sufficient), you can access your MySQL database without being too worried. You should also only write monitoring data into your datasource. Do not save in the same database the data for your application itself - especially no sensitive data! 

To prevent even more possible vulnerabilities, you should of course configure your instance accordingly. Change the admin password and check how you can restrict access further to eliminate the low hanging fruits.  

For even more security you should even check if you can put your Grafana instance plus the databases for monitoring in an extra network. With appropriate firewall rules you can effectively prevent further steps from the attacker after compromising your Grafana instance.