How to Set Up This Next-Level Personal Home Dashboard in Your Kitchen
We saw Cameron’s impressive dashboard in a Grafana Experts Facebook group and asked him to break it down for us.
When friends and visitors see my personal home dashboard in my kitchen, they’re normally of the opinion that it’s “really neat” and looks “impressive.” Often that’s what they take away from it. It’s “cool”… and as much as that may be true, that is not its purpose.
What you see on this dashboard is critical information relating to daily operations in my home. Can I get the information elsewhere? Yes, absolutely. But without Grafana, I can’t easily and quickly retrieve this information in a single glance while making breakfast, preparing for work, and wishing my family well.
This dashboard serves the purpose of equipping myself with the best information available now so I can make better and more informed decisions throughout my day. There’s also an element of peace of mind. Sometimes a decision doesn’t have to be made right now, and with the visual confirmation Grafana gives me at a glance, I can go about my life with just a bit less worry.
What This Commentary Includes:
- Where this information comes from and how to extract it
- Why is this information important to me
- How would I otherwise access this information (in the absence of Grafana)
- How much time it would otherwise take me to access this information (in the absence of Grafana)
- What devices and processes I’m using to extract this information
Disclaimer
It’s easy enough to lecture on processes I have experience with or have personally developed, but it’s important to me that I note, as with any complex software suite or customized software deployment, I do not by any means pretend to have comprehensive knowledge on everything related to Grafana, and I perpetually study Grafana and related subject matter in order to improve my understanding.
The dashboard that I’m commenting on today is in a form that has evolved over a period of about 2 years. It is likely to evolve more as I progress in my knowledge, so we may or may not see it in its current form ever again.
A Word on Platform Cross-Compatibility
With the exception of the live camera feeds, everything you see on this dashboard and all of the related components work on both Windows and Unix-based systems.
In the case of the camera feeds, I’m using a software suite (I will discuss this in detail later) that only operates on Windows; however, I am very confident that something similar and just as easily accessible exists for use on other platforms.
My Background
Except for quite a bit of PHP deployment and Web app facilitation using Apache (thanks to Apache’s accessibility and ease of use), I am a Microsoft Windows systems administrator. As a result, all of my Grafana-related deployments are hosted on Windows computers.
I specialize in graphic design, front-end design, Web frameworks such as Wordpress and Joomla, and I’ve taken traditional training in Cybersecurity and Windows Server administration. I also specialize in business automation, which is why Grafana has become so critical in my day-to-day processes both at home and on the job.
I have personally overseen the development and implementation of two major proprietary Web applications (both of them ASP.Net), and I am currently overseeing a third even larger and more complex web app as we speak.
Insofar as Grafana is concerned, I share everything I learn and develop. Partly I do this because I’m proud of what I’ve managed to accomplish, but also because it pleases me to see others go down a similar path and accomplish something that hasn’t been done before.
You can see my (somewhat outdated) resume here.
And you can see the same resume with a Harry Potter twist here.
The Dashboard
This commentary will be using the following dashboard as reference:
Where This Information Comes From
Let’s get down to the specific sources of my information. Please note that the panel references numbers on the dashboard image above will be necessary for visually identifying exactly which panels I’m talking about.
Telegraf
Panels 4, 6 and 8
Offered for free from Influx Data, Telegraf is very likely the first metrics collector you’ll come into contact with after you set out to create a Grafana dashboard.
It’s designed to collect system metrics like CPU usage, RAM, hard drive space and various I/O states from the system on which it’s running and report those metrics to InfluxDB, which can then be displayed by Grafana.
Workflow:
Telegraf ⇨ InfluxDB ⇨ Grafana
You can view some fairly comprehensive Telegraf installation and usage tutorials [here] (https://g-ee.alexsguardian.net/books/telegraf).
HTML Iframes
Panels 1, 2, 5, 10 and 18
Workflow:
Iframe Source ⇨ Grafana
These panels are where you would embed content from an external source. There are no InfluxDB queries to make.
Grafana version 6.4 does not allow iframes to be compiled by default due to security exploits that iframe content can use for ill gain. You will have to disable HTML sanitization in the Grafana settings file in order for HTML iframes to work.
To get these iframes to display correctly on my dashboard, I had to create a little bit of custom CSS in order to either shrink or enlarge the iframe content via the Zoom and Transform CSS classes. The following is an example of the CSS I used on the Weather panel in order to get it to fit properly on my dashboard (panel #10):
<style>
#embedweather {
position: relative;
left: 30px;
top: 100px;
zoom: 0.71;
-moz-transform: scale(0.60);
-moz-transform-origin: 0 0;
-o-transform: scale(0.17);
-o-transform-origin: 0 0;
-webkit-transform: scale(0.65);
-webkit-transform-origin: 0 0;
width: 200px;
</style>In the CSS example above, I created an ID for the Weather Iframe and then I am assigning CSS classes on that ID and configuring those classes to reduce or increase the zoom and transform levels on the iframe.
See our iframe tutorial here.
Varken for Plex Metrics
Panels 3, 7, 13 and 14
These panels are for those of you who use Plex. Plex allows you to rip your DVDs and Bluray movies onto a computer to be stored digitally, and from there can be shared to members of your household via a Web interface very much like Netflix.
Workflow:
Plex ⇨ Tautulli ⇨ Varken ⇨ InfluxDB ⇨ Grafana
With the necessity for Plex, Tautulli, Varken, Grafana and InfluxDB, this is without a doubt the most complex series of panels on this dashboard. It’s a lot of stuff, but it’s amazing when everything clicks and it all starts working properly.
See the various install tutorials here:
Installing Plex (This is a fairly easy setup.)
Installing Tautulli (This is a bit more complex.)
Installing Varken (There are no Windows tutorials yet – I’m working on that. :) )
Varken comes with a really nice default dashboard, but I didn’t need all of the panels on it, plus I wanted to integrate Varken into my custom dashboard. In order to accomplish this, you must copy the queries from the Varken dashboard to your custom dashboard, or you can follow this tutorial to set them up manually.
Router Up/Down Metrics
Panels 8.1 and 9
Workflow:
Router (DD-WRT) ⇨ Prometheus ⇨ InfluxDB ⇨ Grafana
These panels graph the Up/Down metrics and uptime for my router via SNMP (Simple Network Management Protocol) output. This was a difficult item to get working properly because I found out halfway through my research that in order to get my router to output SNMP metrics, I had to load third-party firmware rather than use the router’s native firmware.
This was frustrating because as far as I knew – and this has regularly been reinforced since I work with telecommunication equipment quite a bit – all routers and switches should have SNMP output by default.
As it turns out, this is not the case for many home routers. Before I would be able to pull SNMP metrics from my router, I needed to find a way to replace the router’s native firmware with alternate firmware that would give me the SNMP output option that I needed.
In order to accomplish this, I used the DD-WRT firmware which seemed to be the most prevalent and mature brand of third-party firmware.
NOTE: It is unlikely that there is a DD-WRT distribution available for leased routers (that is, routers that you have not purchased and are renting from your ISP).
In order to install DD-WRT, you must check to see if your router is compatible with DD-WRT, and you can do that here. (Be sure not to load the wrong firmware because it can destroy your router.)
Once I loaded and configured the new firmware and I confirmed that SNMP metrics were being output from the router by using MIB Walker from SolarWinds, I was then ready to set up some sort of data collector to input the metrics into InfluxDB. I ended up choosing Prometheus with the Prometheus SNMP Connector.
I do not yet have a Prometheus or Prometheus SNMP Connector installation tutorial for Windows, although Alexander H. has created a Prometheus install tutorial for Docker here.
Live HTTP Camera Stream
Panel 18
Using an Annke 8-port DVR, I was able to convert the output RTSP stream into an HTTP stream that can be embedded in a Web page.
Workflow:
Annke DVR ⇨ ContaCam ⇨ Iframe ⇨ Grafana
Please note that not all DVRs have RTSP streams. In my case I used an inexpensive Annke 8-port DVR (From Amazon.com) that has RTSP stream output by default. I have tried to capture RTSP streams from other DVRs only to find out they do not output RTSP streams.
It’s not impossible to embed an RTSP stream into a Web browser, but due to the extremely high throughput it is difficult, so I decided to search for a way to convert the RTSP stream into an HTTP stream first. To do this, I ended up using ContaCam, which is installed on my Windows server inside my network and is totally free. (Although they do ask for donations, and I did give a couple dollars because ContaCam is pretty awesome.)
Download ContaCam here.
Once I pulled the DVR’s RTSP stream into ContaCam, I was able to turn on ContaCam’s built-in Web server in order to serve off the now converted HTTP video streams. Note that enabling the ContaCam Web server still requires a PHP compiler like Apache – or in my case I’m using Xampp to host the PHP files, and the ContaCam Web files will have to be within Apache’s “htdocs” folder. When this was completed, I embedded each individual HTTP video stream into iframes within Grafana.
The video quality stays about the same, but the framerate is controlled through ContaCam. I reduced the framerate to 1 FPS in order to preserve my internal network’s bandwidth, and that reduced my throughput from ContaCam from 10 megabytes per second to about 1 megabyte per second.
See my full ContaCam RTSP to HTTP conversion tutorial here.
NOTE: ContaCam is NOT Unix/Linux compatible. I found a few Unix/Linux-based alternatives with a quick Google search, and I am confident one or more of these should be able to do the same thing.
Plex Movie Poster Display
Panel 15
Plex Movie Poster Display is a third-party script developed by “MattShack” and is available for download on GitHub.
Workflow:
Plex ⇨ PMPD ⇨ Iframe ⇨ Grafana
This neat script from Matt was design to output “Recently Added” and “Now Playing” information with full movie posters onto a big-screen TV. It has a web-based administrative backend and frontend, which I was able to manipulate into an iframe within Grafana.
You must have some version of Apache running on your server in order for this script to work. I’m using Xampp (I was using full-on Apache but I found that I didn’t need that much horsepower).
Pending Bills and Billing Amounts
Panels 12, 16 and 17
These panels display extracted data from authenticated Web pages. In this case, they show my seven major monthly bills.
Workflow:
Website ⇨ Scraper ⇨ InfluxDB ⇨ Grafana
You can download the AT&T Internet scraper here.
Using Python, I created several Web scrapers that go to each web page, log in, navigate to the “Total Due” page and extract the amount due and the due date. In the case of my power company, not only was I able to extract the total due and the due date, but I was also able to extract other useful information like how many kilowatt hours I used last year at the same time, how many KWh I’ve used so far this billing period and my average daily power cost. (Yes, I know it’s very high, and hence very expensive, too!)
These Web scrapers fire at specified times during the day and automatically insert the data they’ve extracted into InfluxDB, at which point Grafana refreshes and shows the new metrics.
These scripts probably have a limited shelf-life. The web pages they scrape are likely to change their configuration at some point, rendering the scripts unable to scrape the data – at which point I’ll have to update the scripts to work with the website’s new configuration.
I will upload the remainder of these scripts to GitHub as soon as I can.
Other Considerations
Starting Applications on System Boot under Windows
The above processes, scripts, and applications must start automatically when Windows starts. I’d hate to have to go to my server and click each program I need every time the power goes out.
Also, you can configure your server (Windows or Unix/Linux) to power on after power loss through the BIOS.
To start a program when Windows starts, just click Start ⇨ Run ⇨ type Shell:startup and hit Enter – at this point, a folder will appear. Create shortcuts in this folder to the applications you want to start when your computer starts.
Once you’ve done this, click Start ⇨ Run ⇨ type netplwiz and hit Enter, then uncheck the option that says “Users must enter a user name and password to use this computer.”
Please note that this is NOT very secure. Converting your apps into background processes is far better, but a little more complex.
Converting Applications into Windows Background Processes
Converting your apps into background processes solves the above problem, and also creates an environment in which you don’t have to log into your Windows server for your apps to start running.
Use this tutorial to convert your apps and scripts into Windows background services.
Why Is This Information Important to Me?
Most of the information on this dashboard is relevant to me, my household, or my family in some way. But why put so much effort into consolidating this information into a single easily-digested glance when this information can be accessed through my cell phone or desktop PC?
I have several reasons:
This information is all in one place. I don’t have to log into several different web pages to consume this information. All I have to do is turn my head. In fact, if you count how many sources of information I have displayed here I’d literally have to log into eight different Web pages and four different computers, and that’s not even taking into account the fact that sometimes I just don’t want to go get relevant information right now. I want that information to come to me.
Security. We very much like the peace of mind that seeing our security cameras give us. In fact, once we complete our long-standing plan of purchasing a house, I plan on installing a Grafana dashboard in each of the major rooms that we occupy – and I won’t have to run BNC cables all over the place in order to see our security camera feeds.
I am more informed. Before this dashboard, I never walked out of my house to face the day equipped with this information. Sometimes I’d look up the weather on my phone before I left, but that was rare, and hence I was not ready for the day. In other cases I’ll be reminded that I need to go pay the rent, or that my wedding is coming up soon, or that my media server is running out of hard drive space – all items that are important. To those of you with excellent memories, I say bravo! But my memory is not what it used to be.
How Much Time Does This Dashboard Actually Save?
I will have to do some estimation on this. I’ll start first with estimating how much time I actually spend “looking” at the display per day – let’s say 10 minutes – and how many times per day I look at it – let’s say 10 times. That’s 1 minute per glance, 70 minutes per week.
Second, let’s estimate how long it would take me to manually glean the same information from the Internet.
There is information here from eight different Web pages, all of which are behind a login form. Let’s say it takes me 20 seconds to log into a Web page, another 5 seconds to navigate to the information I want, and another 10 seconds to read that information. That’s 35 seconds per Web page, times 8 Web pages = 4.6 minutes to visit all eight Web pages, and that’s assuming I don’t get distracted.
And to log into each of my four PCs at home to check the CPU usage, RAM usage, and disk space, I’d call that about 1.5 minutes apiece which amounts to 6 minutes.
All told, I have now spent 10.6 minutes manually reviewing this information once. Multiply that by ten (the number of times I have estimated that I actually look at this dashboard per day) and I am now up to 1.76 hours per day and 17.6 hours per week.
Obviously I would never actually spend this much time manually looking at these metrics. I would probably just continue without it, but I don’t want to. I want this information now, and I don’t want to spend a lifetime getting it.
Professional Use
In addition to the home dashboard that I’ve described to you above, I have heavily integrated Grafana into my professional work. I have a 9-5 job where I am continually understaffed and overworked, and without the automation and data visualization that is made possible by Grafana, there is absolutely no way I would be able to do my job efficiently. It would be practically impossible unless I worked 100+ hours per week. Allow me to explain why.
I administer several hundred digital signage installations around the state in which I live. Most of these screens are local, across town or relatively close – basically within driving distance. Many of them are in the next county over, maybe a 30 minute drive,… and some of them are three hundred miles away.
Using Grafana, a proprietary VPN and some clever automation and four big-screen LCD televisions, for the last two years I have been able to see all of the digital signage installations live. Like, right now. I can tell when a screen is down, I can tell when a screen has the wrong content on it, I can tell when the remote computer is on, functioning properly, when the TV is on or off, or when it’s broken down and needs a repairman dispatched. And I can do all of this while sitting at my desk, without the need to drive to every location every day.
Conclusion
Extracting, manipulating and visualizing data is a fascinating endeavor. Whoever said “knowledge is power” had it right. And I am now left wondering what other metrics I could potentially visualize with Grafana that would benefit my household right now.
