Project Coldplay by Erwin de Keijzer, DevOps Engineer has guided through project phase of building software solution for Adding Music and Announcements to his Home.
While Prometheus has become the most prominent open source monitoring tool in the world, not let alone OpenTelemetry, Grafana and the recent project updates by CNCF, the onboarding into monitoring and observance tools has been main focus on KubeCon and CloudNativeCon 2023.
In this year KubeCon + CloudNativeCon Europe 2023 had more than 10000 members from which 58% has their first time in this global cloud-native conference.
Attention to proof-of-concept and sandbox projects often prove that decision about project final outcome depends not only on software architecture but the iterations of testing in project development phase.
Erwin de Keijzer, DevOps Engineer has been presenting the use of observance and monitoring tools to learn about Prometheus and discover the abnormality detection with PromQL queries.
Project Coldplay- using CNCF Projects for Adding Music and Announcements to My Home Elevator.
Coldplay objectives:
-
know where the elevator is;=
-
add the music when elevator is moving
-
play announcements when elevator stops at a floor
-
don’t break the elevator;
-
don’t break the elevator internal electronics
Alerts:
-
Elevator stuck between the floors; elevator moves too fast
-
elevator moves too slowly
-
elevator out of bounds (< 0 cm or > 550cm)
Attempt #1
ultrasonic sensor attached to raspberry pi, where the python code produces measurements. Distance seems reliable, but the horizontal support beams in the elevator shaft produce echoes for the sound waves, and the elevator shaft is too high (more than 2m), a vertical scaling issue.
Attempt #2
TF-Luna LiDAR range sensor, in range up to 8m, accuracy of 2% of a measure distance, from 8m it would be 16cm it could deviate, frame rate up to 250Hz, and with a multiple ways of a connection, e.g. trigger based mechanism.
Issue with Jitter for the elevator height that has to do with the temperature of the chip itself on the sensor (Unit: 0.01 Celsius; Timestamp- per TF Luna I2C specs)
Monitoring with Prometheus and Grafana
Software Architecture:
1st attempt with microservices approach– on raspberry pi there are two components- measure and speaker, that does measurements and speaks the announcements. The Scientist is the Coldplay song that is a controller of the project- takes all the measurements and decisions about playing music, updates the user interface, and sends data to prometheus. Ui, prom writer (prometheus), and the Scientist use the subnet. It is based on Tailscale, which allows building an overlay mesh network between all your devices and building connections between them on the Tailnet.
2nd attempt with monolith architecture– project called Paradise written in Go, connected with Prometheus and Grafana to read the results.
However after running the project for a few hours it outputs noise that returns ground-below results. Reboot worked for a few seconds, and adapting the time between measurements based on the chip temperature didn’t solve the problem.
3rd attempt was an architectural redesign with rust that takes the measurements and writes them on NATS, then a service called Yellow articulates the measurements to the speaker, updates the user interface and sends the data to Prometheus.
Latency of the Service
The hardware on top of the elevator shaft has the bluetooth speaker but connects with an audio cable that has no latency to make sure the time between elevator starting- playing the music is short. Hardware box is attached. with magnets inside the elevator.
To connect with Raspbery pi SSH into the correct Raspberry Pi for the remote access. Software is built with Go HTML web template and JavaScript, setting the websocket connection from the server to client via event stream incoming message.
Inventor Major General George Owen Squier, credited with inventing telephone carrier multiplexing in 1910, developed the original technical basis for Muzak, and several US patents in the 1920s related to transmission and distribution of signals over electrical lines.
While Muzak had initially produced tens of thousands of original artist recordings by the top performers of the late 1930s and 1940s, their new strategy required a different sound.
Architectural software redesign:
Decision about high-frame rate detection – Ultrasonic Sensor vs a single-point ranging LiDAR with I2C and TF Luna;
Resolving connections- Ssh’ing into the wrong host when testing audio;
Hardware barriers- raspberry pi when unplug the network switch;
Having 2 copies of the measuring software running at the same time;
Bad soldering job on the connectors of th TF Luna;
Cross compiling Rust and Go from M1 Mac to Linux Arm
Reference to the project: https://github.com/gnur/coldplay