Friday, August 11, 2017

DIY home automation, new generation

I've had my DIY home automation system for controlling outlets and reading sensor data running for about two years now. The system has been working fine and I haven't had any need to touch the code since I added the sensor reading to it, until a few months back.

Need for new functionality


Few months ago I got a new IoT toy for a lend from a friend until I'd get my own toys, a ruuvitag sensor beacon. Ever since I found the ruuvitag for the first time from kickstarter I had the idea of getting a bunch of ruuvitags and adding their weather station sensor readings as part of my home automation system.

The original home automation backend included only tellstick compatible devices and was written in Python, and in my mind it was kind of a hack. The ruuvitag beacons communicate via BLE i.e. Bluetooth Low Energy and that meant that I needed to add functionality to read the beacon data via bluetooth.

I found a ruuvitag Python library and initially thought that I'd just use that and extend the existing backend. As I started looking more into the ruuvitag and BLE beacon possibilities I came accross Node.js implementation that would be able to read the beacon data. As I started looking deeper and extended my search within Node.js libraries I also found a telldus library.

I've done some work with Node.js and even though it's not my first go-to choise it seemed that with it I could simplify my existing backend a lot and add the new functionality quite easily.

Backend rewrite


I'm not going to get in details of the backend code as it's basically a REST API that reads sensor data and controls outlets just like the original implementation but here's a link to the backend code and just a few notes about the backend.

The Node.js telldus library uses the native telldus library and is compiled with node-gyp so it needs to compiled on same architecture where the backend is going to be running. The compile time on my original Raspberry Pi was quite long.

The beacon library uses bluetooth library and it has the same restriction as the telldus library.

The readme file contains more information and instructions in case you're interested.

Refresing the UI


The UI of the original home automation system was a vanilla JavaScript experiment and I thought that I should also update that as I already rewrote the backend. This time I chose Vue.js as it's something on the rise and I haven't used it before. Just like with the backend I'm not getting in the details of the code but here's a link to the code and a screenshot.



Wait, what, no tests!


I made a conscious decision not to write any unit or integration tests for this project as without all the hardware (tellstick duo, controllable outlets, bluetooth, ruuvitag) I'd have to mock them and I'm not a big fan of mocks.

Sure i could have written a unit test for parsing the sensor data from the beacon URL but it just didn't seem worth the time and effort as I have manually tested the implementation for hundreds of times while developing.

Integration testing has also been manual while developing and continues to be manual as we use this system daily so I didn't see the need for automating that.

Final thoughts


The rewritten backend of the home automation system seems to be working as reliably as the first generation and I'm really pleased how much cleaner code I got with Node.js.

Vue.js seems to be a viable option at least for small projects like this one.

I already have some ideas how to do some actual automation and analytics based on the sensor data but that's another story.

P.S. I wish my ruuvitags had a light sensor as that would extend the automation possibilities even more.

Tuesday, February 28, 2017

Why am I a consultant

Have you ever wondered what do software consultants exactly do? I'll share my view of how I ended up as a software consultant, what I have gained and what do I do as a consultant.

My brief background


I started my career as a programmer in the early years of 2000's as part of my studies and via summer jobs. Back then I was working with web sites and doing... well what ever we could do back then. Few years later I enrolled to university and at some point started working as a programmer to have some side income and work experience. After I graduated I continued working for the same employer I had been been working for the past few years.

After four years of working with the same employer and on pretty much the same applications and technologies I started to notice that I wasn't learning new skills or technologies as much and fast as I wanted to.

Time for a change


I started looking for job openings and had a few interviews with different companies. I chose to work for a company that offered me a job with a in-premises project where I'd be joining a small team who'd been working on the project for a while. This seemed like a great opportunity to work on something new and learn new skills. I knew I'd be working for a company that did subcontracting on the form of software projects for various clients. What I didn't fully realize then was that that's exactly what consultancies in Finland actually do. So by a sheer chance I became a consultant. I worked for the company for a few years with various clients and really started to like how consulting in the software business works.

I've been working as a consultant ever since and the few times I've changed jobs I haven't seriously considered working in any other position.

I'm a consultant - what I have gained from it


I've thought about it more than few times what it means to me to be a consultant and what it has given me compared to my previous jobs.

Networking


The network I have gained during my years as a consultant is something I could have never achieved if I'd been working in product development company or in a company that has a internal software development unit. The professional network that I have gained as a consultant can be divided to different sections.

Clients

The network of different clients in various business domains. Via this network I know a lot of people from different business fields and I have gained working experience in various domains.

External contacts

External contacts consists of clients and their own software developers and it's common to be working side by side with consultants from other companies.

Internal contacts

Internal contacts are all my great colleagues from all the companies I have worked in.

Learning

By working with various clients in different business domains I have gained a lot of knowledge on different business fields. I can see commonalities and differences on how different clients and business fields do things and I can apply that knowledge and compare various technologies and work habits and processes to each other.

I believe I can learn from every client and from every colleague, internal or external, and I can honestly say that I have learned something while working with each of my clients and colleagues.

Learning as a programmer doesn't always mean learning new technical skills, frameworks or languages but also about soft skills, project habits and management or how to organize work and how to not do things.

What do I do as a consultant


Main area of my expertise is programming and that's the role I have when I start working with a new client.

Depending on the client and their needs I have been doing a lot more than just programming. With some clients I have been doing a lot of investigation tasks related to integrations between different systems or investigations on how some legacy systems work.

At some rare occasions my work has also included some work management in the form of introducing or trying to improve existing methods on how a team could organize it's own work better or make the teams work more transparent.

Besides the work I do for clients I also participate in some internal work. I have been doing recruitment interviews and when needed I give support to our sales people when they need assistance in technical matters and sometimes I take part in meeting our new or potential clients.

It's not all rainbows and unicorns


Even though I see a lot of positive aspects with consulting there are some not so great features with consulting that every consultant will face at some point.

Clients don't always know what they want but they want it done yesterday. The client might want one thing on one day and the opposite thing the next day. In worst case the client blames consultants of doing wrong things.

You don't fit in the team. On the outside everything might look fine but working as a consultant means that you are working with other people and some times people just don't get along with each other.

The job isn't what it was supposed to be. A new gig might be sold to you with buzzwords like new technologies, agile teams, up to date CI/CD pipelines but in reality you end up working with legacy systems or abandoned code bases without tests or maybe the agile team means that objectives yesterday were totally different from what they are today.


Why do I do it


To me all the positive outweighs the downsides of working as a consultant. When things get really bad I have the possibility to change projects/clients, it may take some time but it's always possible.

To me the best part of working as a consultant is the variation of clients and projects. I personally don't like working too long (read: years) on a single project even though some consultants do prefer them.

I get to work on various projects on different business domains and constantly learn new skills without the stress of changing jobs.

Tuesday, February 2, 2016

DIY home automation v2

Previously I set up a simple home automation system for controlling outlets at home. So far I've used the system for controlling various lights at our home. I've been satisfied with oulet controlling via browser but I wanted something more.

Objective


I had a idea of temperature and humidity monitoring via Raspberry Pi. I was already browsing for electronics components needed for building a system with sensors but buying single components seemed a bit expensive and I would also have to actually connect all the components to RPi and write the code to read the sensor data.

Mainly because of the price the components I started looking for a pre-built system with the preferred sensors. Almost immediately I came across Proove's sensor that has indoor and outdoor temperature sensors and a indoor humidity sensor. Best part of the pre-built system is that it's already compatible with TellStick system that I used for controlling outlets.

I decided to buy the pre-built sensor system. I just needed to figure out how to read the sensor values and display them on the same UI that I built for controlling outlets.

Sensor data


The sensor system is nice in the way that it automatically sends it's sensor data in some sort of broadcast and the TellStick Duo automatically receives all the data. Something's also flaky with the sensor system because it sends sensor data from a bunch of sensors. Most of the sensor data is just invalid data and it's up to the user to determine which sensors are sending valid data.

tdtool --list-sensors

Reading and decyphering sensor data


Decyphering which sensors were sending valid data was a manual process of viewing the sensor data and comparing that to the values displayed on the sensor system's lcd screen. I managed to figure out that my system sends outdoor temperature with sensor id 135 and indoor temperature and humidity with sensor id 136.

Reading sensor data via python


The telldus-core library that's used to control the outlets also provides functionality to read sensor data. It was just a matter of figuring out how to read the data and I found a good example from github.

Result


As previously the REST API is written with Python and it uses the C-library to communicate with TellStick Duo and the UI functionality is written with JavaScript. Reading the sensor data and viewing it in the UI is implemented in the same code base as the outlet controller.

I just added a new endpoint /sensors that returns sensor data for the two sensors and added some UI functionality to read and view the data.

Sensor data above outlet control


Few things to know about the code

 

Because the sensor system sends invalid data with bogus id's I extracted the sensors id's to their own module sensorconf.py that defines the id's for indoor and outdoor sensors 

The UI has names for the different sensor values but I didn't separate them from the logic but instead hard coded them.

Sunday, October 25, 2015

DIY home automation v1

For years I've been interested in home automation. I've had remote controllable outlets from a few different manufacturers but I've never been quite satisfied with just the remote. What if I could control my outlets within my local network from any device, now that's something I wanted to have.

Controlling outlets from computer

 

A few years back I bought a three pack of remote controllable nexa outlets. A while ago I discovered that another company manufactured a control unit that's plugged in to a USB port and best of all they provided linux software for it.

To make full use of these I'd need a computer that's always on and that's where I could make use of Raspberry Pi.

Setting up outlets


I had already set up my outlets with the remote that came in the retail pack just follow the manufacturer instructions.
It might be possible to configure the outlets completely via the software at least for some brands but I haven't tried it so I can't be sure.

Required software


To make use of the Tellstick Duo control unit Tellstick provides software and easy to follow instructions. Important thing here is to install both applications telldus-core and tellduscenter.

The telldus-core is the software that does the actual controlling and tellduscenter provides a GUI that I found necessary to configure the outlets for telldus-core.

Pairing outlets and software


To pair the outlets with tellstick software I used the tellduscenter's outlet scanning option. With this option my outlets respond to the remote and the tellstick duo controller unit.

Not good enough


With this setup I can control the outlets from a computer but only from the computer that has the tellstick duo connected to it. I want to be able to control the outlets from anywhere in my house.

Outlets control from network


To control the outlets from any device in my home network I needed a way to share the control unit to my home network.

My first thought was that I could create I library that would execute the telldus software commands and I could then wrap that to a REST service and build a simple HTML and JavaScript UI that would in turn communicate with the REST service.

After the first idea I figured that someone must have already solved this problem in some way and I started to look for solutions that others had created and then I came across with remotestick-server that did exactly what I wanted with the exception that it produced XML.

Modify the existing software


I created a fork of the remotestick-server and modified the code a bit. I removed some options that I didn't need and modified the software so that it produced JSON instead of XML.

As I wanted a simple UI that I could use from any device I also added some HTML, CSS and JavaScript to the same codebase as it could be served via the same software so no need to complicate things and create a separate application for the UI.

TL;DR;


Now I have a partly DIY home automation for outlets with a simple UI to use from any device in my home network.
Next step is to add more outlets and maybe some scheduled automation.


Demo video

Turning lights on and off

Tuesday, August 11, 2015

Three for the price of one

Few weeks ago in the middle of my summer vacation I had a sudden motivation spike when I was browsing my backlog/notepad (analog i.e. paper & pen) where I write down ideas or problems I'd like solve or try out at some point. Two items in the backlog caught my eye and those two produced a third idea in the form of this blog post.

Backlog items


On my backlog I had two separate items that caught my interest.

First item on backlog was a utility library for validating Finnish social security numbers (SSN, HeTu, henkilötunnus) and business identifications (y-tunnus). Both of these have predefined rules on how they are formed and I have implemented utilities for these same validations a few times in the past.
I have discussed about the need for these utilities many times and last time just before my summer vacation with my colleagues because these exact utilities don't exist in the public repositories or if they do we haven't found them and many of us had implemented these utilities several times.

Secondly I had a note that I should try to deploy a library to public Maven repository. This seemed like a idea that I really should and wanted to try out. I just didn't have a suitable library that I could apply to this but the first item provided a solution for that.

The identification validation library


I decided to implement the library with Java 8 for a few reasons. I work a lot with JVM languages so JVM seemed like a safe bet. Java is popular, people who work with any JVM language can use the same library. Despite of what JVM language one uses they can most likely understand Java and my code.

I'm not going to go through the simple code here but provide a link in the end of post where you can review the code yourself.

Publishing the library to public maven repository


I tried first a search with google and ended up having dozens of instructions on how to deploy to Sonatype OSS repository.

After a quick browse through the links I ended up following this blog post http://datumedge.blogspot.fi/2012/05/publishing-from-github-to-maven-central.html and Sonatype documentation http://central.sonatype.org/pages/ossrh-guide.html and from there to Maven specific documentation http://central.sonatype.org/pages/apache-maven.html.

In the end my pom.xml ended up looking a bit different than on the selected blog post but with it I can just issue a maven command mvn clean deploy and my library will be deployed to Sonatype's OSS repository.

Things to notice


Browsing through the documentation and trying to get the library published brought up a few things to remember.
- Make sure you have Sonatype JIRA credentials
- Create a new project ticket
 - Use a domain that you own or come up with a alternative solution

Creating the JIRA credentials isn't a big deal one just has to register to the Sonatype JIRA and verify that you have the same credentials on you Maven settings.xml.

Creating the project ticket is easy as the page http://central.sonatype.org/pages/ossrh-guide.html#create-a-ticket-with-sonatype has a direct link for that.
If you don't own a domain you can do as I did, use a maven group id of your github account e.g. com.github.jorilytter. Now that I have write permissions to my group id I can publish my libraries under all subgroups of that.

The code and the library


The code is available via my github account at https://github.com/jorilytter/finnish-identification-utils and the readme contains instructions on how to add the library as a dependency for your own project.

Final thoughts


Now that I have implemented the library and it's published to a public repository I hope that I can and remember to use it in future projects if needed and hopefully other developers find it and use it.

Now that I know how and have permissions to publish libraries to Sonatype's public Maven repository if I come up with ideas of other common libraries that could be useful I'll be publishing them.

Monday, May 25, 2015

Verifying your web applications works

Recently I joined a project that was built on node.js and views were generated via jade templates. The node.js backend had a few very basic tests in the form of unit tests and frontend didn't have any tests. One of my tasks was to improve the test coverage. Following is a short description of the problem and a example solution how I resolved it.

Unit testing


I started by going through the existing tests and refactoring those so that they worked as expected and added tests for the functionality I had implemented previously. These tests were run via mocha.js and utilized chai.js and supertest.

These were very mainly unit tests except for the one's that utilized supertest which tested that the web application responded correctly to very basic http requests to routes like / or /login.

I want more


In a previous project that I worked in we had mocha tests that were run in a browser so that it tested the running application within a iframe via mocha. I always liked this approach as it was similar to what selenium webdriver does but much faster.

I considered using webdriver for testing the application from end-to-end but for some reason executing webdriver tests has always been slow and as I had seen a faster running solution I rejected webdriver very early and started investigating on how to implement in-browser end-to-end tests.

After half a day of searching I didn't find any sufficient and simple enough instructions on how to implement this kind of test setup but I did run into lots of headless test solutions and from there I found one promising library that could also execute tests via real browsers...

End-to-end testing via real browser


I came across a library that has been around for a few years and has a very basic testing functionality implemented and can run tests via phantom.js or Google Chrome: dalek.js

I gave it a test run against the application we were developing and I was satisfied on how easy it was to setup and run the tests. Dalek.js provides a basic test functionality and it provides a functionality so that one can execute custom javascript functions within tests.

Example test


I createad a simple example test that opens up GitHub front page, submits search form with text dalekjs and verifies the result. Let's go through it line by line:

module.exports = {
  'GitHub example testing': function (test) {
  test
    .open('https://github.com')
    .waitForElement('input[name=q]')
    .assert.title().is('GitHub · Build software better, together.', 'GitHub has the correct page title!')
    .type('input[name=q]', 'dalekjs')
    .submit('.js-site-search-form')
    .assert.text('.sort-bar > h3:nth-child(2)', 'We\'ve found 53 repository results')
    .done();
  }
};

Rows 1-3 setup the test with a given test name on row 2.
Row 4 the test opens up the given url.
Row 5 waits that the github search box is in the dom.
Row 6 asserts that the page has correct title.
Row 7 inputs a search string to the search box.
Row 8 submits the search form.
Row 9 asserts that the search results header is correct.
Row 10 ends the test.

This exactly same simple example test can be found from my GitHub repositories https://github.com/jorilytter/dalekjs-example with instructions on README of the repository for instructions on how to install the required node dependencies and how to run the test headless with phantom.js or in a Google Chrome browser instance.

Sunday, March 1, 2015

Working on the cloud

Recently I've had the opportunity to work with cloud environments. Cloud environments bring benefits but also some new issues or challenges that have to be taken into account.

Benefits


One of the greatest benefits of cloud environments is the scalability i.e. with a single command one instance can be spawned to two, three or twenty instances that are behind some load balancer service of the cloud provider appearing as a single instance to the outside world.

With the simplicity that cloud environments provide the developers can start deploying software as soon as there's something to deploy and keep deploying all the time. In the best case scenario test and production environments can be up and running since day one. Some services also provide a nice feature of roll backing quickly to previous versions if the latest version has introduced a bug that didn't exist previously.

Challenges


As a developer I've set up various development and testing environments but not that often a production environment. With production environments come things that don't necessarily concern other environments, Database connections, logging services, connections to other services, possible firewall openings between environments, domain names etc.
The second to last, firewall openings, is something that I came across the other day when a service providers system accepts connections through a firewall only from predefined ip addresses. This seemed like a real problem because services in the cloud can have what ever ip address or addresses.

Firewall solution


Services in the cloud run dynamically in what ever ip address so all the connections to the protected server have to be redirected through some proxy that accepts connections from our services and has a static ip address. Thankfully there's ready to buy solutions. Also as a cloud service :)
I guess someone else has also faced this issue in the past.

Benefits come with responsibilities


Depending on the development team and the organisation behind the team the cloud environment might be operated by dedicated personnel or if the team consists only of developers the environments might also fall under developers responsibility.

When developers have all the keys to the kingdom it means that the developers also have to take the responsibility of setting up the environments. Something that was previously done by the administrators.
As a developer I like that I have the possibility to maintain the running environments but I don't like that the full responsibility is put to the developers. This is where I'd like to have dedicated people who are the experts in setting up and maintaining environments and have experience on the subject, more in the way of the devops approach.

Cloud environments to the next level


I've seen from the side lines the next level of cloud services, micro services. I don't know much about working with micro services but it seems based on hearsay that they come with some new benefits and with a whole new set of challenges.

Summary


Deploying services to the cloud comes with benefits and it comes with a set of new challenges. Cloud as a platform solves some issues and it's a step forward. Deploying to cloud or using cloud services still needs people with expertise on servers and services though a bit modified from what they're used to in the self hosted servers and in no circumstances should it be expected that a software developer could do the job of the two.