Impossibly tight deadlines, unrealistic schedules, and constant pressure to develop and release applications on time, while at the same time achieving excellent quality. Sound familiar? Faster time-to-market for embedded applications is indeed a critical consideration for embedded development teams. But how do you develop an approach for delivering applications faster while maintaining quality and security as a priority? In this context, the product engineering team behind the OTA software update manager Mender.io, went through the process of selecting the best programming language for developing both the embedded client and server parts of Mender. In this evaluation process Go, C, and C++ were shortlisted as candidates. Some of the lessons learned from this evaluation process are explained and why Go was ultimately selected to develop Mender’s client embedded application.
While this can be quite subjective, Go can be a very productive language for embedded development. This is especially true when it comes to network programming, which is at some point a part of every connected device or application. Go was created by Google to meet the needs of Google developers, and it was mainly developed to try to fix the explosion of complexity within its ecosystem. So efficient compilation, efficient execution, and ease of programming were the three main principles behind developing Go as not all three were available before in the same mainstream language.
However, we want to stress that Go cannot be considered a replacement for C as there are many places where C is and likely will be needed, such as in the development of real time operating systems or device drivers.
Strict requirements for embedded development
After establishing the architecture, the Mender product engineering team evaluated which language would be the best fit to develop the Mender application. The system was supposed to consist of a client running on the embedded device and server acting as a central point where clients were connected to. As such, we had several requirements for the language:
- As the client application would be running on embedded devices, compiled binaries needed to be as small as possible.
- It needed to work with Yocto, the embedded distribution of Linux.
- The complexity of installation clients should be low, thus limiting external dependencies and libraries.
- As it would run on different devices, the language must compile across different architectures.
- The devices that the Mender client application would run on would be an IoT or networked device, so the language needed access to networking libraries.
The requirements for selecting a new language also covered a few non-functional requirements:
- As many programmers as possible within our organization needed to be able to understand the language.
- It had to be as easy as possible to share and reuse the existing code between existing applications written in C and reuse code between the client and server applications.
- Development speed was also considered—we have ongoing pressure to add new features quickly.
A comparison between Go, C, and C++ is outlined in Figure 1. From the comparison, Go was primarily selected for its support of buffer overflow protection, automatic memory management, use of standard data containers, and its integrated support for JSON, HTTP, and SSL/TLS libraries.
RequirementCC++GoSize requirements in devicesLowestLow (1.8MB more)Low (2.1 MB more, however will increase with more binaries)Setup requirements in YoctoNoneNoneNoneBuffer under/overflow protectionNoneLittleYesCode reuse/sharing from CFEngineGoodEasy (full backwards compatibility)Can import C APIAutomatic memory managementNoAvailable, but not enforcedYesStandard data containersNoYesYesJSONjson-cjsoncppBuilt-inHTTP librarycurlcurlBuilt-inSSL/TLSOpenSSLOpenSSLBuilt-in (TLS is a part of crypto/tls package)*Figure 1: A table comparison of features across Go, C, and C++
The evaluation continued to see how Go would stack up against C, C++, and C++/Qt for creating a full-blown custom Linux image using Yocto. Pure image size and image size with network stack were compared, as was how the end application would be delivered to a target image. As Go applications can be statically compiled to a single binary, there is no need for any extra virtual environment (like in the case of Java, which requires a virtual machine in addition to binaries) or additional dependencies running on the device to use the Go code.
CC++C++/QtGoSize overhead of the test component, and all the new libraries pulled in, when using the smallest available Yocto image as a basis8.4MB10.2MB20.8MB14.6MBSize with network stack13.4MB (curl)15.2MB (curl)20.8MB*14.6MBShared dependenciesYesYesYesNoExtra Yocto layer neededNoNoYesNoDeployment complexityBinaryBinaryBinary + QtBinaryFigure 2: How Go compares to other programming languages for Yocto builds
Comparing Go vs. C
Go was also selected for development due to its extremely rich standard library, which allows for much faster development, especially when compared to C. Go inherits a lot of elements from C, C++, and Python including the expression syntax, control flow statements, data structures, interfaces, pointers, the pass by value concept, parameter parsing syntax, string handling, and garbage collection. With its optimized compiler, Go runs natively on embedded devices.
Go did have some disadvantages, which we’ll mention below. They didn’t weigh heavily enough to overcome its advantages, like the development speed and ease of building in the language, but they did factor into our decision. Here’s the other qualities that went into our decision.
Size comparison of Go vs. C
Go is bulkier than C when it comes to size, which was one of the few disadvantages it had. If you take “hello world” as the smallest application you can write in Go and use the built-in println function, after stripping away debugging symbols it is just over 600 kB in size (snippet 1). If you include the fmt package with its dependencies, then the size increases to 1.5 MB (snippet 2).
To compare this to C, if you are building a dynamically linked library, then it is a mere 8 kbytes. If you do it static, then the size increases to over 800 kbytes which is surprisingly even larger than the Go binary (snippet 3).
Snippet 1: smallest Go “hello world” code:
package main
func main() {
println("hello world")
}
$ go build
> 938K
$ go build -ldflags ‘-s -w’
> 682K
$ go build & strip
> 623KSnippet 2: Go “standard” “hello world”
package main
import "fmt"
func main() {
fmt.Println("hello world")
}
$ go build
> 1,5MSnippet 3: C “hello world”
#include <stdio.h>
int main(void)
{
printf("hello world\n");
return 0;
}$ gcc main.c
> 8,5K
$ ldd a.out
> linux-vdso.so.1
> libc.so.6
> /lib64/ld-linux-x86-64.so.2
$ gcc -static main.c
> 892K
$ gcc -static main.c & strip
> 821K
Speed comparison of Go vs. C
Compiled Go code is generally slower than C executables. Go is fully garbage collected and this itself slows things down. With C, you can decide precisely where you want to allocate memory for the variables and whether that is on the stack or on the heap. With Go, the compiler tries to make an educated decision on where to allocate the variables. You can see where the variables will be allocated (go build -gcflags -m), but you cannot force the compiler to use only the stack, for example.
However, when it comes to speed we can not forget about compilation speed and developer speed. Go provides extremely fast compilation; for example, 15,000 lines of Go client code takes 1.4 seconds to compile. Go is very well designed for concurrent execution (goroutines and channels) and the aforementioned rich standard library covers most of the basic needs, so development is faster.
Compilation and cross compilation
There are two Go compilers you can use: the original one is called gc. It is part of the default installation and is written and maintained by Google. The second is called gccgo and is a frontend for GCC. With gccgo, compilation is extremely fast and large modules can be compiled within seconds. As mentioned already, Go by default compiles statically so that a single binary is created with no extra dependencies or need for virtual machines. It is possible to create and use shared libraries using the -linkshared flag. Ordinarily, Go requires no build files and a build can be carried out by a simple "go build" command, but it is also possible to use a Makefile for more advanced build procedures. Here’s the Makefile we use at Mender.
On top of the above, Go supports a large number of OSes and architectures when it comes to cross compilation. Figure 3 highlights the wide array of OSs and platforms supported by Go.
GO OS / GO ARCHamd64386armarm64ppc64leppc64mips64lemips64mipslemipswasmrisc64aixXandroidXXXXdarwinXXdragonflyXfreebsdXXXillumosXiosXjsXlinuxXXXXXXXXXXXnetbsdXXXopenbsdXXXXplan9XXXsolarisXwindowsXXXX
Debugging and testing in Go
Many developers, in order to see what’s going on within a program while it executes, will use GDB. For heavily concurrent Go applications, GDB has some issues in trying to debug them. Fortunately, there is a dedicated Go debugger called Delve that is more suitable for this purpose. Developers only familiar with GDB should be able to use it in majority of cases without any issues
It is extremely easy to add testing and unit testing to Go code. Testing is built into the language—all that is needed is to create a file with a “test” suffix, add a test prefix to your function, import the testing package, and then run “go test”. All your tests will be automatically picked up from your source and executed accordingly.
Concurrency support
Concurrency is well supported in Go. It has two built-in mechanisms: goroutines and channels. goroutines are lightweight threads just 2 kB in size. It is very easy to create a goroutine: you only need to add the “go” keyword in front of the function and it will execute concurrently. Go has its own internal scheduler, and goroutines can be multiplexed into OS threads as required . Channels are “pipes'' for exchanging messages between goroutines, which can be either blocking and non-blocking.
Static and dynamic libraries; C code in Go
Go has many less known features that allow you to pass instructions to the compiler, linker, and other parts of the toolchain using special flags. These include the -buildmode and -linkshared flags, which can be used to create and use static and dynamic libraries both for Go and C snippets. With a combination of specific flags, you can create static or dynamic libraries with generated header files for C, which can be invoked by C code later on.
That’s right! With Go we can call C code and have the best of both worlds. The cgo tool, which is part of the Go distribution, can execute C code, which makes it possible to reuse our own or system C libraries. In fact, cgo and implicit usage of C system libraries is more popular than one might think. There are cases when standard C library routines are picked over pure Go implementations by the Go language constructs themselves.
For example, when using a network package on Unix systems, there is a big chance that a cgo-based resolver will be picked. It happens to call C library routines (getaddrinfo and getnameinfo) for resolving names in cases when native Go resolver can not be used (for example on OS X when direct DNS calls are not allowed). Another common case might be when the os/user package is used. If you want to skip building cgo parts you can pass osusergo and netgo tags (go build -tags osusergo,netgo) or disable cgo completely with CGO_ENABLED=0 variable.
There are tricky things to look out for when mixing C and Go together. Because Go is garbage collected but C is not, so if you are using any C variables within Go code that are allocated on the heap in the C code, then you need to explicitly free those variables inside Go.
C++ can also be used inside Go code, which is facilitated through a simplified wrapper and interface generator (SWIG) that binds C and C++ with languages such as Python.
Go advantages outweigh the disadvantages
Our experience using Go shows both advantages and disadvantages. Starting with the negatives, there are a lot of external libraries in the community but the quality varies greatly and you need to be very careful with what kinds of libraries you use. This is improving greatly as the language matures and gains adoption.
When you have some C bindings within your Go code, things usually get more complicated, especially cross compiling, which cannot be done easily without importing the whole C cross-compilation infrastructure.
There are still a lot of positives about using Go for embedded development. It’s quick and easy to transition from C and/or Python to Go and get productive within a couple of days. Go offers very nice tooling and a standard library with more than 100 packages. You can easily set and control runtime settings such as how many OS threads you want to use (GOMAXPROCS), or if you want to switch on or off garbage collection (GOGC=off). Libraries and code can be shared easily between the server and client development teams like in our case.
It may be controversial to say this, but the forced coding standard speeds up development. Regardless of the code you are inspecting, it always looks the same. There is no time wasted wondering which coding standard to use.
We developed a few demos of embedded Go applications that you can play with using popular hardware such as Beaglebone Black or Raspberry Pi: One is called “Thermostat” and is based on Beaglebone hardware. It uses infrared distance, temperature, and humidity sensors and exports all the sensor readings through a web interface. The other one is a PRi-based robot car with a camera that can be controlled using a provided web interface. Enjoy!
Marcin Pasinski is the Chief Embedded Engineer at Mender.io. If you want to learn more about the best approaches to over-the-air software updates for IoT gateway devices, visit Mender.io and try the OTA software updater for free.