Today, CIP is thrilled to welcome IoT.bzh as the newest member of the project. The Civil Infrastructure Platform strives to create an open source “base layer” of industrial-grade software to enable the use and implementation of software building blocks in civil infrastructure projects. Embedded systems are key to the civil infrastructure. IoT.bzh’s expertise in IoT and embedded as well as its deep history with open source, make them a welcomed voice to the CIP Project.
“As we enter into an era of ongoing security risks to our most critical infrastructure, things like updates and security are crucial. Now, more than ever, supporting CIP means investing in the long term support and maintenance on the very foundational infrastructure we all rely on, said Yoshitake Kobayashi , Technical Steering Committee Chair of CIP. “For that, we are thrilled to have IoT.bzh as a new CIP member”
IoT.bzh, leading open source company for secured embedded systems provides redpesk®, a software factory in a white box enabling users to speed up and control embedded developments from the initial design cycle until product end of life. IoT.bzh works with developers from Industrial IoT markets (automotive, marine, military, energy, aeronautics etc) to help them focusing on the differentiating applications that bring value to their business
“We are thrilled to welcome IoT.bzh to the CIP Project. As an organization, they have great experience with helping the very audience CIP also aims to support,” said Urs Gleim, CIP Board Chair. “As members of the CIP Project, we look forward to working together.”
Interested in becoming a CIP member, learn more here.
Today, the Civil Infrastructure Platform has multiple requirements that need to be maintained. This is where the CIP Testing Work Group (TWG) comes in. The TWG configures and manages the automated test infrastructure for the CIP project and ensures all systems are operating correctly. Currently led by Chris Paterson (patersonc), the TWG’s main focus is on maintaining the LAVA instance that the project uses. Overall, the TWG provides the infrastructure needed to test the various CIP projects such as the Super Long Term Support (SLTS) Kernels and CIP-Core reference filesystems.
The CIP project aims to provide support for the Linux Kernel for a comparatively long time. Over time the amount of testing required will keep increasing as the project grows, so it is important to have as much of that testing as automated as possible. Without automation, the cost of testing would be prohibitive.
Under the hood
Our Continuous Integration (CI) setup is driven by GitLab CI/CD which dynamically boots up AWS EC2 on-demand instances for our build jobs using our gitlab-cloud-ci tool.
Test jobs are also created and submitted to our LAVA instance, where they are run on QEMU virtual machines and on physical devices.
The GitLab CI pipelines that we use to build/test the Kernel are hosted in a separate GitLab repository.
Currently, CIP has two LAVA master instances (production & staging) and 5 LAVA workers (Cybertrust, Denx, Mentor, Renesas & “Chris” (staging)) in use, hosting a total of 284 devices.
We support all of the CIP reference platforms. We are working to expand the number of devices available, increasing reference platform availability whilst reducing test times.
Testing WG and the ecosystem
This group is a critical part of the overall CIP ecosystems, working with other CIP WGs as well as external open source projects. For example, CIP Testing works with all of the other CIP projects and working groups as most, if not all require the ability to test their software. Outside of CIP the testing group collaborates with other open source projects such as KernelCI, LAVA and Linaro’s test definitions. CIP also builds and boot tests each stable Linux Kernel release candidate in a number of different configurations.
On the Horizon
CIP has recently started work on their third SLTS Kernel, based on v5.10.y, which means that our automated testing needs to be expanded accordingly.
On the roadmap is collaborating further with the KernalCI project on testing management. The TWG is currently working with the KernelCI project to set up CIP’s own instance of KernelCI’s back/front-end. This will allow the project to better manage its testing and automatically process and check the results for any regressions. The front-end GUI that KernelCI provides is much better for reviewing test results then the setup CIP is currently using.
Get Involved
We are always happy to collaborate with others to expand and improve our setup, whether it’s upgrading the core infrastructure or simply adding support for more test cases.
Starting early next year Civil Infrastructure Platform will start development for the next major super long-term support (SLTS) kernel version based on upstream kernel 5.10.
This will be the third SLTS kernel maintained by CIP for the extended time frame of 10 years. The SLTS kernels differentiate from regular LTS releases in that they accept certain hardware-enabling backports of upstream accepted changes. By having the latest kernel features and device supports, the new SLTS kernel will give a new starting point for long term support. This will benefit users who are planning to embark on new industrial-grade device developments or Board Support Package (BSP) developments.
If you are relying already on CIP SLTS 4.4 or 4.19 kernels or plan to make use of the upcoming version, please consider joining the project to ensure its sustainability and help expanding SLTS support also in the future. Being a member furthermore allows to influence the project direction, the choice of reference hardware and kernel configurations that will be supported and tested.
By starting the SLTS kernel development, CIP would be ready to align with a new Debian release which is expected in 2021. The Debian Project aims to provide Linux-based operating system, Debian, to be widely used with long-term support. This enables CIP to take advantage of their activities to achieve CIP’s goal.
End-users of CIP include systems for electric power generation and energy distribution, oil and gas, water and wastewater, healthcare, communications, transportation, and community management. These systems deliver essential services, provide shelter, and support social interactions and economic development. They are society’s lifelines, and CIP aims to contribute to and support these important pillars of modern society. Developing the next major SLTS kernel version helps CIP continue on its goal to create an interoperable open source software platform that is secure, reliable and sustainable for at least 10 years.
The Civil Infrastructure Platform is excited to participate in this year’s Open Source Summit EU/ Embedded Linux Conference EU!
The Open Source Summit series always provides unique opportunities to learn and connect, even when we can’t be in the same space together. We are looking forward to this year and all the ways to come together with the broader open source community.
Interested in catching up on the latest with CIP at the event? We have you covered! Through talks, our booth, Slack, and our CIP Mini Summit, there are a variety of ways to learn more about CIP.
CIP Talks
At this year’s OSS EU, we are excited to have four CIP related talks on the schedule
The CIP Mini-Summit is a 90-minute, single-track event on the topic of industrial open source system which is based on Linux. The main goal of this event is to provide technical details and an overview to develop an industrial-grade CIP open source base layer. Sub-groups of CIP will talk about current development activities as well as future plans. Attendees will get to know how their products can leverage CIP’s SLTS(Super Long Term Support) to develop Industrial grade products.
Topics to be covered:
State of Civil Infrastructure Platform
CIP Kernel Team Activities towards Super Long Term Support
Status update for testing within CIP
CIP Security towards achieving industrial-grade security
To register for the CIP Mini-Summit, add it on to your Open Source Summit + Embedded Linux Conference Europe registration.
CIP Booth
As a sponsor of the event, we will have an event “home base” for all things CIP. Stop by our booth for more information on the project and ways to get involved.
The Civil Infrastructure Platform has several work groups that ensure things keep running. Below is a Q and A with the CIP Kernel Team.
1. What is CIP Kernel Team (What does this team work on, what issues does it solve)
While the CIP project aims to establish an open source base layer (OSBL) of industrial grade software to enable the use and implementation of software building blocks for civil infrastructure, CIP Kernel Team is responsible for Linux kernel in OSBL to sustain industrial grade systems or devices during their life cycles.
2. What is the primary goal of this team?
The goal of the team is to provide CIP kernels with more than a ten year maintenance period by fixing versions to fulfill the required level of reliability, sustainability, and security.
3. What is the development principle of CIP?
CIP adopts the upstream first as our development principle. The “Upstream First” principle allows patch commits only if those patches are already in the upstream. By following this principle, if a desired patch is not in the upstream yet, this patch should be accepted by the upstream at first. Therefore, it may take time to introduce the desired patch to our project.
But, it enables us to share our outputs with the upstream. At the same time, the risk of conflicts can be eliminated.
CIP is aiming to sustain target systems and devices during their life cycles which are very long by their nature. So the Upstream First principle is essential to achieve our goal.
4. What is “Upstream First” for the Kernel Team?
For the CIP kernel team, upstreams are Linux mainline and LTS. The team collaborates with upstream projects. Before using their outputs, the team upstreams what the team has and doesn’t keep them locally.
As marked 1, “Contribution” is our first action. Feature upstreaming is done by CIP member developers. On the other hand, the CIP Kernel Team contributes to upstream in a more general manner. The team developed open source tools in order to work on contributions effectively..
As marked 2, “Use” is the second action. The team uses LTS kernels to release CIP SLTS kernels. For those releases, automated testing plays a very important role. Therefore the CIP kernel team is closely working with the CIP testing team.
As marked 3, “Integrate” is the third action. By integrating those SLTS kernels with CIP Core packages and additional packages, industrial systems or devices can be developed and maintained.
5. How does the team use LTS kernels?
The team uses LTS for CIP SLTS kernel bases.
CIP SLTS kernels are based on LTS 4.4 and 4.19. The first releases of SLTS 4.19 and 4.19rt were done in 2019. The team plans to maintain them until 2029 for ten years. The first releases of SLTS 4.4 and 4.4rt were done in 2017, and likewise the team supports them for ten years till 2027.
Both LTS 4.4 and 4.19 are maintained for 6 years by the LTS project. So, the remaining 4 years will be maintained by the CIP Kernel Team.
6. How can CIP kernels be used?
By integrating the SLTS kernels with CIP Core packages and additional packages, industrial systems or devices can be developed.
CIP refers to Debian for userland packages. If you would like to use Debian source packages, you can use Yocto/Poky as a build system.
CIP core packages contain tens of packages which may not be sufficient for the development of end products. So, you can add necessary packages from Debian by writing recipes.
7. What has this team accomplished so far?
Currently SLTS 4.19 is released twice a month and 4.4 is once a month. SLTS 4.19-rt is once a month and 4.4-rt once every two months.
So far the team has steadily released CIP SLTS kernels by following release frequencies below.
(as of June 7, 2020)
8. What are some future goals?
The team made major releases in 2017 and 2019. So, a major release frequency is once per two years so far. Another two years is going to pass, and Year 2021 is approaching. So, the team started to discuss new SLTS kernels.
9. How can people get involved?
To get the latest information, please subscribe and contact:
Session Highlight: CIP Installed: Sustainable Software Stacks in Long-living Products
Civil Infrastructure Platform is excited to participate in the Linux Foundations’ OpenSource Summit North America Event. We are showing up in a variety of ways, as a sponsor, with several sessions and hosting the CIP Mini-Summit.
Check out talk details below and read on to learn how to register for the event.
Since the Civil Infrastructure Platform project launched in April 2016, we work with other open source communities to develop a super long-term supported (SLTS) open source “base layer” of industrial grade software that enables the implementation of building blocks in civil infrastructure projects. Long-term sustainability becomes a more major issue for not only industrial IoT systems, such as power plants, traffic lights, communications and weather systems, but also consumer IoT systems such as automotive and smart homes. In this talk, we will show the latest results of the CIP community in first half. Then we move to introduce examples of how CIP is used in real-world use cases.
The Civil Infrastructure Platform is thrilled to be sponsoring the Linux Foundations’ OpenSource Summit North America Event. In addition to having a virtual booth, CIP reps will be giving several talks as well as hosting the CIP Mini-Summit.
Check out talk details below and read on to learn how to register for the event.
Locking down embedded Linux devices via secure boot is almost solved these day. Combining this with rollback-capable over-the-air updates shouldn’t be hard then. But as often, the devil is in the detail. When he comes out, you can easily end up with an insecure system or one that does not update anymore. Or both.
In this talk, we will present patterns and tools for secure OTA system updates that are being developed in the Software Update Workgroup of the Civil Infrastructure Platform project. We will introduce an OTA pattern consisting of redundant update images that are deployed and managed by SWUpdate and switched by a boot loader. We will discuss the options and implication of securing those images, for the boot process as well as the runtime of the images. Then we will walk through UEFI-based secure boot processes, explain shortcomings of commodity boot loaders are and where to use the embedded boot loader EFI Boot Guard instead. Finally, we will also have a look at plain U-Boot-based setups, discuss if its new UEFI mode can help to unify architectures and explain what to do when it is not available.
Session Highlight: CIP Kernel Team Activities to Accomplish Super Long Term Support
At the end of June, CIP will be participating in many ways at the Linux Foundations’ OpenSource Summit + ELC North America Event. In addition to having a virtual booth, CIP reps will be giving several talks as well as hosting the CIP Mini-Summit.
Check out talk details below and read on to learn how to register for the event.
CIP (Civil Infrastructure Platform) project aims to support industrial-grade systems in secure and reliable manners. CIP kernel team was launched in 2016 under CIP to provide and maintain Linux kernel for 10+ years, because life cycles of such industrial-grade systems are very long by their nature.
By steadily releasing SLTS (super long-term support) kernel based on LTS4.4 and LTS4.19, the team has continuously improved the release processes and tools to facilitate the team activities. The team works with LTS and other open source projects to share its findings and contribute outputs. Also, test automation has been strengthened. During the long support period of 10+ years, a large number of minor releases are planned, so the cost reduction effect by test automation will be enormous. Open source tools like “cip-kernel-sec” and “classify-failed-patches” were introduced to track the status of CVEs and to identify patches needed to apply to stable kernel, respectively.
This presentation updates CIP kernel team activities, by featuring collaborative works with LTS , the status of test automation using KernelCI and LAVA, and experiences of using the open source tools.
On June 29- July 2 the Linux Foundation is hosting Open Source Summit North America. Open Source Summit is a virtual event that connects the open source ecosystem under one roof. It’s a unique environment for cross-collaboration between developers, sysadmins, devops, architects and others who are driving technology forward. bringing together.
CIP will be participating in many ways at the event including having a virtual booth, several talks by CIP reps as well as hosting the CIP Mini-Summit.
Find details below on two talks given by Wolfgang Mauerer with Technical University of Applied Sciences Regensburg / Siemens AG.
Well-known, large communities and open source projects like the Linux kernel are an often pursued goal of scientific analysis, and questions of interest cover a broad range — core OS design, collaborative software engineering, software architectural questions and community health, to just name a few. However, many research questions are biased towards what can be nicely published, and not on the most pressing problems of projects.
This leads to a gap between what OSS communities need to know, and the insights science can provide. In this (likely opinionated) talk, we discuss this gap from two often opposite sides: As a researcher, the author has never understood why industrial belief in software engineering research seems to often stop at using design patterns, and why industry does not try to benefit more from scientific insight. As an industrial practitioner, the author has never understood why academia would need to tell industrial engineers that have participated in OSS projects for years what they have done, post facto, and why research does not listen more closely to what industry is interested in, and needs to know. We suggest some possibilities to shrink the gap.
Embedded Linux is a standard core component of systems deployed in challenging and critical scenarios. Machine learning and statistical techniques are increasingly used to ascertain or even predict various quality properties — the number of open issues to judge reliability or maximum latencies for real-time systems –, or to improve development and maintenance processes: Techniques to automatically select patches for back-porting or to identify security critical fixes have recently been suggested.
While machine learning undoubtedly has its advantages, it is by no means a panacea for solving all engineering issues that have been around for decades, and issues like lack of explainability or over-confident trust in results often cause unease. But it is also unwise to dismiss them just because they differ from traditional engineering approaches.
In this talk, we survey recent uses of ML techniques in OSS systems development and maintenance, address their benefits and disadvantages, and give recommendations on how especially industrial system integrators and solution providers can enjoy the benefits of new ML-based engineering methods without suffering from new problems.
The CIP Project has had an eventful week at the Linux Foundation’s Open Source Summit Europe and Embedded Linux Conference in Lyon, France. In addition to the project’s interactive booth, complete with live demos, the project also had a slew of informative and well-attended talks and hosted a completely sold out mini-summit.
Open source thrives on collaboration, and having this face to face time with the open-source community is so valuable in moving our project forward. Check out below for the details of CIP at OSS EU+ ELC.
CIP at OSS EU. Photos by Masato Minda
The Talks
CIP and CIP members had a combined total of five talks over the three-day event.
With more than 200 people in attendance, “Debian and Yocto Project-Based Long-Term Maintenance Approaches for Embedded Products,” given by Kazuhiro Hayashi, Toshiba & Jan Kiszka, Siemens AG, really struck interest from the OSS EU audience.
Jan Kiszka, Siemens AG and Kazuhiro Hayashi, Toshiba
In this talk, the duo explained the overall build and test setup in their talk about Debian and Yocto-based embedded Linux approaches that satisfy the requirement for 10+ year maintenance in industrial products, specifically around security fixes, reproducible builds, and continuous system updates.
In their talk, “Open Source Projects to Live Long and Prosper: Linux for Smart Infrastructure and Industry,” Yoshitake Kobayashi, Toshiba Corporation & Urs Gleim, Siemens AG gave an overview of the project and what happened during the last year. Especially the working groups of CIP (on kernel, real-time, core packages, security, and software update) led to great progress in providing a sustainable base for any industrial-grade Linux distribution.
SZ Lin (林上智), Moxa & Pavel Machek, Denx focused on the long-term maintenance strategy of the kernel in their talk called “Activities of Super Long Term Support Kernel Workgroup in Civil Infrastructure Platform Project.”
Wolfgang Mauerer, Siemens AG, contributed to The List is our Process: An Analysis of the Kernel’s Email-based Development Process. Together with main author Ralf Ramsauer, Technical University of Applied Sciences Regensburg, Sebastian Duda from U Erlangen, and L. Bulwahn from the ELISA project, discussed analysis methods how to track the flow of patches into the kernel, and applied their results to specific subsystems of the Linux kernel. Their work contributes to a reliability analysis of the kernel development and can be used to detect remaining weak spots.
Michael Adler from Siemens AG, and Chris Paterson from Renesas presented “A Guide to CIP and Testing,” where they walked the audience through the CIP testing approach at the Automated Testing Summit.
The CIP Mini-Summit
For the first time, the CIP project organized a mini-summit, a half-day, single-track event covering Linux-based industrial open source systems. With this event, CIP gathered those interested in open source to provide technical details and in-depth insights to further develop the industrial-grade CIP base layer which is built on the work of established and stable work from the likes of Debian, Yocto Project, Real-Time Linux. The sold-out event included topics such as
The State of Civil Infrastructure Platform
CIP SLTS kernel development (e.g. Patch management for collaboration with stable kernel team)
Security in industrial systems and its future
Safe software updates for industrial IoT devices
Use cases of the CIP open source base layer
The CIP Booth
In addition to great content in the form of talks and sessions, the CIP community also interacted with hundreds of attendees in the CIP booth in the Sponsor Showcase. Within the booth, both Plat’Home and Toshiba demoed their technologies which are built on CIP.
Open Source in Research and Reality
