Skip to main content
All Posts By


VES LLC Joins CIP as a Silver Member

By Announcement, Blog, In the News

Leader in custom Government off the Shelf (GOTS) infrastructure solutions becomes the newest member of Civil Infrastructure Platform (CIP)

Today, the Civil Infrastructure Platform (CIP) welcomes VES LLC as its newest member. VES is a small business Headquartered out of Aberdeen Proving Ground, Maryland with a focus on solving the Department of Defense’s (DoD) hardest Software Systems Integration challenges. VES is joining CIP to further their development of custom Government off the Shelf (GOTS) infrastructure solutions, integrating Mission Command systems, and prototyping emerging technologies for use in the Army and Joint tactical architecture.

The Civil Infrastructure Platform strives to create an open source “base layer” of industrial-grade software to enable the use and implementation of software building blocks in civil infrastructure projects. Embedded systems are crucial to civil infrastructure, including within Army operating systems and across the DoD. Given VES’ area of expertise, and CIP’s mission to establish an open source “base layer” of industrial-grade software, there’s strong alignment with both CIP and VES.

“As CIP grows, it is exciting to bring in a broader array of organizations wishing to establish a Linux-based open source base layer for industrial-grade, civil infrastructure.” said Yoshitake Kobayashi, Technical Steering Committee Chair of CIP, “We are excited to have VES on board and welcome all future collaboration within the CIP community.” 

Matthew Vidovich

“We are very excited to join the CIP and become an integral member of an expansive network focused on open source solutions with other industry leaders.” said VES CEO, Matt Vidovich.  “Each member of our core VES leadership team brings over 17 years of open systems architecture experience across the Department of Defense, commercial, and international markets.  We look forward to expanding our relationships and impact with other stakeholders sharing the same purpose and passion on solving the toughest open source problems with enduring solutions.”

Brad Lilly, VES Chief Technology Officer (CTO) for Systems

Brad Lilly, VES Chief Technology Officer (CTO) for Systems, stated “As a segment leader in custom DoD Linux Distributions, VES is committed to the ongoing security and maintainability for our customer’s systems. CIP has given us a strong base to build on, and we are excited to begin contributing back to help ensure CIP’s long term success.” 

Established in 2014, VES has specialized expertise in building GOTS versions of embedded Linux for Army operating systems needs, and in developing and deploying the Army Mission Command Infrastructure architecture.

Interested in becoming a CIP member, learn more here. 

Welcome as a CIP Member

By Blog, Work Group

Today, CIP is thrilled to welcome as the newest member of the project. The Civil Infrastructure Platform strives to create an open source “base layer” of industrial-grade software to enable the use and implementation of software building blocks in civil infrastructure projects. Embedded systems are key to the civil infrastructure.’s expertise in IoT and embedded as well as its deep history with open source, make them a welcomed voice to the CIP Project.

“As we enter into an era of ongoing security risks to our most critical infrastructure, things like updates and security are crucial. Now, more than ever, supporting CIP means investing in the long term support and maintenance on the very foundational infrastructure we all rely on, said Yoshitake Kobayashi , Technical Steering Committee Chair of CIP. “For that, we are thrilled to have as a new CIP member”, leading open source company for secured embedded systems provides redpesk®, a software factory in a white box enabling users to speed up and control embedded developments from the initial design cycle until product end of life. works with developers from Industrial IoT markets (automotive, marine, military, energy, aeronautics etc) to help them focusing on the differentiating applications that bring value to their business

“We are thrilled to welcome to the CIP Project. As an organization, they have great experience with helping the very audience CIP also aims to support,” said Urs Gleim, CIP Board Chair. “As members of the CIP Project, we look forward to working together.” 

Interested in becoming a CIP member, learn more here. 

CIP Testing Working Group

By Blog, Work Group

Today, the Civil Infrastructure Platform has multiple requirements that need to be maintained. This is where the CIP Testing Work Group (TWG) comes in. The TWG configures and manages the automated test infrastructure for the CIP project and ensures all systems are operating correctly. Currently led by Chris Paterson (patersonc), the TWG’s main focus is on maintaining the LAVA instance that the project uses. Overall, the TWG provides the infrastructure needed to test the various CIP projects such as the Super Long Term Support (SLTS) Kernels and CIP-Core reference filesystems.

The CIP project aims to provide support for the Linux Kernel for a comparatively long time. Over time the amount of testing required will keep increasing as the project grows, so it is important to have as much of that testing as automated as possible. Without automation, the cost of testing would be prohibitive.

Under the hood 

Our Continuous Integration (CI) setup is driven by GitLab CI/CD which dynamically boots up AWS EC2 on-demand instances for our build jobs using our gitlab-cloud-ci tool.

Test jobs are also created and submitted to our LAVA instance, where they are run on QEMU virtual machines and on physical devices.

The GitLab CI pipelines that we use to build/test the Kernel are hosted in a separate GitLab repository.

Currently, CIP has two LAVA master instances (production & staging) and 5 LAVA workers (Cybertrust, Denx, Mentor, Renesas & “Chris” (staging)) in use, hosting a total of 284 devices.

The current device status can be viewed at

We support all of the CIP reference platforms. We are working to expand the number of devices available, increasing reference platform availability whilst reducing test times.

Testing WG and the ecosystem 

This group is a critical part of the overall CIP ecosystems, working with other CIP WGs as well as external open source projects. For example, CIP Testing works with all of the other CIP projects and working groups as most, if not all require the ability to test their software. Outside of CIP the testing group collaborates with other open source projects such as KernelCI, LAVA and Linaro’s test definitions. CIP also builds and boot tests each stable Linux Kernel release candidate in a number of different configurations.

On the Horizon

CIP has recently started work on their third SLTS Kernel, based on v5.10.y, which means that our automated testing needs to be expanded accordingly.

On the roadmap is collaborating further with the KernalCI project on testing management. The TWG is currently working with the KernelCI project to set up CIP’s own instance of KernelCI’s back/front-end. This will allow the project to better manage its testing and automatically process and check the results for any regressions. The front-end GUI that KernelCI provides is much better for reviewing test results then the setup CIP is currently using.

Get Involved

We are always happy to collaborate with others to expand and improve our setup, whether it’s upgrading the core infrastructure or simply adding support for more test cases.

Reach out to us on IRC (Freenode #cip) or via the cip-dev mailing list.

More information on the activities of the TWG can be found on the CIP Wiki.

CIP to Embark on Kernel 5.10 Development for SLTS

By Announcement, Blog

Starting early next year Civil Infrastructure Platform will start development for the next major super long-term support (SLTS) kernel version based on upstream kernel 5.10.

This will be the third SLTS kernel maintained by CIP for the extended time frame of 10 years. The SLTS kernels differentiate from regular LTS releases in that they accept certain hardware-enabling backports of upstream accepted changes. By having the latest kernel features and device supports, the new SLTS kernel will give a new starting point for long term support. This will benefit users who are planning to embark on new industrial-grade device developments or Board Support Package (BSP) developments.

If you are relying already on CIP SLTS 4.4 or 4.19 kernels or plan to make use of the upcoming version, please consider joining the project to ensure its sustainability and help expanding SLTS support also in the future. Being a member furthermore allows to influence the project direction, the choice of reference hardware and kernel configurations that will be supported and tested.

By starting the SLTS kernel development, CIP would be ready to align with a new Debian release which is expected in 2021. The Debian Project aims to provide Linux-based operating system, Debian, to be widely used with long-term support. This enables CIP to take advantage of their activities to achieve CIP’s goal. 

End-users of CIP include systems for electric power generation and energy distribution, oil and gas, water and wastewater, healthcare, communications, transportation, and community management. These systems deliver essential services, provide shelter, and support social interactions and economic development. They are society’s lifelines, and CIP aims to contribute to and support these important pillars of modern society. Developing the next major SLTS kernel version helps CIP continue on its goal to create an interoperable open source software platform that is secure, reliable and sustainable for at least 10 years. 

CIP at Open Source Summit Europe 2020

By Blog, Events

The Civil Infrastructure Platform is excited to participate in this year’s Open Source Summit EU/ Embedded Linux Conference EU! 

The Open Source Summit series always provides unique opportunities to learn and connect, even when we can’t be in the same space together. We are looking forward to this year and all the ways to come together with the broader open source community. 

Interested in catching up on the latest with CIP at the event? We have you covered! Through talks, our booth, Slack, and our CIP Mini Summit, there are a variety of ways to learn more about CIP.

CIP Talks

At this year’s OSS EU, we are excited to have four CIP related talks on the schedule

CIP Mini Summit

The CIP Mini-Summit is a 90-minute, single-track event on the topic of industrial open source system which is based on Linux. The main goal of this event is to provide technical details and an overview to develop an industrial-grade CIP open source base layer. Sub-groups of CIP will talk about current development activities as well as future plans. Attendees will get to know how their products can leverage CIP’s SLTS(Super Long Term Support) to develop Industrial grade products.

Topics to be covered:

  • State of Civil Infrastructure Platform
  • CIP Kernel Team Activities towards Super Long Term Support
  • Status update for testing within CIP
  • CIP Security towards achieving industrial-grade security

To register for the CIP Mini-Summit, add it on to your Open Source Summit + Embedded Linux Conference Europe registration.

CIP Booth

As a sponsor of the event, we will have an event “home base” for all things CIP. Stop by our booth for more information on the project and ways to get involved.

Visit* the CIP Booth in the Sponsor Showcase


*Please note that you will need to be registered for the event as well as Accelevents to view the booth. 

If you can’t wait until Oct 26th to connect, no need to! Visit us at, find us on twitter at, on LinkedIn at  and direct any membership questions to

CIP Kernel Team: Helping CIP Sustain Industrial Grade Systems

By Blog, Work Group

The Civil Infrastructure Platform has several work groups that ensure things keep running. Below is a Q and A with the CIP Kernel Team. 

1. What is CIP Kernel Team (What does this team work on, what issues does it solve)

While the CIP project aims to establish an open source base layer (OSBL) of industrial grade software to enable the use and implementation of software building blocks for civil infrastructure, CIP Kernel Team is responsible for Linux kernel in OSBL to sustain industrial grade systems or devices during their life cycles.

2. What is the primary goal of this team?

The goal of the team is to provide CIP kernels with more than a ten year maintenance period by fixing versions to fulfill the required level of reliability, sustainability, and security.

3. What is the development principle of CIP?

CIP adopts the upstream first as our development principle. The “Upstream First” principle allows patch commits only if those patches are already in the upstream. By following this principle, if a desired patch is not in the upstream yet, this patch should be accepted by the upstream at first. Therefore, it may take time to introduce the desired patch to our project.

But, it enables us to share our outputs with the upstream.  At the same time, the risk of conflicts can be eliminated.

CIP is aiming to sustain target systems and devices during their life cycles which are very long by their nature. So the Upstream First principle is essential to achieve our goal. 

4. What is “Upstream First” for the Kernel Team?

For the CIP kernel team, upstreams are Linux mainline and LTS. The team collaborates with upstream projects. Before using their outputs, the team upstreams what the team has and doesn’t keep them locally. 

As marked 1, “Contribution” is our first action. Feature upstreaming is done by CIP member developers. On the other hand, the CIP Kernel Team contributes to upstream in a more general manner. The team developed open source tools in order to work on contributions effectively..

As marked 2, “Use” is the second action. The team uses LTS kernels to release CIP SLTS kernels. For those releases, automated testing plays a very important role. Therefore the  CIP kernel team is closely working with the CIP testing team.

As marked 3, “Integrate” is the third action. By integrating those SLTS kernels with CIP Core packages and additional packages, industrial systems or devices can be developed and maintained.

5. How does the team use LTS kernels?

The team uses LTS for CIP SLTS kernel bases. 

CIP SLTS kernels are based on LTS 4.4 and 4.19. The first releases of SLTS 4.19 and 4.19rt were done in 2019. The team plans to maintain them until 2029 for ten years. The first releases of SLTS 4.4 and 4.4rt were done in 2017, and likewise the team supports them for ten years till 2027.

Both LTS 4.4 and 4.19 are maintained for 6 years by the LTS project. So, the remaining 4 years will be maintained by the CIP Kernel Team.

6. How can CIP kernels be used?

By integrating the SLTS kernels with CIP Core packages and additional packages, industrial systems or devices can be developed. 

CIP refers to Debian for userland packages. If you would like to use Debian source packages, you can use Yocto/Poky as a build system. 

CIP core packages contain tens of packages which may not be sufficient for the development of end products. So, you can add necessary packages from Debian by writing recipes.

7. What has this team accomplished so far?

Currently SLTS 4.19 is released twice a month and 4.4 is once a month. SLTS 4.19-rt is once a month and 4.4-rt once every two months.

So far the team has steadily released CIP SLTS kernels by following release frequencies below. 

(as of June 7, 2020)

8. What are some future goals?

The team made major releases in 2017 and 2019. So, a major release frequency is once per two years so far. Another two years is going to pass, and Year 2021 is approaching. So, the team started to discuss new SLTS kernels.  

9. How can people get involved?

To get the latest information, please subscribe and contact:

  • CIP Mailing List:

You can get CIP SLTS kernels:

Also, you can get open source tools the team is using:

  • CIP Linux kernel CVE tracker
  • CIP Linux kernel failed patches tracker

CIP at Open Source Summit North America 2020

By Blog, Events

Session Highlight: CIP Installed: Sustainable Software Stacks in Long-living Products

Civil Infrastructure Platform is excited to participate in the Linux Foundations’ OpenSource Summit North America Event. We are showing up in a variety of ways, as a sponsor, with several sessions and hosting the CIP Mini-Summit.  

On June 29 from 3:20-410 pm CT, Urs Gleim, Siemens AG & Yoshitake Kobayashi, Toshiba Corporation will be giving a talk called, “CIP Installed: Sustainable Software Stacks in Long-living Products.

Check out talk details below and read on to learn how to register for the event. 

Since the Civil Infrastructure Platform project launched in April 2016, we work with other open source communities to develop a super long-term supported (SLTS) open source “base layer” of industrial grade software that enables the implementation of building blocks in civil infrastructure projects. Long-term sustainability becomes a more major issue for not only industrial IoT systems, such as power plants, traffic lights, communications and weather systems, but also consumer IoT systems such as automotive and smart homes. In this talk, we will show the latest results of the CIP community in first half. Then we move to introduce examples of how CIP is used in real-world use cases.

Register for Open Source Summit NA today to attend this virtual session. 

Open Source Summit North America 2020

By Blog, Events

Session Highlight: Secure Boot and Over-the-Air Updates – That’s Simple, No?

The Civil Infrastructure Platform is thrilled to be sponsoring the Linux Foundations’ OpenSource Summit North America Event. In addition to having a virtual booth, CIP reps will be giving several talks as well as hosting the CIP Mini-Summit.  

On June 30 from 9:30 to 10:20 am CT,  Jan Kiszka with Siemens AG will be giving a talk called, “Secure Boot and Over-the-Air Updates – That’s Simple, No?”  

Check out talk details below and read on to learn how to register for the event. 

Locking down embedded Linux devices via secure boot is almost solved these day. Combining this with rollback-capable over-the-air updates shouldn’t be hard then. But as often, the devil is in the detail. When he comes out, you can easily end up with an insecure system or one that does not update anymore. Or both.

In this talk, we will present patterns and tools for secure OTA system updates that are being developed in the Software Update Workgroup of the Civil Infrastructure Platform project. We will introduce an OTA pattern consisting of redundant update images that are deployed and managed by SWUpdate and switched by a boot loader. We will discuss the options and implication of securing those images, for the boot process as well as the runtime of the images. Then we will walk through UEFI-based secure boot processes, explain shortcomings of commodity boot loaders are and where to use the embedded boot loader EFI Boot Guard instead. Finally, we will also have a look at plain U-Boot-based setups, discuss if its new UEFI mode can help to unify architectures and explain what to do when it is not available.

Register for Open Source Summit NA today to attend this virtual session.

CIP at Open Source Summit + ELC North America 2020

By Blog, Events

Session Highlight: CIP Kernel Team Activities to Accomplish Super Long Term Support

At the end of June, CIP will be participating in many ways at the Linux Foundations’ OpenSource Summit + ELC North America Event. In addition to having a virtual booth, CIP reps will be giving several talks as well as hosting the CIP Mini-Summit.  

On June 29 from 9:30 to 10:20 am CDT,  Masashi Kudo, Cybertrust Japan Co., Ltd. & SZ Lin  (林上智), Moxa Inc. will be presenting “CIP Kernel Team Activities to Accomplish Super Long Term Support” 

Check out talk details below and read on to learn how to register for the event. 

CIP (Civil Infrastructure Platform) project aims to support industrial-grade systems in secure and reliable manners. CIP kernel team was launched in 2016 under CIP to provide and maintain Linux kernel for 10+ years, because life cycles of such industrial-grade systems are very long by their nature.

By steadily releasing SLTS (super long-term support) kernel based on LTS4.4 and LTS4.19, the team has continuously improved the release processes and tools to facilitate the team activities. The team works with LTS and other open source projects to share its findings and contribute outputs. Also, test automation has been strengthened. During the long support period of 10+ years, a large number of minor releases are planned, so the cost reduction effect by test automation will be enormous. Open source tools like “cip-kernel-sec” and “classify-failed-patches” were introduced to track the status of CVEs and to identify patches needed to apply to stable kernel, respectively.

This presentation updates CIP kernel team activities, by featuring collaborative works with LTS , the status of test automation using KernelCI and LAVA, and experiences of using the open source tools.

Register for Open Source Summit NA today to attend this virtual session.

CIP at Open Source Summit North America: Session Preview

By Blog, Events

On June 29- July 2 the Linux Foundation is hosting Open Source Summit North America.  Open Source Summit is a virtual event that connects the open source ecosystem under one roof. It’s a unique environment for cross-collaboration between developers, sysadmins, devops, architects and others who are driving technology forward. bringing together. 

CIP will be participating in many ways at the event including having a virtual booth, several talks by CIP reps as well as hosting the CIP Mini-Summit.  

Find details below on two talks given by Wolfgang Mauerer with Technical University of Applied Sciences Regensburg / Siemens AG. 

Open Source in Research and Reality
June 30, 2020
4:15 pm to 5:05 pm CT

Well-known, large communities and open source projects like the Linux kernel are an often pursued goal of scientific analysis, and questions of interest cover a broad range — core OS design, collaborative software engineering, software architectural questions and community health, to just name a few. However, many research questions are biased towards what can be nicely published, and not on the most pressing problems of projects.

This leads to a gap between what OSS communities need to know, and the insights science can provide. In this (likely opinionated) talk, we discuss this gap from two often opposite sides: As a researcher, the author has never understood why industrial belief in software engineering research seems to often stop at using design patterns, and why industry does not try to benefit more from scientific insight. As an industrial practitioner, the author has never understood why academia would need to tell industrial engineers that have participated in OSS projects for years what they have done, post facto, and why research does not listen more closely to what industry is interested in, and needs to know. We suggest some possibilities to shrink the gap.

Safety, Security, Quality: Artificial Intelligence versus Common Sense
July 1, 2020
3:05 pm to 3:55 pm CT

Embedded Linux is a standard core component of systems deployed in challenging and critical scenarios. Machine learning and statistical techniques are increasingly used to ascertain or even predict various quality properties — the number of open issues to judge reliability or maximum latencies for real-time systems –, or to improve development and maintenance processes: Techniques to automatically select patches for back-porting or to identify security critical fixes have recently been suggested.

While machine learning undoubtedly has its advantages, it is by no means a panacea for solving all engineering issues that have been around for decades, and issues like lack of explainability or over-confident trust in results often cause unease. But it is also unwise to dismiss them just because they differ from traditional engineering approaches.

In this talk, we survey recent uses of ML techniques in OSS systems development and maintenance, address their benefits and disadvantages, and give recommendations on how especially industrial system integrators and solution providers can enjoy the benefits of new ML-based engineering methods without suffering from new problems.

Register for Open Source Summit NA today to attend these sessions.