Shifting our focus to Less Lead Time

Lead Time is the time taken by the process to reach the finish line. Here, I am talking about technological changes that involve code and infrastructure. Leadership expects to deliver product (or) feature (or) changes faster, They want us to move fast in this agile world. I was thinking holistically about how we as a security engineer can be lean and contribute to the security and quality of the product with less acceptable lead time. To start with, let us understand the security sign off process

Security Sign off

Security Sign off is to provide confidence with tradeoffs in releasing the code change/feature change to the production. The diagram below depicts the security sign-off workflow holistically involving Developer, QA and Security.

I chose a sequence diagram to capture the dynamic nature including the interaction between state objects. I have drawn it from the perspective of the developer. In the above diagram, you can observe multiple personnel having their own goals. But the ultimate goal is to move faster and push code to production faster leading to less lead time.

Shared Goals

1. A developer wants to push the code faster thereby having less lead time.

2. QA wants to ensure the code free from bugs and the product is reliable.

3. Security wants to ensure the product, code and infrastructure are safe and secure from the threat actors.

One metric, important for all the personnel involved, is having less lead time to push safe, reliable and quality features and code to customers. How could we do a good job with less lead time and good quality? Before talking about it, we should understand what type of change is introduced.

Type of changes

1. New Feature

2. Changing the existing feature

3. Revamp

In each type of change, some checkpoints need our attention. For example, if you ask your colleague who travels to Bengaluru, India from his native place, how was the flight? They would answer revolving around time. How much time did I waste reaching inner Bengaluru from the airport? if you have noticed, they would not mention the time taken to have breakfast, someone they met on the way, cause the waiting and travelling time looks big and tiring and that’s what matters to them and us. Similarly, points of change are breaking points that matter from a security perspective. Points of change are changes that are crucial for security teams to review. Any changes that could trigger a security review. This change can happen at any point in time till the developer pushes to prod since we are agile. The points of change are

1. Requirement

2. Design

3. Infrastructure

4. Code and Modules

Including the point of change ensures coverage in security sign off. Coverage is an important quality to have in security sign off.

Minimising Time

This section talks about how we can optimise for less lead time. The important thing in minimising time is increasing knowledge sharing in minimal time. Before diving into it, Let’s understand ownership.

Ownership

From the above, Product managers own the feature/product and they do not own any technical aspects such as code, design or infrastructure. From the above, I could deduce these ownership types

1. Product Ownership

2. Service Ownership

What is the product? feature? service?. Products are the ones that customers can see and use. Products have sub-products that might not be directly visible to customers. All sub-products are a collection of features. Products and Features uses multiple services to create those services.

The Product Manager owns the product and features. Their goal is to increase the revenue by X times. Developer owns services. In a startup, the Development team can own sub-products. There will be development teams that own sub-products and are named after them. In Mid Size to Big Orgs, the Developer owns services. There comes the question of What kind of ownership security teams should have. Cause security is stretched from product requirements till the release of the change, Security should have both product and service ownership. The cons of having only service ownership is that you do not know all other features of the product which might open insecure states.

Knowledge Sharing

Knowledge Transfer/Sharing is dependent on the ownership mentioned above. To minimise the lead time we need to have a faster knowledge sharing of the product and the services that the security engineer owns. When the security engineer does not know much about the product and the services, it would be difficult and inefficient to review thereby increasing the lead time. At the time of onboarding, understanding the product and the services plays a big part in reducing the lead time.

Pair testing

Pair testing is similar to pair programming which increases knowledge transfer. When a new security engineer gets onboarded, do a buddy/pair testing with a senior engineer or the product owner to understand what the driver(cross-product owned security engineer) knows. Getting the security engineer involved with a buddy makes the knowledge transfer quicker. Decide on the goals of pair testing and how are we going to do it. before doing one. Plan regular cross-service pair testing with other security colleagues to understand the security debts and product knowledge regularly.

Blocking and non-blocking review

Security reviews can be done in 2 ways. Blocking and non-blocking review. Blocking reviews are time-bound based. For example, you can set up a pair review session with all the stakeholders and have a security review. Let the developer be the driver explaining and security engineers take the turn to discuss the possible issues. This could be done in a design review. A non-blocking review will help us review parallelly but will increase lead time due to context switching involved. Security engineers are not handling only one review per week. They are already short on bandwidth and have more to review. In non-blocking review, urgent work and important work will come in the middle, thereby delaying the lead time. It is best to have a blocking review.

Reduced Time to fix

Reduce the time to fix metrics by identifying issues with quicker feedback from CICD automation tools. Having SAST, SCA and other security automation tools in CICD provides quicker feedback and developers could fix them asap. The security team should make sure to provide support when the automation tool gets broken and is easily accessible to developers. Developers should be educated on how to fix, where to see vulnerabilities, How they could ignore vulnerabilities, reduce false positives and how to fix them from the tools themselves. Our goal is to reduce the time to fix such that it will contribute to less lead time.

Shift left is all about reducing the lead time.