Scaling & Prioritizing Product Security with Zendesk

Zendesk, Inc. (NYSE: ZEN) is one of the fastest-growing customer support platforms in the world. With over 150,000 customer accounts representing nearly every industry across 160 counties, Zendesk is held to the highest security standards and complex data compliance requirements. For that reason, security has been a priority from the beginning.

In a recent virtual roundtable, we sat down with Scott Reed, Senior Manager of Product Security at Zendesk, to discuss how they incorporate bug bounties throughout their product security strategy and scaling security at a high-growth organization. Take a look at some of the highlights of our conversation below.

Q: Zendesk just surpassed the five year anniversary of its public bug bounty program on HackerOne last month. Can you share a little bit about that origin story and how the program has evolved over the years?
A: When I joined Zendesk six years ago, there wasn’t even a product security team. We were running a responsible disclosure program at the time, mailing t-shirts around the world and managing a hall of fame page. Over time, the logistics of handling this became overwhelming. With big goals to become a global company, we needed a better way to determine our effectiveness which led us to launch the bug bounty program. Five years later, we have approximately 1,000 engineers in offices around the world and the program allows us to properly block and tackle when it comes to product security.

Q: Zendesk grew to over 4,000 employees since 2007. Yet, resources are often tight in cybersecurity, especially right now. You’ve had to scale quickly without scaling resources tremendously. How does your team manage to scale the efforts of a limited product security team with an engineering team of approximately 1,000?
A: Scaling is a challenge that requires continuous work. Our current approach to addressing scale is to provide a “golden path,” which means making the easy path the secure path where possible. Of course, it’s not always possible with teams with different tech stacks and acquisitions with various development processes that take time to fully integrate. We’ve seen success scaling in partnership with HackerOne by using triaging services, penetration testing, vulnerability scanning tools and automation solutions to cut through the noise. Ultimately, the goal is to put power in developers' hands and let them own their own security, but in order to empower them, you need to provide tools, requirements, processes and metrics that they can leverage.

Q: Measuring success is tough, and sometimes a hotly debated topic in security, because you’re often trying to measure what doesn’t happen. How do you measure return on investment with bug bounties?
A: We have a number of metrics that we track across all of security for example covering corporate security awareness, annual developer training, tool adoption, but I think one of the most important metrics is how quickly vulnerabilities get addressed. Do you have SLAs in place and are you meeting those SLAs? This number can show that teams are engaged or that their time is focused elsewhere. With strong numbers here, at least if your bug bounty program identifies an issue you can get it fixed in a reasonable amount of time. 

Q: In terms of a playbook, what kind of advice or lessons learned would you share from the Zendesk bug bounty program? Where’s a good place to start? Any pitfalls or learnings from your journey?
A: Companies should start by having a vulnerability management process in place because if you can’t fix the vulnerabilities you are finding, then you are going to kill engagement from researchers. Also, start by running vulnerability scans to make sure you are getting the easy stuff out of the way. You want your researchers focused on the high reward findings. Finally, I recommend starting with a private program in order to work out the kinks of this new process. You might need to change your policy, scope or bounty amounts. This also gives you time to get familiar with communicating with researchers and get some insight into the findings so if there are areas of weakness you can shore these up before launching publicly and getting the flood of reports. And on that note, be sure that you have enough dedicated headcount. You can take advantage of the triaging services to help you with the workload. 

Q: What are some of the standout success stories from the program? Any memorable hacker interactions?
A: Within the first months of launching our private program, a Server Side Request Forgery (SSRF) vulnerability was reported. At the time, SSRF was still fairly new and the potential impact was not well understood. We initially believed the impact to be limited to some disclosure of information about internal infrastructure, warranting only a moderate risk. However, with some clever changes to their payload, the researcher was able to elevate the impact of the vulnerability resulting in remote code execution. Throughout the process, the researcher was a pleasure to work with  - always very professional and helping us understand the full impact of the issue. Because of that healthy interaction and the severity of the vulnerability, we ended up awarding three times our max bounty award. 

Q: As you look towards the future, what excites you about cybersecurity at Zendesk? What’s your vision for hackers as part of that roadmap?
A: The Zendesk Engineering team is incredibly innovative, always leveraging the cutting edge of technology. It’s not easy, but keeping up to date and always learning how to secure that bleeding edge tech gets me excited. Also, Zendesk has continued to grow at a phenomenal pace, so continuing down that journey of scaling security through automation and empowering engineers to make good security decisions is exciting. And, of course, we will continue to grow the Product Security team in parallel and I’m always excited to meet our new team members and learn about their different personalities and  perspectives. As this growth continues, our bug bounty program and the hackers of HackerOne will always be one way to measure the effectiveness of our work and guide us on areas of the product that may need more attention from our team. 

Q: Anything else? Anything you’d like to say to the hackers that have participated over the last five years or those that you’d like to engage in your program?
A: As a service-first CRM company, we are hyper-focused on helping companies improve relationships with their customers across all channels at all stages of the lifecycle. Now, our customers really need us and my team is committed to making sure we are supporting all their security needs. To all the hackers that have participated in our bug bounty program over the years, your relationship is important to us too! I just want to say a big thank you for your hard work and contributing to a more secure Zendesk customer experience across all of our products these past few years. We hope to continue building those relationships and making new relationships and we look forward to future submissions across the hacker community.

To learn more about the Zendesk bug bounty program on HackerOne, visit https://hackerone.com/zendesk.