Security maintenance — and the ability to apply security updates quickly — is part and parcel to open source project success.
Updating is typically done as part of the normal software release cycle, however, there are times when a security advisory needs to be released ASAP. A strong incident response plan builds a first defense line to mitigate and patch vulnerabilities.
But what does a successful security response look like in action?
On the heels of a recent Drupal security update on August 1, 2018, Mediacurrent’s Senior Project Manager Christine Flynn had the same question. To find out, she interviewed our Open Source Security Lead, Mark “shrop” Shropshire, to get a layperson’s perspective on the security team’s approach.
“An off-cycle Drupal security advisory dropped on August 1, 2018. What does that mean for folks who aren’t developers?”
Flynn: I was watching the Slack channel as our team fixed sites, and I got some idea of what was happening. I’m not going to jiggle anybody’s elbows while they’re applying a security update, but I’m really curious now that the fixes are all in.
Flynn: I read all of those links while the team was applying the security update, but I feel like I didn’t totally understand the implications. I’d love to get a better picture from you of what they mean.
Shrop: You bet! I hope you can hear me, I’m at a coffee shop right now.
Flynn: Are you on their unsecured WiFi?
Shrop: Nope! I’m on a hotspot and on VPN. It’s funny, the more you know about security, the more it changes what you do. Other people think you’re paranoid. But you’re not! You just understand the realities.
Flynn: Ha! Why am I not surprised? All right, let’s dig in.
“What was the security update for?”
Shrop: Drupal Core was updated because there were some security releases for Symfony. We call those “upstream” in the biz, which means that Drupal depends on them, and they are actively worked on outside of Drupal. I understand the Symfony project worked closely with the Drupal Security Team to make sure Symfony and Drupal were both updated and ready to be announced publicly at the same time. Drupal version 8.5.6 pulls in the Symfony updates as part of the Drupal update process.
Flynn: Was that the only update?
Shrop: No, at the same time, there was also an update to Zend Framework, but that was only an issue for users who were making use of modules or sites that used Zend Feed or Daictoros. There is a core issue to update the related Zend libraries for those who require or need the updates.
“If not updated, what could a malicious user do to a site?”
Shrop: This is a hard one to answer this soon after the release of the security advisory. I’m going to do some checking to see if I can get more information on this for academic purposes, but the Drupal Security Team is not going to make any statements that could help someone attack a site. It is up to security teams and researchers to dig into the code and determine more about the risks involved.
Based on the Symfony project’s blog post, it appears that a specially crafted request could allow a user access to a URL they do not have access to, bypassing access control provided by web servers and caching mechanisms. That’s a fancy-pants way of saying that a website visitor could gain access to pages you don’t want them to see.
“When will we know more?”
Shrop: Within days - sometimes hours - we might start to see exploit methods posted on the Internet. Taking security seriously and responding quickly once a drupal.org security advisory is announced is a way to stay ahead of these concerns.
Mediacurrent doesn’t want to fearmonger, but it is better to be safe than sorry. That’s why I always push to update as soon as possible while weighing in on mitigating factors that may lessen the severity of the issue for a particular application. But I will keep digging. I’m curious!
“If you had to tell a CEO or CFO the value that implementing this security update swiftly provided, what would you say? Let’s say this CEO does not have a strong background in technology or security.”
Flynn: I could see an executive with a strong public safety or physical security background being pretty understanding of why you want to apply a security update for a potential vulnerability quickly, but what if it’s someone who doesn’t have that experience, and isn’t a technologist?
Shrop: Check out this link from Acquia about the security update. This helped me so much. They published this shortly after the PSA came out, and although they’ve updated the text since then, they said at the time, “It is advised that customers set aside time for a core upgrade immediately following.” When I read, “immediately,” I knew that we had to get the update out within hours. If I was asked to get on a call with the executives from any company, at that point, I am confident. If Acquia is saying it, we need to do it. That’s enough to stand on with anybody. I’m not saying that the Acquia team has more information, but they have a very robust security team. They always dig in quickly. They have to, to know if they can mitigate the issue by adding web application firewall rules.
Flynn: Firewall rules? How does that work?
Shrop: The last few core updates, Pantheon and Acquia put mitigations into their WAF - that’s Web Application Firewall. Pantheon confirmed the night of the security advisory release that they were blocking attempts on their platform, and Acquia did the same thing. So if someone tried to exploit a site that was hosted there before Drupal was updated, they were there, helping to prevent that site from being attacked successfully. It’s a great extra layer of protection. Now, me and Acquia and Pantheon will always still want to update Core on each site, because WAF-level mitigation might not catch everything. But I am super happy when I see it because there’s a good chance that it will catch anything that happens while a team is still implementing a security update.
Security is all risk assessment and mitigation. You want to layer defenses. And something like this, we are going to make sure we deal with this problem. That’s why Acquia, Pantheon, Platform.sh, and others in the community immediately add those extra mitigations to their firewalls. It’s to buy time so that people can get their updates in. That’s not where mitigation ends, but it helps.
“What type of sites were affected by this? Does everyone use Symfony?”
Flynn: When I first read about the upcoming security advisory, I saw that it affected “third party libraries.” That made me think that some of our clients might not be affected because it would only affect certain modules. Can you tell me what types of sites were affected?
Shrop: Got a link for you, but basically, anything on Drupal 8 was affected. Drupal 8 uses components from the Symfony project. The Drupal community made the decision to use Symfony so that we didn’t have to maintain everything ourselves. So this is a great example of the power of open source, with the Symfony and Drupal security teams working together to release this fix. We all end up benefiting from having a larger community to fix issues. There’s no way an internal team working by themselves can write as secure applications on their own compared to open source software, in my opinion. It has nothing to do with how good you are, it’s the nature of development. With open source, you have a greater team with Drupal and then again, with Symfony, an even greater team to lean on. With each community that is included you are expanding your team and your ability to detect and prevent threats.
“How was the security vulnerability discovered?”
Shrop: That’s generally never disclosed because you never want to tell malicious users how you found an opening.
But we do have a few people to thank: Michael Cullum and @chaosversum were thanked by Symfony for separately reporting the two issues addressed in Symfony security releases. They also thanked Nicolas Grekas for implementing the fix. I would also give a huge thanks to Symfony and the Drupal Security Team for coming together to implement the fix and for coordinating the announcements. It’s hard work, and it shows the community at its best.
“So when we have an off-cycle security release, first the PSA comes out. Can you tell me a bit about what Mediacurrent does from the time the PSA comes out to just before the security advisory drops?”
Flynn: As someone on the team at Mediacurrent, I can see some of the things you do. But I’m wondering what else happens behind the scenes?
Shrop: The first thing that happens is that I’m notified about the PSA coming out. I’m signed up for updates via email, Twitter, and RSS feeds from https://www.drupal.org/security, and so are a lot of other folks at Mediacurrent. Internally, we have some processes that we have standardized over time for how to deal with security updates that we follow across the company. We centralize information we have on the security PSA/advisory, recommend client communications, and talk about how to prepare as a team. We have multiple communication threads internally, as well, so no one can miss it. I send an email to the staff and I post in our Slack in a few places to get us ready.
Flynn: I know that we often clear time in advance for the team to implement the security updates.
Shrop: Yep. All of us share more information as a team as official information is released or as our own investigations reveal information. For example, early on the day the security advisory was released, our DevOps Lead, Joe Stewart, noticed that Symfony had put out a notice that they were also going to be releasing a security update that day, so that gave us a heads up that it might be related. We couldn’t know for sure until the security advisory actually came out, though. No one can do it by themselves, which is why we have a whole team working on it - it’s the only way to handle these things.
“So then the security advisory drops. How did we go about fixing the issue?”
Shrop: First, we reviewed the advisory to assess risk and for any mitigations that help determine how quickly we need to perform updates. With this advisory, it was needed pretty much immediately, so we started to update Drupal core for our clients and pushed to test environments. Our QA team performed regression testing related to the update. Once QA approved each update for each client, we worked with folks to approve the updates and release them to the live environments.
The important points are to line everyone and everything up in advance, have the talent in-house who can work on clients of all shapes and sizes and needs, and then to work as a team to resolve the issue on every client site as quickly as possible.
“Were there any sites that were trickier to update? Why?”
Shrop: Clients that were on older versions of Drupal Core, who had delayed upgrading, were harder to update. Every site was updated within a short time, regardless, but even though they started at the same time, those clients did not finish first, because there was more development and testing needed on each site.
Flynn: What was different about the process to update those sites?
Shrop: If a client wasn’t on version 8.5.x, the lead technologist on the project had to work on an alternative update to secure the site or application, since there wasn’t a security update released for it. Figuring out an alternative process on the fly always introduces risk. It’s part of the value that we bring, that we have team members that have the expertise to evaluate that sort of thing. For example, we had one new client that was on an older version of Drupal 8 core. So one of our Senior Drupal Developers, Ryan Gibson, had to go in and determine what to do. He ended up updating Symfony itself to mitigate the risk.
Flynn: I’m guessing that we are going to recommend to that client that we update Drupal core for them very soon?
Shrop: Yes. The big takeaway is you’re lowering your risk of problems by staying on the most recent, up-to-date minor version of Drupal 8. Version 8.5.x is current and stable right now, so you should be on that.
Flynn: Why would a client not update?
Shrop: There are always dynamics. I hear lots of good excuses, and I’m not exaggerating, they are good, real reasons! The client is busy, the client has multiple workstreams, it’s hard - but it is getting to a point where I want to recommend even more strongly to clients that it is more expensive to not upgrade. It is going to cost them more when there is an update because we have these additional evaluation and update tasks. The whole point of Drupal 8’s release cycle is to spread the maintenance cost over years rather than getting hit all at once.
Flynn: And it introduces greater risk. A security breach is an order of magnitude more expensive than extra mitigation steps.
“When is the next version of Drupal Core coming out?”
Shrop: Version 8.6.0 will be released in September. Our teams are already starting to test the early versions of this release on some of our projects. If a security update comes out in September, we want all of our clients to be prepared by being on the currently supported version of Drupal core. That way, they will receive security updates.
Flynn: One of the nice things about the Drupal development community is that they provide the betas of the next version of Drupal core so you can get ahead of the next release, right?
Shrop: Yes. When the community starts releasing betas or release candidates, especially release candidates, you want to start testing ahead of time. If you have a Drupal site, you can get your developers to test. If you find a problem, it may not be with your site, it might be an issue with Drupal core and this is a great opportunity to contribute your findings back to drupal.org and help the greater community. There might be a security release weeks after a version comes out and you want to be prepared to implement it.
Flynn: It goes back to risk mitigation.
Shrop: If you are on, say, an 8.2 site right now, you’re on the higher risk side, unfortunately. We advise our clients that it is in their best interest to be on the current, stable version. It costs our clients more in the long run if they don’t update on a steady basis.
Flynn: So if you’re on an older version of Drupal Core, you might not get an easy-to-implement security update when a vulnerability is discovered?
Shrop: The quotes from the Drupal Security team I really want to emphasize are, “Previous minor releases will become unsupported when a new minor release is published,” and, “Any additional security updates for officially unsupported branches are at the sole discretion of the security team.” This is important to understand. For the SA Core 2018-002 fix earlier this year they provided release updates for older versions of Drupal… but they didn’t have to. In the case of the fix last week, they did not.
“What was the best gif exchange of the Drupal core security update process?”
Flynn: I nominate this one, from mid-afternoon:
“What story didn’t we tell yet?”
Shrop: I think we covered most of it. The last thing I’d put out there is for the technical folks reading this. You need to read the security advisories, join Drupal Slack, read what Acquia, Pantheon, and others are saying about each announcement. Then, you take all of that in and make your assessment of what actions you are going to recommend your organization take. This should lead your organization to a documented security plan that you follow. But, you know…
Flynn: “Update all the things”?
7 Ways to Evaluate the Security and Stability of Drupal Contrib Modules | Mediacurrent Pantheon Guest Blog
Security by Design: An Introduction to Drupal Security | Mediacurrent Webinar