Could the Equifax Hack Have Been Prevented by a Microservices Architecture?

Attention: The following article was published over 7 years ago, and the information provided may be aged or outdated. Please keep that in mind as you read the post.

When I heard that the Apache Struts open source framework played a role in the recent Equifax hack, I wanted to do some research to understand how it happened. Struts is a commonly-used Java framework that I have applied in the past. And I’m not alone in that: it is reported that in 65% of Fortune 100 companies currently implement Struts in some way.¹ ²

So, I did a little digging and performed a thought experiment asking myself the following question: “If Equifax had a pure-play Microservices Architecture in place, would it have solved the problem and subsequently prevented the hack?”

Bear with me on this, as I have to make some assumptions as I walk through this thought experiment to a conclusion.

How And What Happened

Equifax was transparent about how the hack happened: a known vulnerability in the Struts 2 framework (CVE-2017-5638)³ was not patched in a timely manner.

The vulnerability in question allowed remote code execution (RCE) in the file upload mechanism provided by the Struts Framework. There were actually two RCE attack vectors. One was a XML payload deserialization flaw, and the other allowed execution of a string expression. Both allowed execution of operating system commands.

The Struts framework is typically used to implement HTTP APIs, either supporting RESTful-type APIs or supporting dynamic server-side HTML UI generation. The flaw occurred in the way Struts maps submitted HTML forms to Struts-based, server-side actions/endpoint controllers. These key/value string pairs are mapped Java objects using the OGNL Jakarta framework, which is a dependent library used by the Struts framework. OGNL is a reflection-based library that allows Java objects to be manipulated using string commands.

That is what has been published so far about how hackers accessed potentially 140+ millions of customers’ data.

Examples

Here is an example of an OGNL string code expression that will produce a directory listing of the server:

${#[email protected]@DEFAULT_MEMBER_ACCESS,#file=new
java.io.File(#parameters.INJPARAM[0]),#files=#file.listFiles(),
#[email protected]@toString(#files),
#[email protected]@getResponse().getWriter(),
#writer.print(#files),#writer.close(),1?#xx:#request.toString}

Here is an example execution of a Linux command…. YIKES!!!

${#[email protected]@DEFAULT_MEMBER_ACCESS,
#[email protected]@getRuntime(),
#e=new java.lang.String[3],#e[0]="/bin/sh",#e[1]="-
c",#e[2]=#parameters.INJPARAM[0],#r.exec(#e),
#[email protected]@getResponse().getWriter(),
#r.print(#parameters.INJPARAM[0]),#r.close(),1?
#xx:#request.toString}&INJPARAM=touch /tmp/InjectedFile.txt

This String OGNL expressions could be placed in the Struts file upload mechanism. The flaw will execute OGNL string expressions in a file upload request header.

This exploit probably occurred by using the flaw to get user ID/password or to read the file directory of the server to find database connection credentials, etc. (Which if true, is another whole vulnerability discussion, but this is just a guess.)

These vulnerabilities were known for a long time, and all they had to do to resolve is update their Struts dependency to a version where these flaws were removed. If they had, the breach would probably not have happened. This bring us to the following question:

Why did Equifax not just apply the patch?

To answer this question, let’s do some thinking about the Equifax website as a whole.

It can arguably be deduced that the hacked Equifax site was a Java-based monolithic application, probably with some kind of connection pooling mechanism to a single database. Of course there could have been load balancers and couple of instances set up to assist with performance.

Here is a diagram of what it would hypothetically look like:

Given this architecture, applying a new version or patched Struts dependency would require the entire application be recompiled, built, tested, and deployed. Assuming the application is likely essential to the business (and bug fixes and new features development would be in flight by the development organization), the application of the patch and testing would have to be scheduled.

As the Struts framework code is the entry point for ALL UI features, the test surface area and time-to-test features could be time consuming. Applying an agile development process is supposed to free this up, but a monolithic code base can be a real Agile killer, especially when QA has to do its thing.

This is possibly one reason why Equifax IT did not apply the updated Struts dependency in a timely manner. I’m guessing other large organizations have been in the same boat, but unlike Equifax, they just did not get hacked.

So this leads me to the assertion…

Could the Equifax Hack have been prevented by a Microservices Architecture?

So, I’ll start with this disclaimer: a Microservices Architecture (MSA) is not a silver bullet. Additionally, I’m making a lot of assumptions about the Equifax application architecture. Maybe it is an MSA, but I doubt it.

Here at Keyhole, some of us have certainly helped build monolithic applications in the past. However, personally, I would not do it again. I believe a physically decoupled and distributed architecture is our best bet for agility and the ability to adapt to change. Applying a critical framework patch falls into this assertion.

I believe that a handful of key Microservices patterns could have helped to prevent this major security hack. Let’s go through each of them.

Single-Page Application

If the application’s user interface is implemented as a JavaScript single-page application, then the UI is physically decoupled from the server side code. This also eliminates server side generation of HTML.

Considering that the Struts vulnerability was the multipart file upload mechanism, the UI could disable this feature (along with the Struts config) without having to do a rebuild and deploy of the entire application.

Edge Controller

In a Microservices implementation, all requests for authentication from the UI are routed through the Edge Controller. A filter or rule could be applied to sanitize and block invalid headers that contain OGNL expressions until the patch could be applied.

Again, this could be done without building and deploying the entire application.

API Gateway

With Microservices, API calls from the UI are routed to the API gateway. It is responsible for invoking distributed services to carry out application behavior and to synthesize API calls into a UI specific shape.

Service-based Design

When application features and functions are distributed across multiple services, only the service containing the file upload behavior would need to be initially patched, updated, and deployed. This would help with speeding up the fix, or at least providing a speedy stopgap.

Bounded Context

Again, this is speculation, but let’s assume the Equifax application data is stored in a single database. If so, I’m guessing that’s a contributing factor to how hackers got all that personal information.

The Microservices approach is that a database, or at least schemas, are bound to a service’s domain context. This results in spreading an application data out across different data stores, which can make it more difficult for hackers to access all the data.

Containerization

Managing an entire software stack as an immutable executable artifact (i.e. App Server, Application Binaries, drivers, etc.) is essential in managing the explosion of elements introduced in an MSA. Also, quality assurance is sped up due to the entire software stack being tested, not just the application binaries.

So, in the Equifax hack, applying the new Struts dependency to the API Gateway once tested, could be moved from environment to environment with high level of confidence of success.

DevOps Platform

This is a big pattern. It involves the automated, continuous deployment of containers to environments, at any time or day. This is arguably the hardest element to apply, and probably the biggest factor for success. But essentially, the DevOps platform provides a way to build, manage, and deploy microservice elements to a infrastructure platform, in an automated way.

If Equifax had this type of platform in place, then the vulnerable service could be fixed. Then, assuming the platform had an a/b or canary-in-the-coal-mine-type deployment, the fixed service could be moved into producing without having to bounce servers, etc. And, more importantly, it could be done at any time. In other words, they would not have to schedule a production deployment in the wee hours of the mornings on a Saturday.

Final Thoughts

It is my opinion that a handful of key Microservices patterns could have each helped to prevent this major security hack on Equifax.

I’ll be honest in the fact that adopting a Microservices Architecture can be a huge and disruptive process. Additionally, the benefits of agility and tolerance for change that it provides will not happen right out of the gate.

But, it is also my opinion that Microservices can help make your platform more secure. The Equifax hack might provide another reason to consider moving in the Microservice direction.