502 Proxy Error / Uploading from Apache (mod_proxy) to Tomcat 7

14 Dec 2012

I've been working on a product lately where we need a reverse proxy from the Apache, serving our static content, to the Tomcat 7 where our API is. We're using mod_proxy to do the job and everything is hosted on Amazon's EC2.

Everything was working perfectly, but sometime ago we've started to see some errors, when uploading big files, like this:

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request POST /app/api/upload.

Reason: Error reading from remote server

Apache/2.2.22 (Amazon) Server at our_app_server.com Port 443

We started looking for solutions to this problem and tried a lot of things, like configuring the timeout on Apache's httpd.conf:

# Configuring the timeout
Timeout 2400
ProxyTimeout 2400

# Configuring the reverse proxy
ProxyPass /app http://localhost:8080/app
ProxyPassReverse /app http://localhost:8080/app

After we've changed the timeout, it got a little better, but didn't solve the problem completely. So, after a lot more googling, we found out that this error on Apache's log:

(103)Software caused connection abort: proxy: pass request body failed

is caused by a malformed server response and, to ignore it, we could set this option on our httpd.conf:

ProxyBadHeader Ignore

After this change, it finally revealed to us the real problem:

org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.web.multipart.MultipartException: Could not parse multipart servlet request; nested exception is org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. Read timed out
        org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:894)
        org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:641)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
        org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:312)
        org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
        org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
        org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:324)
        org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
        org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:324)

We started looking for the solution again and found a interesting Tomcat setting called disableUploadTimeout. People usually think you should set it as true, since the name is a little confusing, but if you read the docs you'll see that the default is true and you have to set it false if you want the timeout to be longer for uploads. Now this is my Connector at Tomcat's conf/server.xml file:

<Connector port="8080" protocol="HTTP/1.1" URIEncoding="UTF-8"
           connectionUploadTimeout="36000000" disableUploadTimeout="false"
           connectionTimeout="60000" redirectPort="8443" />

If you're having the same problem, I hope this post has saved your life... ;]