不合法的查询字符导致服务500错误

Problem 

ActionController::BadRequest
Invalid query parameters: invalid %-encoding (miconazole%202%%20cr)

Rack::QueryParser::InvalidParameterError
invalid %-encoding (miconazole%202%%20cr)

ActionController::BadRequest
ActionView::Template::Error: Invalid query parameters: invalid %-encoding (%2Fsearch%2Fall%Forder%3Ddescending%26page%3D5%26sort%3Dcreated_at)


In a full stack apps, sometimes users manually edit query strings in the address bar, and make requests that have invalid encodings.

In a api apps, sometimes front end or mobile engineer forget to encoding the query strings, so they just give raw query strings to your apps, then they left it for years until it became very hard to fixed on the client side, and the management decide the server side should fix it.

These corrupted query strings are not that important, but users are receiving 500 server error pages.

What can we do?

I can offer two choice: Drop the query string OR Fix the query string.


A. Drop the query string 

I got this from this post right here.

First, make a middleware:


# You can put this everywhere
# In my case, I put this in app/middlewares/hande_bad_encoding_middleware.rb

class HandleBadEncodingMiddleware
  def initialize(app)
    @app = app
  end

  def call(env)
    begin
      Rack::Utils.parse_nested_query(env['QUERY_STRING'].to_s)
    rescue Rack::Utils::InvalidParameterError
      env['QUERY_STRING'] = ''
    end

    @app.call(env)
  end
end


Second, use this middleware before Rack::Runtime:


# config/application.rb

require_relative 'boot'

require 'rails/all'

# If you get NameError: uninitialized constant,
# you have to use require like this one:
require_relative '../app/middlewares/handle_bad_encoding_middleware.rb'

Bundler.require(*Rails.groups)

module YourAppsName
  class Application < Rails::Application

    ...
    # add this:
    config.middleware.insert_before Rack::Runtime, HandleBadEncodingMiddleware
    ...

  end
end


The result is, you drop the query string:


# from this
"https://your-url.com?query=alcohol%2070%"

# to this
"https://your-url.com"



B. Fix the query string 

I did some research and modify the middleware. This is the step:

First (optional), I use gem rack-utf8_sanitizer to remove invalid UTF-8 characters from the query strings. This prevents errors like "invalid byte sequence in UTF-8".

I personally prefer rack-utf8_sanitizer than utf8-cleaner because it is more robust, more option, and less bugs.

Second, make a middleware:


# You can put this everywhere
# In my case, I put this in app/middlewares/hande_bad_encoding_middleware.rb

class HandleBadEncodingMiddleware
  def initialize(app)
    @app = app
  end

  def call(env)
    begin
      Rack::Utils.parse_nested_query(env['QUERY_STRING'].to_s)
    rescue Rack::Utils::InvalidParameterError
      params = URI.decode_www_form(env['QUERY_STRING'])
      query_string = URI.encode_www_form(params)
      env['QUERY_STRING'] = query_string
    end

    @app.call(env)
  end
end


There are many ways to encode and decode url:


URI.escape
URI.unescape
URI.encode_www_form
URI.decode_www_form
URI.encode_www_form_component
URI.decode_www_form_component
CGI.escape
CGI.unescape


I've tried them all, and I picked the best for me, which are URI.encode_www_form and URI.decode_www_form. Probably you have to pick which is best for you.

Third, use this middleware before Rack::Runtime:


# config/application.rb


require_relative 'boot'

require 'rails/all'

# If you get NameError: uninitialized constant,
# you have to use require like this one:
require_relative '../app/middlewares/handle_bad_encoding_middleware.rb'

Bundler.require(*Rails.groups)

module YourAppsName
  class Application < Rails::Application

    ...
    # you add this for the rack-utf8_sanitizer,
    # as described in the gem's readme:
    config.middleware.insert 0, Rack::UTF8Sanitizer
    # add this:
    config.middleware.insert_before Rack::Runtime, HandleBadEncodingMiddleware
    ...

  end
end


The result is, you fix the query string:


# from this
"https://your-url.com?query=alcohol%2070%"

# to this
"https://your-url.com?query=alcohol%2070%25"



Final Word 

If you have any problems or questions, let me know in the comment.

If you are helped, I am glad I can help you.


RSpec.describe HandleBadEncodingMiddleware do

let(:app) { double() }
let(:middleware) { HandleBadEncodingMiddleware.new(app) }

it "request with query string is unchanged" do
expect(app).to receive(:call).with("PATH" => "/some/path", "QUERY_STRING" => "name=fred")
middleware.call("PATH" => "/some/path", "QUERY_STRING" => "name=fred")
end

it "request with no query string is unchanged" do
expect(app).to receive(:call).with("PATH" => "/some/path")
middleware.call("PATH" => "/some/path")
end

it "request with invalid encoding in query string drops query string" do
expect(app).to receive(:call).with(
"QUERY_STRING" => "",
"PATH" => "/some/path" )
middleware.call("QUERY_STRING" => "q=%2Fsearch%2Fall%Forder%3Ddescending%26page%3D5%26sort%3Dcreated_at",
"PATH" => "/some/path")
end

end

阅读量: 596
发布于:
修改于: