Java Servlet Filter

## Intro Our company provides web analytics services to customers. We require development of a module or a library that can be added to java based websites. This module should be easy to add to websites and will enable integration between our service and the websites of the customers. The module should allow two main functionalities: 1. Add configurable tracking code to the html response. 2. Capture, store and make available the response as will be detailed below. ## Technical components • Configuration. • Filter: Detect html, add tracking code, store content in cache provider. • Cache providers: Store buffers in memory or database, manage expiry. • Request handler: Returns cached data if passed matching identifying token and passing security checks. • Debugging/troubleshooting page. ## Deliverables ## Configuration • Top and bottom tracking codes: two blocks of html code to add to the response as well as regular expressions to control where the codes are added and when they shouldn’t be added. See examples of the format at: [login to view URL] (“ScriptsFile??) • Some sort of control over what parts of the website the filter is installed on. • What caching provider to use and the settings of each provider. o Limit on the number of items to keep in cache, or limit on the amount of time for which to keep the data in cache. (“MaxCachedPages??, “MaxCachedSeconds??) o Where to keep the data in. Could be address of a memory location, disk location or database location. • Whether to remove the cached data after the first request or let it expire. (“DeleteAfterPull??) • What IP is allowed to request data. (“AllowedAddresses??) • Whether debugging is allowed and to what extent. (“AllowDebug??) • Whether to log info, warning, error messages and where (to what path). • Name and value of a special cookie that causes the Filter to ignore the request (“DoNotProcessCookieName??, “DoNotProcessCookieValue??) ## Code samples [login to view URL] - open source [login to view URL] - closed source ## Detailed specification of the components ### Filter The filter should monitor all the requests to the site (preferably only some of the urls/pages could be configured for monitoring). When a request of type text/html is found, the module needs to: • Check some configurable cookie for name and value match and skip processing if matched. Referred to as “DoNotProcessCookie?? in the samples and below. • Generate a cryptographically strong 128bit unique id for the request. The unique id will be referred to as just “id?? from now on. • Collect the response that was meant to be sent to the visitor. • Save the response in the configured cache provider assigning it to the id. • Note: you should consider if to use strings or binary format and should support proper operation with different encodings of the page. • Inject the top and bottom tracking codes into the response while filling in the id in the designated location of the bottom tracking code. Injection is conditioned with a regex and so is the location of the injection. ### Cache provider Stores the data captured by the Filter in some type of temporary storage. There should be different providers to choose from. On insert or at some more optimal schedule, the provider needs to remove old data items to control the size of the cache. When requested from the Request handler, data should be returned and optionally deleted from the store. ### Request handler A handler that is able to process request to a predetermined url, similar to: http://domain/[login to view URL] where xxxxxxxxxxxxx is the id of the data. The url may refer to a physical file on disk “[login to view URL]? or it may be a virtual url that the Filter would handle as the Request handler. The handler should • Verify that the request comes from an allowed IP range. The range is configurable. If the requestor is not allowed the handler will return “http forbidden?? error code. • Request the id from the cache provider. • If the id is incorrect or missing, or if the provider doesn’t have the data for this id, the handler will return “http not found 404?? error code. • Return the data to the requester. Encoding should be correct and page should be possible to render as in the original. ### Debugging / troubleshooting page Just as with the Request handler the url may refer to a physical file on disk “[login to view URL]? or it may be a virtual url that the Filter would handle as the Request handler. Access to this page should be configurable. The page should • List the current settings. • List the state of the DoNotProcessCookie value and how it affects the “skip filtering?? status. • List the size of the cache. • Show latest 10K of the log if log enabled. • On clicking of a button, perform a “capture/replay?? test using two frames with synchronized loading. The first frame will load a page in a way that will cause the Filter to capture it and add it to the cache. The second frame will load the Request handler to show that the process worked. The page should contain a random value as part of the body. See example in the PHP implementation. ## Additional requirements • Logging should be included and should be controlled via the configuration. • Should be multithreading safe. Specifically a case when two separate requests are processed at the same time. • Should support high load, have low impact on the site and be implemented in an optimal way. For example, pre loading configuration that is located in disk files, etc. • The scripts file ([login to view URL]) should be loaded to memory and should be monitored for changes and reloaded if necessary. • Should cooperate with others filters that may be installed on the site. • We will need written instruction (targeted to developers) about how to deploy this module including setting up java web server with a dummy java site on a completely new windows machine. ## Questions Please answer when bidding! ### About the purposed solution • Configuration, how easy will it be? • What providers will you be able to implement? Preferences to providers using memory. • What versions of java will be supported? • Will we be able to distribute the component in binary form or only in source code form? • What control will we have about what parts of the site are filtered? ### About the developer • Have worked with filters before? • How much experience writing servlets? • What experience with web security? • How much experience with ASP.NET? • How much experience with PHP?