Testing Javascript bug fixes in production environment

Overview

This is an elaborate post on how I go about testing a Javascript bug fix directly in production environment. I believe the method that I am about to explain is very useful in that it enables one to quickly test a fix without going through below cycle of:

  1. Reproduce the bug in local.
  2. Fix the bug.
  3. Test the bug fix in local.
  4. Deploy to environment for verification.

The technique that I am going to enlist below is possible mainly due to one of the features enabled by JavaScript language called “monkey patching” - the ability to change the definition of a function while or after browser parsing of the function.

Setup

We need a sample HTML/Javascript project with a deliberate “bug” introduced so that we can see how to dynamically patch a fix and test if it works or not. Note that the technique covered here should work on all modern browsers but specifically it is more useful for IE6+. Even though modern browsers have improved the tools required to effectively debug UI apps I still think the below techniques are complementary to them rather than a replacement.

Debug

The below are the few heuristics that I follow to reverse-engineer the root cause for an UI issue. I am assuming that you are thrown into the deep end to fix an issue with neither having access to code base nor any knowledge/memory about the functionality. If not then still the heuristic is helpful.

Ok, here it goes using the above sample project as a bug:

  1. Follow the steps to recreate. (In the above project, the page must display a circle but only a semi-circle is visible.)

  2. Once you had recreated the bug, check if the issue is browser specific. If not browser specific, you can use any browser to debug further. Else, use the browser specifically causing the bug. (In the above project, it is happening in more than one browser so I choose chrome to debug it but it can be any browser. At work, I would have chosen IE.)

  3. This is the important part: Isolate the element that is not behaving as expected a.k.a. the “bug”. (In the above project, the svg element is not behaving as proper.)

  4. Open developer tool (in IE press F12. For other browsers, check their documentation on how to open its developer toolbar).

  5. To search the code in js, first start with the element and check if it is using any id or class. If yes, then search the Javascript file for that id or class or element name. Else if you don’t find anything on the element, check if there are any onload events declared. Or check the css file that is styling the element and search the js files that are loaded after or before the css file to limit the number of js files that you need to look at to debug further. (In the above project, I found the onload event and added a breakpoint there. I could have also searched for element name “svg” because there was only one such element.)

  6. Now we need to find out the code that is the precursor of the “bug”. That is, the code which executed correctly just before the “bug” happened. (In the above project, it is the onload handler)

  7. After adding the breakpoint via dev tool at line #2, I ran the page again and this time the page stopped at the breakpoint location line #2. From here I need to slowly step through the code to understand where it is breaking. (In the above project, I guess it is breaking at line #9 where the d3 js is creating the circle)

  8. If you find the precursor but still are not able to add breakpoint because the js is dynamically generated, then find the js file that is injecting the dynamic script. This can be found via network tab or the html source that has the list of js files that it loads. Then put a breakpoint just before and after this dynamic creation of script. After the script is created, it will appear in the sources window and you can now go and add additional breakpoints that you need. Sometimes (esp. I noticed this in IE) even if you add a breakpoint in the generated script, the browser may not stop at that line. Check the “Fix” section on how to handle this thornier case.

  9. Enable cache (temporarily while you are testing) so that when you reload the page again, the dynamic scripts are retrieved from cache with your added breakpoint intact.

Fix

There are only two techniques you need to test your fix directly in the developer tools/console.

A. While the browser is waiting at the breakpoint line #2, open console in dev tools and paste the below code and click run or press enter. This code effectively redefines the drawCircle() function to the bug-fix version. Now go and run the debugger to completion.

function drawCircle() {
 var container = d3.select("body")
		.append("svg")
		.attr("width", 500)
		.attr("height", 500);
 container.append("circle")
  .attr("cx", 20)
  .attr("cy", 20)
  //note the change from original 30 to 20
  .attr("r", 20)
  .style("fill", "black");
}

In my case, this fix worked and I am done. For complex cases, you might be doing multiple iterations on the fix before you finally conclude that it works in all cases.

How cool is that!

B. Now, for the corner case when your breakpoint may not be working esp. for dynamically injected scripts, when the browser stops at the breakpoint just after dynamic script injection call, you can redefine the function inside the dynamic script with the same function definition except at the first line inside the definition add the “debugger;” statement.

function drawCircle() {
 /* 
Yes, this is the single most important debugging aid that almost nobody is aware of and the best part is this works on all the browsers (<https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/debugger>).
*/
 debugger; // debugger statement

 var container = d3.select("body")
		.append("svg")
		.attr("width", 500)
		.attr("height", 500);
 container.append("circle")
		.attr("cx", 20)
		.attr("cy", 20)
		// note the change from original 30 to 20
		.attr("r", 20)
		.style("fill", "black");
}

Then you need to manually invoke the “bootstrap” function of the dynamic script via console. (In the above example, you need to call “drawCircle()” via console)

drawCircle();

This will stop the browser at this line and then you can add more breakpoints or continue to step-into/step-over via the debug bar.

Caveat1: Remove “debugger;” statement before you check-in the fix. It should only be used as a debugging aid.

Caveat2: Note that this dynamic injection of a function redefinition will not work if the browser already started executing within the function that you are redefining. So always put the first breakpoint just one call stack below the function that you are going to redefine.

Note: In order not to repeat step B every time the browser dynamically injects the script while refreshing the page, you can enable caching via dev tool.

Conclusion

This has turned into a longer article than I anticipated but I hope it helps someone to effectively debug their frontend apps. The method captured above are browser agnostic and has personally helped me save many person hours in debugging and effectively testing the fixes.
Happy debugging!

Few thumb rules for speeding up onload API response times. And some caveats.

Overview

Performance tuning is always one of my favorite software engineering activity. But it is filled with wrong turns and potholes for the unwary. Also, in my opinion, benchmarking the system the right way doesn’t seem to be an average engineers strong suit.

Anyways, below are the few thumb rules to improve onload API response times. Since software engineering is all about tradeoff’s please take the below suggestions with a grain of salt and always validate them for your own environment. So, YMMV1.

Enough rambling! Here we go:

  1. If two or more API’s are independent of each other, then initiate those calls asynchronously from the browser.

    Caveat:

    Modern browsers initiate parallel calls while loading static assets and async has become the de facto mode to make Ajax calls. But this also means the web API tier is going to be hammered with more calls per second than before. For directly exposed API’s one can solve this by using rate limits. For internal API’s we rate limit where it makes sense in addition to performance testing to make sure if the increased load plus buffer can be handled by the system. But even if web layer can handle all that load, it’s downstream could still be affected by heavy load and cascade that failure back to web layer. In this case, the recommended strategy is to use Circuit Breaker 2 at web layer to protect itself from any downstream system failures.

  2. If two or more API’s are dependent on each other so that one call’s response becomes another call’s request, then just make one call to a facade (simplified API interface) from UI and let the web tier take care of orchestrating the multiple dependent calls before returning the response.

    Caveat:

    Now, UI layer doesn’t have to worry about all the messy details of orchestrating multiple API calls at the expense of complicating the web API layer. Web API layer would have to spawn threads for each of the API to be called and wait on all the response before sending back the response to UI layer. Using threads implies using thread pools to make sure we are not over utilizing the OS resources. Also the Web layer must protect itself from the cascading failure of it’s downstream using Circuit Breaker but this time any of the multiple threads could fail. So now we also have to handle those extra types of exceptions like InterruptedException etc.

  3. Preload the page’s data in a cache before the user lands on the page.

    Caveat:

    The downside is, the web API layer has to depend upon an extra caching layer which introduces its own set of complexity. Web API layer could call the caching layer which in turn call downstream when data is not present in cache and populate it before returning the same data to web API layer. Even though this is a common pattern, we cannot help but notice and handle the new failure scenario’s that are exposed by this technique.

Conclusion

Performance tuning is a rewarding work if you know what you are doing. Else, it is the software engineering’s equivalent of shooting yourself in the foot.

Footnotes

1 Your Miles May Vary

2 Circuit Breaker

Two succinct code refactorings in Java.

Overview

Given a long career in Java programming one ought to have encountered many surprise techniques in programming language syntax that reduces lines of code without sacrificing the correctness of the program. I have certainly come across many such gems in my career but below I am going to show two techniques that I particularly like.

1. Instantiation and member function Initialization

// verbose: traditional way of instantiating the
// subclass and then
// setting the individual
// member values in separate set of steps
Person a = new Student("abc123");
a.setAge(24);
a.setFirstName("Abra");
a.setLastName("Cadabra");

// succinct: Instantiate
// and set the member values at once
Person b = new Student("abc123"){ {
    setAge(24);
    setFirstName("Abra");
    setLastName("Cadabra");
} };

2. Sorting Collections

//Initialized using the succinct technique from 1.
List<Integer> ages = new ArrayList<Integer>(){ {
   add(12);
   add(32);
   add(5);
   add(3);
} };

//verbose: sort ascending order
Collections.sort(ages, new Comparator<Integer>(){

    @Override
    public int compare(Integer o1, Integer o2) {
        return o1.compareTo(o2);
    }
});

//verbose: sort descending order
Collections.sort(ages, new Comparator<Integer>(){

    @Override
    public int compare(Integer o1, Integer o2) {
        return o2.compareTo(o1);
    }
});

//succinct: sort ascending order
Collections.sort(ages,
 Collections.<Integer>reverseOrder
 	(Collections.<Integer>reverseOrder()));

//succinct: sort descending order
Collections.sort(ages,
 Collections.<Integer>reverseOrder());

Epilogue

That’s all folks! I hope you enjoyed reading about these two techniques as much as I enjoyed writing.

Java code quality notes from the trenches.

Overview

Whether you are a newbie or an old hat Java programmer, you can save yourself some embarrassment by avoiding the below most common programming errors.

  1. Not serializing objects (and nested objects within) that are stored in HttpSession.
    • See Effective Java Second Edition (EJ2), Item 74
  2. Throwing generic exception instead of application specific exception.
    • See EJ2, Item 61
  3. Ignoring or swallowing a thrown Exception while re-throwing.
    • See EJ2, Item 63, 65
  4. Not overriding hashcode() when equals() is overridden.
    • See EJ2, Item 9
  5. Mutating static member via an instance member.
    • See EJ2, Item 15 and Item 66

Epilogue

All of these will be caught by any good code quality scanner. Please use one.

It is time to bring the tools.

What ?

I have written many utility tools for my side projects and they are all scattered across my multiple personal machines. Come this year, I decided to throw in all the personal scripts that I had written into github repository so that I can stop re-inventing the wheel again when I needed them.

How ?

I have created a living repository in github called devops-tools (https://github.com/AskDrCatcher/devops-tools). This repo contains all the current and future scripts that I might reuse again. Note that it may or may not be useful for others but enough time is spent on making the scripts simple and easy to read and modify. Some scripts are written in javascript, some are in perl and some could be written in a language that might not be as well known like tcl or nimrod. I am a computer language aficionado and have personally used many esoteric languages than I care to keep track. One day, I will create my own programming language to stand next to the other 700 programming languages.

Why ?

What if there are languages that can express same logic in half the code size ? These are called scripting languages and are ideal for creating quick tools whether one-off or not.

License

All scripts in this folder are of Apache 2 License.