The problem is resolved now and I would like to share the solution so i am answering my own question.
So the issue was i was getting wrong travel modes from my service itself and since these travel modes were not actually defined in my network so they were causing this issue.
I was getting wrong travel modes due to the face i am running this on a ArcGIS portal where we had a default setting due to which the travel modes were getting overridden with the actual ones which i defined.
I have to remove the default settings of portal (Under Organization -> Utility Services -> Directions and Routing -> Routing Utilities URL)
Now when I removed this from our staging portal I am getting the correct travel modes and the widget is working as well.