At what point can should you walk away from family?

At what point can you say this is to much to deal with and walk away from your family? My husband and I have been together for about 5 years now, and we both had wonderful relationships with both of our families. When the economy tanked, we both lost our jobs, and decided to go back to school. He is 46 and I am 30. We now have a house and new car. We have always lived on our own, but barely made ends meet. Now we are able to have things we want, and go places we want. All of a sudden in the past year his family talks to us very little, and when we do see them there is always an argument about something stupid. My family is now starting to be the same way. Every time I call to take my grandmother to lunch she is busy with my cousin or her children. When we do come around their is always an argument. This last time over which flip flops were my daughters and which were my cousins. At the end I took my daughter and left with her barefooted, and just went to WalMart and bought her another pair of shoes. Why is it now that we have worked hard, made sacrifices, and are doing better than ever our families are having so many issues? This last visit home his drunk brother in law started calling him a punk and me a stupid bitc*, and after a verbal altercation we packed our children up and left. Not to mention we were 8 hours away from home and it was midnight. Except for a year ago we were just Joe and Pat, now we are all that. We don't brag about what we have, if fact they were mad that we didn't tell them we had a new car? They always point out what we should be doing, instead of what we are doing. They try to tell us how to raise our kids, and they do not even have any. At what point do you just say enough and walk away. We are tired of the drama that both families are bringing. By the way we are a bi-racial couple, and the only one in either family. It never was a problem before, but seems to be now. Are they just looking for reasons to pick an argument? Help please before we loose our minds. They are family and we love them, just not all the drama. When we see them we just want to chill and have fun with them, not have arguments.