I don't think, that you're doing something particularly wrong, but for a big project like linux you have to keep the changes as isolated as possible, otherwise

I don't think, that you're doing something particularly wrong, but for a big project like linux you have to keep the changes as isolated as possible, otherwise it will be quite hard to identify the causes, if there're problems. So it's better to base your changes on the commit of a linux release, which has been tested quite a lot, which has some kind of defined state.