You will indeed have an immune response against the chimp virus (though you'll probably get some reaction against the micelles that the Pfizer vaccine is delivered in, though unlikely as much) – there's an issue that you are, in fact, co-vaccinating against both COVID and chimp adenovirus, so you might become immune to future doses delivered in the chimp virus vector. This is a known for any viral vector platform and they can tweak the delivery vector virus.
The majority of people who get either vaccine have no or a very minor reaction.