This means that Ali is not infallible because he has done bey’ah with two oppressive Nasebi kafirs, and has verified them through such; this is a flaw in his infallibility and a form of helping oppressors in their oppression, something that an infallible imam would never do. It is either this, or the Shia have to accept that what Imam Ali (as) did was totally correct; because Abu Bakr and Umar were both just and mu’min and righteous khalifahs. That means that when the Shia refute them as kafirs and their caliphate, they have opposed their own imam! So we have a dilemma here; do we follow the way of Imam Ali (as), or his Shia who are practically opposing him?!
First of all, although the Shia has some criticism towards the khalifahs, they don’t see them as kafirs; the same way they don’t consider any of the other Sahabah as kafirs. In dealing with the khalifahs, the Shia follow the footsteps of the imams, especially Imam Ali (as), in how they would deal with them, not anyone else, because the Shia consider themselves the followers of their infallible imams. 
Secondly, what was of dire importance to Imam Ali (as) and the other imams, to the extent that they sacrificed everything, even their lives for it, was the preservation of Islam. It was for the preservation of Islam that Imam Ali (as) would cooperate with the khalifahs and give them the best consultation he could, whenever needed. In many a time, he would send off his children to wars that were at the command of the khalifahs, the reason being that the imam didn’t want anything for himself; anything he wanted was for Islam’s flourishing and since, in any case, Abu Bakr and Umar were the heads of the Islamic state then, and opposing them, in that sensitive and pivotal time in which Islam was expanding its boundaries and there were conquests, and Islam had many adversaries, and the Roman and Persian empires were great threats, Imam Ali (as), through his deep insight, took the right stances, not making Islam any more vulnerable than it already was, and in this way, putting an end to the dreams of the enemies of Islam taking advantage of the circumstances.
Third of all, what was asserted in the question about Imam Ali (as) accepting the caliphate of the khalifahs, isn't correct, because on top of all the objections and oppositions he had before the martyrdom of Lady Zahra (as), in the following years, especially during his own rule, he would strongly criticize their actions and would question their caliphate itself. The third sermon of the Nahjul-Balaghah, known as the sermon of Shiqshiqiyyah, is clear evidence of this claim:
In this sermon, the imam (as) says: “…Then I began to think whether I should assault or endure calmly the blinding darkness of tribulations…”  .
Therefore, neither does the Shia consider Abu Bakr and Umar as kafirs, nor did Imam Ali (as) totally accept them and their caliphate; all he did was, when it was to Islam’s benefit, to cooperate with and give help and consultation to them.