Despite the wildly successful flank attack and the Federal losses of May 2nd, the Union army still held a dominant position when dawn came the following morning. General Hooker's massive army still menaced the Southerners, boasting more than twice General Robert E. Lee's numbers. Stonewall Jackson's wing, now led by the flamboyant yet well respected cavalry commander Major General James Ewell Brown "JEB" Stuart, had not yet joined with Lee's small force of only about 14,000 men. If left isolated on the field, Lee's two divisions would remain vulnerable to the massive Union force, which could turn on each of the divided sections in detail. The Confederate Army would remain dangerously separated as long as the blue coats held their position on the crucial ground at Hazel Grove, pictured here, which jutted forward from the ground held by the Federals between the two arms of the Southern army.

However, Union General Joseph Hooker decided to withdraw his troops to what he thought was a more defensible line. After choosing to abandon Hazel Grove early on May 3rd, the Union Army established their artillery batteries at Fairview, to your left. You can still see the original gun pits, dug by the Union Army, covered in high grass in the mid-ground. The Confederate Army, no longer divided, drove the Union forces from this spot and captured Fairview. JEB Stuart would later write of this action, "As the sun lifted the mist that shrouded the field, it was discovered that the ridge on the extreme right was a fine position for concentrating artillery. I immediately ordered thirty pieces to that point, and, under the happy effects of the battalion system, it was done quickly. The effect of this fire upon the enemy's batteries was superb." [9] Once he possessed Fairview as well, Stuart would turn the Confederate guns in the opposite direction and, along with the infantry he now commended, proceeded to decimate the Union lines and destroyed the Chancellor House, headquarters of General Hooker.

During the Confederate artillery barrage of the Union line near Chancellorsville Manor, a shell exploded near the commanding General, likely leaving him concussed and rendering him temporarily unable to lead his men. Union Major General Darius N. Couch, commanding the Second Corps, stood near General Hooker as the Southern shell nearly made him the Army of the Potomac's commander. "The open field seized by Jackson's old corps after the Third Corps drew off was shortly dotted with guns that made splendid practice through an opening in the wood upon the Chancellor House, and everything else, for that matter, in that neighborhood. Hooker was still at his place on the porch, with nothing between him and Lee's army but Geary's division of the Twelfth and Hancock's division and a battery of the Second Corps. But Geary's right was now turned, and that flank was steadily being pressed back along his entrenched line to the junction of the Plank road and the turnpike, when a cannon-shot struck the pillar against which Hooker was leaning and knocked him down. A report flew around that he was killed. I was at the time but a few yards to his left, and, dismounting, ran to the porch. The shattered pillar was there, but I could not find him or any one else. Hurrying through the house, finding no one, my search was continued through the back yard. All the time I was thinking, "If he is killed, what shall I do with this disjointed army?" Passing through the yard I came upon him, to my great joy, mounted, and with his staff also in their saddles." [9] General Hooker would be taken by his staff to the Bullock House behind Chancellorsville for medical attention. [C] A short time later, General Hooker called for General Couch. "At the farther side of an open field, half a mile in the rear of Chancellorsville, I came upon a few tents (three or four) pitched, around which, mostly dismounted, were a large number of staff-officers. General Meade was also present, and perhaps other generals. General Hooker was lying down I think in a soldier's tent by himself. Raising himself a little as I entered, he said: "Couch, I turn the command of the army over to you. You will withdraw it and place it in the position designated on this map," as he pointed to a line traced on a field-sketch. This was perhaps three-quarters of an hour after his hurt. He seemed rather dull, but possessed of his mental faculties." [9]