Daniel Wilkins netted the only goal of the game mid-way through the second half after firing into the bottom corner with help from a clever knock down by Alex Curran.
Taddy started the contest brightly however and got control from the beginning building attacks up from the back.
They were kicking themselves in the early stages after somehow not converting a goal mouth scramble.
Cam Murray threaded a through ball in behind to Tom Corner who's powerful close range strike was well thwarted by Colne stopper Hakan Burton.
The visitors couldn't get rid of the ball however as Murray and then Corner again somehow couldn't break the Colne back-line and force over the line.
Michael Ingham in the Taddy goal was called upon a couple of times in the first forty five and did superbly to keep out both Phil Dean and Daniel Wilkins.
Full-back Chris Howarth was providing an attacking outlet down the left-hand side after getting round the back several times in the cup tie but on this occasion Josh Greening couldn't quite squirm through.
Josh Barrett and Andy Milne were looking confident and comfortable at the back for Albion and Barrett created a great chance for his side after setting Josh Greening away in behind.
The attacker cut inside onto his weaker right-foot and curled narrowly off target.,
Half-Time: Tadcaster Albion 0-0 Colne
Eight minutes into the second half and defender Josh Barrett was adjudged to have handled inside the penalty area which saw referee Neil Guest point to the spot.
Michael Ingham is a very big frame in between the sticks but penalty taker Michael Fish didn't even test the former Northern Ireland International after clipping the outside corner of the bar.
As Albion pushed increasingly harder for the opening goal, this left more of the desired space in behind for their opponents and they were starting to exploit it.
Popular figure Danny Frost was introduced before the hour mark and he soon forced Hakan Burton into making a save.
Colne were still looking dangerous on the break as substitute Chris Anderson broke the offside trap before cutting inside Andy Milne but again, Michael Ingham was outstanding in goal to save.
However, Colne did score the one and only goal of the tie in the 68th minute.
Phil Dean's cross from the left was intelligently guided down into the path of Daniel Wilkins by Alex Curran and Wilkins emphatically found the bottom corner finally beating Ingham.
This left Taddy with no choice but to chase the game even more and they were appealing for a penalty of their own when Aiden Savory's shot bounced up but referee Neil Guest assertively waved away the protests.
The gaps really started to appear in the final five minutes as the tie could have easily gone either way.
Goalkeeper Michael Ingham again was something else as he astonishingly kept out four shots from point blank range.
One of which saw 18-year-old Curran race clean through on goal leaving a two v two in Taddy's half.
Curran squared to substitute Adam Morning who took a touch to compose but still couldn't seal it as Ingham saved to his right.
Taddy had two gilt edged chances of their own in the dying stages and both fell to Tuesday night's hat-trick hero Aiden Savory.
Firstly, Frost tried to barge down the Colne defence before it fell kindly who sliced over.
Then in the last minute, another scramble broke out in the penalty box from a long Josh Barrett throw.
Danny Frost had a couple of bites st the cherry but couldn't find the half a yard to pull the trigger, Andy Milne had a go too before Jake Ellam slid in Savory again who drove narrowly over the mark.
Full-Time: Tadcaster Albion 0-1 Colne
Tadcaster: Ingham, Hardy, Howarth, Davidson, Milne (c), Barrett, Murray, Mycoe (Sellars 70), Corner (Frost 58), Savory, Greening (Ellam 81)
Unused: Roper, Akeroyd, Crouz
Colne: Burton, Taylor (Gregory 51), Pugh, Coleman, Lynch, Nangle (c), Dean, Curran, Wilkins, Fish (Morning 66), Morrison (Anderson 56)
Unused: Hanley
ATT: 200
Referee: Neil Guest