What tipped off the Don?

I always thought it was his ability to read people in a situation.

What sets Vito Corleone apart from all of the other Dons in the Five Families is his masterful perception. The meeting was to quell the bloodshed and find some semblance of truce with Don Tattaglia, but Vito was quick to sniff out Barzini's eager interjection into the matter with regards to allowing drugs and his over eagerness in agreeing to peace. Remember how Barzini was the first one at the meeting to state they all agreed to the new peace and then spoke on Vito's behalf by stating that he agreed to allow narcotics into the Mafia economy? That should have come out Tattaglia's mouth instead.

It was pretty obvious Tattaglia was a pimp. He never could have outfought Santino.

Brazini was a mook.

I think Tattaglia came off as too simple minded and weak by accepting Vito's peace, kind off like Don Fanucci when he accepts Vito's deal in part II.

He knows Tattaligia didn’t have the wherewithal to pull off the events that we see, and he noticed how Brazini seemed to take charge during the meeting

In addition to the earlier answers, there also seems to be someone in the room passing messages between Tattaglia and Barzini (one of them whispers something to this messanger, who then goes and whispers to the other one). The camera shows Vito as his eyes follow the messenger, or flick between the 2 mob bosses like he's at a tennis match.
I think there's also a moment where Tattaglia looks to Barzini before agreeing to something, and only when Barzini nods does Tattaglia agree to this. The camera then cuts to Vito, who has picked up on this but has to hide his reaction at the time.
Plus, as Redban says above, Barzini seemed to control the meeting and was doing a lot of Tattaglia's negotiations for him.
Most people in that meeting might not have noticed this as they'd only be focussed on what was being said and what deals were reached. As Bourbonking says earlier in this thread, it shows how perceptive Vito is that, even in his recovery state and getting out of his sickbed to attend this meeting, he's able to read the situation so well.

It wasn't any one, single thing, it was, as others have partly noted, Vito's ability to read people. Then there was the fact that, Barzini, perhaps unconsciously assumed the seat at the head of the table, and presided over the meeting. Vito's network of informants would have kept him informed that Barzini was allied with Tattaglia. And then there is the fact that the whole thing just worked out to Barzini's gain. Barzini comes away with his position improved, with more power, increased standing, and Vito knows from that he's the puppet master behind Tattaglia. A man like Vito Corleone would have understood the wisdom of reducing any given situation to its bottom line. One of the keys to that lies in the old question cui bono? Who benefits? The way events played out, the Corleone and Tattaglia families tear on each other up, spending their strength. The Barzini familiy, who has been quietly standing behind the Tattaglia's all the while, come away with their power and influence increased, as Don Barzini plays the statesman and brokers a peace that restores order amongst the five families. Vito sees Don Barzini's affected display of magnanimity and diplomacy, and how Barzini comes away with greater status, and he understands: ah ha, he's the one who pulled this off.