Jimmy Fallon has around 700,000 viewers, Steve Colbert around 680,000, and Jimmy Kimmel is always being beaten by the others shows as he only brings in around 510,000 (Conan has almost as many viewers on TBS). Most television shows bring in 3.5 million viewers. 60 Minutes has 8.6 million viewers and The Big Bang Theory has 13 million.
The late night shows have become programming fillers. In the days of Carson, Leno, Letterman, etc., they brought in a few million viewers each and the networks gave the shows more attention, but now with all the competition at night, and changing viewing habits in the younger generation, they just aren't that successful. One report I read showed advertising revenues for late night shows were down over 30% in the last seven years.
The bottom line is these shows just don't have the political impact that some might think. Hosts like Colbert and Kimmel are just playing to their audience; singing to the choir, so they love hearing Trump trashed. Trump shouldn't have even gotten involved in giving them attention with his angry tweets (he has around 24 million followers); as the late night shows just aren't worth it and I'm sure most of his followers don't watch these shows anyway.