If it's two season use and you want more room, consider a quilt. Also, think of the sleep system as a whole rather than the parts. So a better mat requires less of a bag/quilt. I find it harder to get it right in the Summer than when it's cold and a quilt makes regulating temperature easier, you simply have more body uncovered. I've never had any qualms about using down, it just needs a little care, the newer Hydrophobic down, as mentioned above, make it even easier. However, the synthetics have also improved in the last few years, so I wouldn't rule either out. Lastly - Two seasons, you're not going to freeze if it goes wrong.
