Split cycle air conditioners are the most energy efficient solution, and will cost you the least to run over winter under most circumstances. These are a bit out of grasp for some so generally electric fan heaters are the most accessible, and the cheapest heaters you can buy, although their running costs can get high.
To balance the outright cost and the running costs look for heaters with a lower power source wattage. This number indicates how much power the heater needs, and you can do some smart calculations to figure out how much it will cost you to run over the season.
To give you some indication, we've done some math for you already. During the previous financial year, the average kWh cost was $0.14 across Australia. Running a 2000W heater would therefore cost around $0.28 per hour. As a point of comparison, a 60W air purifier costs less than $0.01 per hour at this same price. If you run your heater for eight hours per day, over the course of our 92 Winter days that comes to an extra $206 tacked onto your bill.
Using this same data we calculated how much you can expect to pay over Winter depending on the heater's wattage. These are estimates based on averages, but it should give you an idea as to the cheapest heater to run in Australia.