While in HKG we stayed in the Tai Wan hostel. The hostel was right in the heart of Kowloon and a street away from the harbour and Hong Kong island. It really is as central as it gets on a budget!
Double room cost
For a double room with private shower/bathroom we paid $480 HKD for 2 of us for 2 nights. Be aware though, the prices rocket during the weekend ($580 for Friday night) but in the week they are cheap
Facilities and staff
Staff are helpful but you have to ask for a towel, they don't leave one in your room. There is fresh water supplies at reception and free wifi. This coupled with it's central location makes it a really inexpensive way to see HKG.
Downside
There are some pitfalls to this place (Chungking Mansions) the many sellers outside the front door trying to sell you suits, trips and other illegal stuff gets pretty annoying but on a positive side you can get a mean curry from their shops....
We booked through http://www.hostels.com/hostels/hong-kong/twh-tai-wan-hotel-hong-kong/17642