A simple question with a complex answer! It is complex because security issues arise from a variety of sources: your code, your server, the other things running on your server, the users, etc. While Joomla itself is relatively secure, you may still experience problems if the server is compromised or if a user gives up a password.

First of all you should check the forums at Joomla.org - There is a wealth of knowledge to be gained from the highly experienced Joomla Core Developers; however we will provide a general overview of the more common steps to take and things to avoid when building and maintaining your new Joomla site.

SERVER SECURITY -

• You should check with your hosting company and find out the steps they have taken to secure the server and unauthorized access to your website for the server level. 
  
• Avoid cheap shared servers: Be sure users on your shared server can't view each other's files and databases, for example through shell accounts and cpanels.
• Performance matters: Choose a host that limits the number of users per machine and the average CPU load per machine to some reasonable number (depending on hardware). Be sure they proactively move user sites as needed to balance load. Check the number of domains on a server using reverse IP lookup.
  
• Data center: Know your hosts data center. Check the data center infrastructure, such as redundant Internet access, hot swappable backups, full daily backups, environment and access controls, emergency generators, etc.
  
• Grow with your site: As sites grow in complexity, resource requirements, and security requirements, they may need to be moved off of a shared server environment and moved into a dedicated server which offers the best possible security and performance.  R&B offers dedicated managed servers at an affordable price that can be optimized specifically for your Joomla site. (Check the menu for R&B Hosting Options),
• Backups:  Be sure that your hosting company is maintaining full backups of your completed website; however never rely totally on your host.  Make a complete copy of your website files and store them for save keeping.  When major changes are made replace this copy with a new one.  Keep regular backups of your database.  Typically due to frequent content changes, these should be made a more frequent basis (daily - weekly or monthly depending on frerquency of changes).

JOOMLA! CORE-

• Be sure to check your Joomla! installation's overall security and health by Using the free Joomla extension, Joomla! Tools Suite (JTS), which is a Joomla! environment audit, maintenance and diagnostic application written in PHP. The JTS suite of tools can diagnose, report and advise on common installation, health and security issues, including performing several common performance and recovery actions.
  Project Home: http://joomlacode.org/gf/project/jts/
  
• Be sure to change your Joomla! default admin password.
• Be sure your register Globals are off - If you hosts server is defaulted with them set of "On" you can turn them off for your website using your .htacess file - Merely place this line of code within the .htaccess and then recheck your installation.  You will see that your Globals are not closed - php_flag register_globals off