Transparency is a simple place to start. Do you have direct access to your accounts? These should include your Google Adwords accounts, Amazon accounts, Google analytics and so on. Can you track the dollars going from you to the ad programs themselves? If you terminated your agency tomorrow, would you have access to all of the accounts and data that have been created while you've been using them? If not, then that is a big problem.
Communication is also a major factor in transparency. Are they communicating with you as much as they said they would? Are they doing what they said they would since you spoke with them last? Do you feel that you are getting good advice from them over time related to your site and your campaigns? The communication between you and your online marketing agency should be fluid, transparent and open.
At the end of day it all comes down to results. If you're not getting the results you want than it's time to figure out if your agency is doing the work you're paying them for, if it's still the right kind of work to be doing, or if you need to go in a different direction.
Agency partners all have certain areas that they specialize in. Because of this, their advice can often be biased towards the kind of work they understand best, whether or not it's actually in your interest. If you are dealing with an agency that is predominantly a pay-per-click management agency, then they will probably tell you that pay-per-click should be your main focus. If you are dealing with an agency that focuses on social media, then everything in your campaign should be social media related, etc.
Each agency is going to try to sell you on the services that they have to sell, even if it is not necessarily what your business needs. It's up to you to develop the right overall strategy and then bring in agency partners to fill in the gaps around your internal staff. If you don't have the expertise on staff to build out your strategy then bring in someone who does. Asking vendors to develop your strategy almost always leads to lopsided campaigns and lower ROI over the long term.