Start with reading Writing a payment method module post in Drupal Commerce documentation.
General guidelines
User experience review
These issues pertain to the experience of the merchant or site builder while using the module. Critical issues are those that will severely impact the ability of merchants to set up and use the module. Not critical issues are those that would increase conversion and user happiness if fixed, but you think can be postponed until after launch.
Functionality review
Functionality tests include using the module as it is intended to be used and confirming that it does what it is supposed to do. These tests are done by installing and using the module. Critical issues are those where a feature either fails to work, or works so poorly as to be useless. Not critical issues are those where the feature is somewhat flawed but a merchant would feel comfortable going live with the feature anyway.
Architectural review
Architectural review looks to see how the module was built. Does it use the Drupal hook system in the right way? Does the module abuse any Drupal patterns (eg storing too many things in variables; too much functionality in hook_init(); improper use of blocks or content types; lack of caching where appropriate)? In short, does the module come close to the architecture it would have if Commerce Guys built it?
Security review
Review of the use of Drupal security tools and vulnerability to common exploits. For example, are database inputs sanitized? Is check_plain() used? Are there appropriate permissions in the menu router items?
Useful resource: Security Review module
Drupal Commerce
Whenever a payment is attempted, a payment transaction should be created that references the order for which payment was attempted. This includes transaction attempts that failed, that require further action, or were processed successfully as reflected in the transaction's status: Failure, Pending, Success.
Provide detailed installation and configuration instructions in INSTALL.txt file as well as on module project page (or alternatively on module documentation page, linked to from module project page).
Separate configurations for sandbox (test) and live?
Option to log all communication (in the PCI compliant way).
Proper implementation of hook_settings_form() callback (see hook_commerce_payment_method_info() documentation for more information on payment method callbacks).
Payment function (immediate auth and capture) using the RESTful web service method.
Void/refund support through the Drupal UI. A site admin should be able to refund or void a transaction on an order from the order edit screen.
(optional) Card on File integration using gateway's card tokenization service.
(optional) Support for recurring payments.
On-site payment methods
Neither full credit card number nor CVV2 code should be saved anywhere in the database or any other local storage (PCI compliance).
Proper implementation of submit_form, submit_form_validate and submit_form_submit callbacks.
Off-site payment methods
Proper implementation of redirect_form, redirect_form_validate and redirect_form_submit callbacks.
Drupal
Follow Drupal standards, security and best practices guidelines, especially coding standards.
Follow Module documentation guidelines.
Test your module with Coder (including CodeSniffer).
Make sure there are no spelling errors in strings displayed on the site, all strings are translatable, and all messages are clear and self-explanatory.
TBC.