Bitcoin Payment Processor Extension

[dmo]

Hi everyone,
I'm Dylan with the Stanford team working on the bitcoin payment processor. Firstly, I want to apologize  to the developer community for what happened on the IRC channel, I promise it does not reflect the group's outlook on this project or CiviCRM.

Second, I was wondering if we could seek some guidance on our project. Our team was instructed to set up a local instance of bitcoind to process the bitcoin transactions, so we rented a server off of Amazon (since none of us have linux machines and couldn't get bitcoind working from there) and setup bitcoin and CiviCRM through XAMPP. Following a recommendation from Kevin, we set up CiviCRM on Pantheon using the Starter Kit (here is the Pantheon site: http://dev-bitcoin-processor.gotpantheon.com/) . We are able to put a sample extension on our AWS server using a sample greeter extension from the documentation (http://54.200.150.246/drupal/civicrm/greeter) but we are still getting to some errors on both the XAMPP and Pantheon end, so we wanted to ask the following questions:

Pantheon side:
1) I know it's for the testing environment, but how can we have Pantheon talk to bitcoind and generate extensions?

AWS side:
1) When we setup the extensions, we get this error in a little hover box:

Code: [Select]

The CiviCRM public extensions directory at https://civicrm.org/extdir/ver=4.4.0|cms=Drupal could not be contacted - please check your webserver can make external HTTP requests or contact CiviCRM team on CiviCRM forum.
Now, I am still able to install and create extensions, so not sure how serious this error is. I did the post-install step of civix, and even after and every single time I generate an extension I get this error:

Code: [Select]

<p>Initialization Error</p><p><pre>Array
(
    [callback] =&gt; Array
        (
            [0] =&gt; CRM_Core_Error
            [1] =&gt; simpleHandler
        )

    [code] =&gt; -25
    [message] =&gt; DB Error: extension not found
    [mode] =&gt; 16
    [debug_info] =&gt;  [DB Error: extension not found]
    [type] =&gt; DB_Error
    [user_info] =&gt;  [DB Error: extension not found]
    [to_string] =&gt; [db_error: message=&quot;DB Error: extension not found&quot; code=-25 mode=callback callback=CRM_Core_Error::simpleHandler prefix=&quot;&quot; info=&quot; [DB Error: extension not found]&quot;]
)
</pre></p><p></p>

                                                                                                                                                                                   
  [ErrorException]                                                                                                                                                                 
  Warning: fopen(/opt/lampp/htdocs/drupal/sites/default/files/civicrm/ConfigAndLog/CiviCRM.b09a48abc8105a63d33a210e5b8cc064.log): failed to open stream: Permission denied in /opt 
  /lampp/htdocs/drupal/sites/all/modules/contrib/packages/Log/file.php line 216            



In other words, even though everything is "working"  in the sense that we were able to get a dummy extension up and running, should I be worried about these errors?

2) I understand that I have to set up a module extension, but how do I create one of type payment processor? I know that it would require some API calls, and I've been looking at the documentation, but I'm not sure what section is relevant (here is what I am looking at: http://wiki.civicrm.org/confluence/display/CRMDOC/Create+a+Module+Extension). What header should I be focused on if I want to generate a payment processor, and what would be the best strategy to test, etc.?

Thanks, and looking forward to hearing back! Let me know if something wasn't clear.

[Kurund Jalmi]

Dylan,

bitcoin payment processor extension, great

Few things:

- https://civicrm.org/extdir/ver=4.4.0%7Ccms=Drupal  seems to be working fine, do you want to check your environment is blocking this.
- have you configured extension directory correctly with proper file permissions? Check <your-site>civicrm/admin/setting/path?reset=1 for Extension Directory

HTh
Kurund

[kreynen]

Because Pantheon uses load balancing to send requests to multiple servers to process PHP, the paths to files have to be dynamically configured in the civicrm.settings.php.  Pushing and pulling code and database changes between dev -> test -> live can require running through the same steps as http://wiki.civicrm.org/confluence/display/CRMDOC/Moving+an+Existing+Installation+to+a+New+Server+or+Location

I'm hoping @totten can answer your Civix question.

Is this the Bitcoind you are referring to what's documented here https://en.bitcoin.it/wiki/Bitcoind?  Requiring that be installed on the server is going to really limit who can use this.  That couldn't be installed on Pantheon, CiviHosting, or most shared hosts.  You'd be limiting the userbase to sites running on a VPS or dedicated server. 

Have you looked at?

https://bitpay.com/downloads/bitpayApi.pdf
https://bitpay.com/bitcoin-payment-gateway-api 

BitPay already has support for WordPress, Magento, and Ubercart.  In general, porting the PHP for these gateways from one system to another is pretty easy.
 

[totten]

As others have said, a Bitcoin payment processor would be great! It's also pretty ambitious.

At a high-level, you guys have jumped into something that's pretty tough. A few challenges:

• The techniques for creating a payment processor have evolved in Civi over the years, but the documentation for creating a payment processor hasn't been brought forward. (Case-in-point: civix doesn't support payment processors, and the docs based on civix don't talk about payment processors.)
• It's easiest to implement a payment processor that uses credit-card numbers and submits them to a backend service (like Authorize.net). It's harder when the data structures and data flows differ (as they most likely do for bitcoin).
• Because this work aims to be shared with an open-source community (and not in-house project used by one org), the packaging becomes pretty important.
• The bitcoin community doesn't have a clear single service provider (ie one needs to decide whether to use bitcoind, bitpay, or some other service).

With that said, a few comments/replies in no particular order:

• There's a wiki page at http://wiki.civicrm.org/confluence/display/CRMDOC/Create+a+Payment-Processor+Extension . Despite its name, it really talks about writing a payment processor in general (and doesn't talk at all about extensions per se).
• There have been...3... or 4... ways to write a payment processor. The basic code and interfaces are the same -- in all cases, one writes a subclass of CRM_Core_Payment and then registers that class. The techniques generally vary in how you name/register the class. More specifically:
   
  • One can write the code in core. We generally prefer that new payment processors be written as extensions because the core team doesn't have resources/incentives to maintain all the payment processors -- and extensions allow the maintenance work to be shared/spread-around more.
  • One can write an extension with "type=payment" in info.xml. This was encouraged for Civi 3.3-4.1 and still works in all 4.x releases. The advantage of this technique is its documentation -- http://wiki.civicrm.org/confluence/display/CRMDOC/Example+of+creating+a+payment+processor+extension is more cogent than the documents for any other technique. In the example, note that UCMPaymentCollection.php includes a subclass of CRM_Core_Payment; also, note that info.xml uses the <typeInfo> clause to define metadata.
  • One can write an extension with "type=module" in info.xml. This works in Civi 4.2+. It's not documented as well, but it's more flexible -- for example, if the payment processing requires a new web-service, a new form-validation rule, or a new database table, then it's possible with a "type=module" extension (but not a "type=payment" extension). There's an example for the Stripe payment processor . Note that CRM/Core/Payment/Stripe.php defines a subclass of CRM_Core_Payment. The <typeInfo> is gone from info.xml, but similar metadata appears in hook_civicrm_managed.
         
    • info.xml: https://github.com/drastik/civicrm_stripe/blob/master/extension-4.2/com.drastikbydesign.stripe/info.xml
    • CRM_Core_Payment_Stripe: https://github.com/drastik/civicrm_stripe/blob/master/extension-4.2/com.drastikbydesign.stripe/CRM/Core/Payment/Stripe.php
    • hook_civicrm_managed: https://github.com/drastik/civicrm_stripe/blob/master/extension-4.2/com.drastikbydesign.stripe/stripe.php#L117
       
• I don't know what the inputs should look like on a bitcoin payment form, but they'll probably look pretty different a standard credit card form, and you'll probably need to make some changes to the payment screens. For example, you could use the Region API ( http://wiki.civicrm.org/confluence/display/CRMDOC/Region+Reference ) to replace the content of the billing block.
  

Code: [Select]

CRM_Core_Region::instance('billing-block')->update('default', array(
  'disabled' => TRUE,
));
CRM_Core_Region::instance('billing-block')->add(array(
  'markup' => 'Some new markup',
));


• kreynen's point is quite good -- for a lot of orgs, bitcoind will be harder to setup/maintain than bitpay. On the other hand, bitcoind does give more independence (i.e. there's no middle-man charging a fee). There are probably legitimate cases for each.
• Aside: When using bitcoind, is it necessary to run bitcoind on the same host as the webserver? The wiki page mentions a JSON-RPC interface -- as long as the JSON-RPC includes some security measures (authn/authz), that might mean that bitcoind could run one place (like Amazon EC2) while the main website runs somewhere else (like Pantheon).
• Do you have any diagrams of the pageflow or dataflow? Mockups for what the payment screen looks like? I don't want to push you, but those kinds of materials will help others understand your team's approach/solution/requirements better.

[vinamratas]

Hi guys! I'm Vinamrata, one of the team members of the project. First of all, thanks for all your help/advice. We really appreciate it, although we're super overwhelmed haha.
In terms of using bitcoind vs. bitpay, the person that we're working with from FSF, John Sullivan, specifically outlined that we had to use a local instance of bitcoind, generate a new address, and display that as a QR code. We have that part working and have it independently tested, and we currently have an extension generated from civix, but now the part of integrating that working QR code with civicrm is where we are struggling, to be exact.

@totten @kreyren, we're still kind of overwhelmed in terms of how to layout our module. I guess I should be specific as to what we exactly need to get done:
*once someone clicks that they want to donate with bitcoins, it should just generate a random address through bitcoind and display it as a QRCode.
* we also generate an entry in the db about the bitcoin address and location, and ping the local instance of bitcoind to check if the donation has been received. Once it has, we change it to "received" instead of "pending."
So we don't really need an input form, we just need a button that says "Donate using Bitcoins" and then it displays a QRCode. Again, we have that part working on EC2 (I can't find the URL right now) but we're struggling to understand exactly how we integrate that with CiviCRM. We just need access to the database so we can write to it and display the donations, basically.

Let us know if we need to clarify anything else. Thanks for being so understanding, everyone 

[xavier]

Hi,

Don't hit directly the db, use the api to read/create/update.

Isn't your bitcoin address/location some kind of transaction reference? there is already a field for that in the contribution entity.

"normally" the payment processor will call the IPN backend to inform civi the payment has been paid/rejected. bitcoind seems to be the other way around and you have to call the payment processor to update your payment status?

If I understood it correctly (I find it weird), you'll need to create a new job that is run regularly to fetch the url and see if the status needs to be updated.

[totten]

Ditto @xavier

Follow-up question on this:

once someone clicks that they want to donate with bitcoins, it should just generate a random address through bitcoind and display it as a QRCode.
...
So we don't really need an input form, we just need a button that says "Donate using Bitcoins" and then it displays a QRCode.

Just to be clear -- this is saying that you don't need credit card fields? However, when it comes to asking for information -- name, email, t-shirt size, pricing options, premiums, etc -- one would still present those fields, right? (If one doesn't prompt for other information, then the donations are anonymous, and there's no way to get in touch with the donor. Maybe that is the goal -- after all, BTC and FSF appeal to a crowd that is conscientious about privacy -- but it's a sufficiently off-beat approach that one needs to be quite clear about it.)

Has there been any thinking about topics like:
 * What happens when the payment submitted in BTC doesn't match expectations? (For example, if a constituent says that he'll be a "Silver Patron with $250 donation" but only sends in half that value with BTC?)
 * The BTC-USD exchange rates seem pretty wild. Is the goal to hold on to BTC or to exchange it quickly? To provide a realtime conversion of price-points between BTC/USD? To define separate price-points for BTC and USD?

[vinamratas]

To clarify:

1) I don't think we need credit card fields. They just said we need to generate an address where the donation can be sent. I'm not sure how the mechanics works but that's what they told us to do.

2) In terms of the BTC-USD rate, we store the value in dollars, and we call an external API to do the conversion (we have that part working independently too).

3) Yes, we need to continually query bitcoind to see if the donation was received.

Any other suggestions of how to use the API/create the payment processor extension?

[johns]

xavier makes some important points I think: Don't try to act on the DB directly, use the API. The existing transaction ID field on a CiviCRM contribution record should work for storing some information about the bitcoin transaction. CiviCRM has a framework for regularly running jobs, which can be used as the basis for the part where you have to poll the bitcoind in order to find out if a transaction has been completed.

We specifically want to use bitcoind, in order to have a fully self-hosted operation using only free software. It should not be a requirement that bitcoind be on exactly the same server as CiviCRM, though.

[kreynen]

Why not use the existing Paypal Standard processor's workflow as the starting point on the CiviCRM side?  That workflow redirects users to paypal.com (a URL outside of the CiviCRM site) to complete the transaction.  You could do the same thing... with the same potential issue of users failing to complete payment during the process.  The upside is this would give site builders an option of where to run bitcoind and the PHP wrapper you've already developed making it possible to run this on a site where installing bitcoind won't be an option. 

The pieces you are missing is the extension that extends CRM_Core_Payment so CiviCRM knows to pass the transaction off to your bitcoind wrapper, the code in your bitcoind wrapper to pass data about completed transactions back to CiviCRM, and something in CiviCRM to receive and process that data.

Take a look at...

https://github.com/civicrm/civicrm-core/blob/master/CRM/Core/Payment/PayPalImpl.php

Couldn't the Bitcoind Wrapper just report the payment using something like Instant Payment Notification (IPN) the same way PayPal Standard does? 

https://github.com/civicrm/civicrm-core/blob/master/CRM/Core/Payment/PayPalImpl.php#L498

Code: [Select]

$notifyURL = $config->userFrameworkResourceURL . "extern/ipn.php?reset=1&contactID={$params['contactID']}" . "&contributionID={$params['contributionID']}" . "&module={$component}";
Someone who's more familiar w/ these IPN files in external/ and payment processors in general will need to chime in about the best way for an extension to add IPN functionality.  We have a few of these IPN files...

https://github.com/civicrm/civicrm-core/blob/master/extern/ipn.php
https://github.com/civicrm/civicrm-core/blob/master/extern/pxIPN.php
https://github.com/civicrm/civicrm-core/blob/master/extern/authorizeIPN.php

With the Stripe extension, it looks like it's defining civicrm/stripe/webhook in xml/Menu/stripe.xml...
https://github.com/drastik/civicrm_stripe/blob/4.3-1.7/com.drastikbydesign.stripe/xml/Menu/stripe.xml

Then uses CRM_Core_DAO::executeQuery to update the contribution in the database

Code: [Select]

CRM_Core_DAO::executeQuery("UPDATE civicrm_contribution
SET contribution_status_id = '1'
WHERE id = %1",
            $query_params);
https://github.com/drastik/civicrm_stripe/blob/4.3-1.7/com.drastikbydesign.stripe/CRM/Stripe/Page/Webhook.php#L108

Pretty straight forward, but is there a better way to do that than executeQuery?

The IPN approach is also something being actively discussed in the Bitcoin forums...

http://bitcoin.stackexchange.com/questions/12035/building-a-bitcoin-ipn-need-unique-identifier
https://bitcointalk.org/index.php?topic=56801.0
https://bitcointalk.org/index.php?topic=305876.0

[marxistvegan]

Hey everyone,
this looks really interesting and we at glocal.coop are looking into this as well. where is this extension being developed and discussed as it seems it has been a couple of weeks since the last update.

[kreynen]

The Stanford students gave up on the project.  I've added it to the Google Summer or Code project idea list.  http://wiki.civicrm.org/confluence/display/CRM/Google+Summer+of+Code+-+2014

Hoping that with more structure and time another student can finish this, but if you get something working before then... even better.

[andyw]

Hi All,

Not sure if anything further ever got done on this, but I've been working on something to do this .. just as a bit of a personal dev project at Circle - and because I'm interested in it.

Have got an extension that adds two payment processors - one for BitPay, one for bitcoind, basically as discussed above, and have got some time off in a week or so, so was hoping to finish those off, get something available for testing.

.. if anyone's still interested.

Cheers!

Andy

[Eileen]

Andy - I've been integrating omnipay which is a multiple processor library. It supports a few bitcoin processors & I set myself out with a bitpay account to test & then found out that the effort in obtaining a bit of bitcoin to test with seemed to involve a trip into town / presenting 8 forms of Id & signing in blood so it stalled.

You can see the list of omnipay gateways here

https://github.com/omnipay/omnipay

I got paypal working as a test & have been focussed on a new one (cybersource) with a new (to Civi) flow

[Chris Burgess]

Eileen - if you want a few BTC to play with, check out https://mybitcoinsaver.com/ which Webscope have just launched. Might provide an easier method to get your hands on them than the blood route.