Day 3 of the Chaincode Labs Lightning Residency

Day 2: https://medium.com/@pierre_rochard/day-2-of-the-chaincode-labs-lightning-residency-669aecab5f16

Day 1: https://medium.com/@pierre_rochard/day-1-of-the-chaincode-labs-lightning-residency-ab4c29ce2077

Building Lightning into Bitrefill

Justin Camarena

https://www.bitrefill.com/

What is Bitrefill? Originally prepaid phone credits seller in 107+ countries, recently expanded into gift cards.

The Path to Lightning Integration

Hired to remove Bitrefill’s dependence on third party APIs, essentially build btcpayserver before it existed.

Scaling Issues

Merchants were hit hard due to fee spikes and backlogs. Bitrefill was affected due to many low value products in some countries. Bitrefill started using SegWit very early, and started accepting altcoins because some users wanted to use them. Started integrating Lightning.

Actually Integrating Lightning

Original backend was btcd + LND. Justin was a tester for LND for over a year before. Nodejs backend + LND’s gRPC interface.

Application backend calls out to bitcoin backend api server when generating a lightning order or deposit invoice. Memo uses backend app’s UUID. Application backend keeps state of orders being paid or refunded. Expiration is handled on the lightning end which is great to avoid payments after expiration. Much better UX.

SubscribeInvoice on the bitcoin backend service, sending relevant notifications to the application backend, resends notifications using listinvoices. Data passed is memo, amount, settled, and amount paid in satoshis.

Deploying Lightning

Originally testnet only. This was before beta and shortly after segwit activated, working in 2017.

Challenges integrating Lightning

Unstable, bugs, money loss on testnet, wallet issues out of state (ghost UTXOs with LND), Interoperability issues (channels force closing due to protocol disagreements or fees). Sometimes eclair, clightning, and LND worked together, sometimes not.

Lightning Nodes

Merchants should be routing nodes, not everyone agrees on this. Bitrefill is a routing node, already running 24/7, suited to route. Non-routing merchants right now are at mercy of routers for UX.

Liquidity Lightning

Incoming capacity is not a problem, if anything Bitrefill has more than enough incoming capacity. Issues offloading funds due to exchanges that don’t use lightning.

Avoiding Dust UTXO

Minimum incoming channel 0.01 btc has helped avoid small channels that could more likely close with small local balances. Can not actually avoid small utxos.

Integrating Lightning Payments

Focus on receiving pmts, and refunding on-chain initially. Stealing funds via routing fees with refunds. Receiving is easy, sending is hard!

Fee Siphoning Attack

Attacker pays for service that fails and requires a refund. Attacker has originating node and routing node, attacker sets up imbalanced channels and gouge the routing fee on the refund. This can be automated and scam the merchant.

Mitigations

Rate limit refunds to nodes but that may cause problems for those using a shared custodial node. Fee limits do not solve this issue only limit fund loss to a slower rate. Must pass on routing fees to the user, refund x satoshis less and use that for routing fees. Before any service/exchange integrate sending lightning pmts, implement fee limiting, need best practices to develop.

Other Work

Automatically restarting LND after crashes with supervisord. Migrated node to bitcoind. Lightning deposits and withdrawals.

There’s hope!

Even with all the issues, they can be fixed or mitigated. Many of them have already been fixed or are being worked on right now. Lightning will only get better with recent improvement proposals, maturity of documentation, and new developers being onboarded.

Effective Lightning Node Management

Alex Bosworth

Channel Management

Payment Routing on Public Channels.

How to be a good citizen router.

Uptime

Please use public channels to do routing.

Public channels should have a high uptime. Aggressively re-connect your node, minimize maintenance downtime. Avoid not-connectable nodes. Curate your channels. Look for others who are also curating connected nodes and channels. Look at your peers’ peers’ uptime.

Balance

Try to make sure your channels are balanced. Imbalanced channel is the second most frequent cause of failure. Minimize it by maintaining a reasonable balance. Do not open a huge number of channels and only have outbound capacity. You will not be able to route and you will be the source of problems. Don’t do the reverse either, merchants with only inbound channels can’t send out. Don’t over-open or under-open. Try to keep a 1:1 balance of channels to you and from you.

Re-balance

Watch for where you need to be re-balancing. You can do self-routes to fix balance issues. Use submarine swaps, use exchanges by opening a channel to them, buy btc from them. Spend, pay your employees and suppliers with lightning.

Activity

Charge fees so that activity is meaningful, they used your resources and you need a signal that you assigning capital well. Carefully track and adapt capacity assignments. Watch where fee-paying payments are going.

Simple Balance Guidelines

Too large is bad (>2x max size of a payment), too small is bad ($1). Autopilot in LND is a stub right now, lots of room for improvement. Room for multiple different implementations of autopilot. Increase capacity with usage, decrease capacity with no usage.

Node Centrality

Some centrality is good. Don’t use 1ml and pick the top 10 channels. We can collectively attempt to limit centrality. If you tie up all your funds with one node, it can start selling information about routing and isn’t necessarily reliable. 1ml top 10 aren’t profitable anyway, they’re already over-served. Find under-served parts of the network.

Reputation

Routing nodes are creating a reputation. Channel age can be used as a signal for stability. Channel IDs encode the height of the funding transaction, very easy to know how old a channel is it. Age means more PoW, don’t worry about reorgs, social consensus. Build up more age for the channel set. Helps with routing. We can use any signal, so do a visible good job.

Predictions

Look for opportunities. Information is valuable. You could benefit from insider info about a new lightning service that is successful, layout capacity ahead of time. Predict if peers will go offline, cooperatively close first.

Termination

You can avoid terminations using fees, disable flag. Flags, closes represent a history about your node. More public nodes isn’t necessarily better for the network. Increase the fees on paths to unreliable channels.

Who should route?

Existing need for a node. Market expertise, people who are good at trading capacity. Existing liquidity, already sitting on bitcoins.

Tradeoff

Why wouldn’t you want to be a router?

Sacrifice privacy, advertise what UTXOs are yours. Routing manipulation attacks, inflation type attacks (on-chain inflation bug could hurt you on LN). Sitting on large amounts of locked funds.

Other Costs

Channels are not a free operation, you have to pay the on-chain fees. Fees are low right now so people aren’t thinking about this. Standard hot wallet risks. Too much fun, you can spend too much time on this interesting topic

Building Zap — a look under the hood

Jack Mallers

https://github.com/LN-zap

https://github.com/LN-Zap/zap-desktop

React Native Goals

• Allow a team to move faster
• Only write code once for iOS/Android
• Improve upon the developer experience

Why not React Native?

• Highly ambitious, immature and moves fast. Building on LN is already very fast moving, find a stable base for your UI.
• Still requires native development (native modules), not 100% portable cross-platform.
• Performance and app size not as good as native.
• Attracting contributors is difficult.

https://github.com/LN-Zap/zap-iOS

Ensure code quality…

Not lightning-specific

• Flowtype
• Eslint
• Stylelint
• Prettier
• Coveralls
• Storybook: designs should be separate from app runtime, faster iteration

Zap going to use query route to tell the user the fee before sending a payment.

Automated workflows

• Github
• Travis
• AppVeyor
• Electron-Builder
• GoReleaser

High level look

• Electron + React
• Local LND with neutrino + remote node + BTCPay

Electron and React

• Write code once for all platforms
• Strong community support
• More favorable and flexible for designers
• Reusable components for potential/future web apps

Zap + LND

All of it is open source.

Lightning class has a state machine to understand what state the node is in. This state ended up in the UI, had to be factored out. Errors used to be due to users stopping and starting LND to get it to work. Use a state machine to avoid problems like this.

LND + Neutrino

Package LND into your application by spawning the process.

Communicating with LND

• gRPC
• REST

Managing LND + BTCD

• Continually track HEAD
• Manage our own LND fork
• Ability to bundle pre-built binaries with our codebase (developers don’t need to set it up themselves)
• Ability to bundle binaries with experimental features enabled that are not available in released binaries.
• Ability to bundle binaries with new / upcoming / unmerged PR’s

lnd-binary

Install LND from npm.

Backend components

• BTCD
• LND
• Docker
• Kubernetes
• Google Cloud
• All infrastructure components versioned and deployable with a single command

Building Satoshis Place 2 with LightningK0ala

Cross-platform

• Responsive design
• Touch and mouse support
• Multi-touch “pinch” / “zoom” gestures

Unfairly Cheap

• Fixed pricing of 1 sat per pixel
• No limit on pixels drawn
• No “owning” pixels

Simple and delightful

1. Welcome screen
2. Avoid cluttering the UI.
3. Minimum 3 clicks for invoice, look for shortcuts!
4. Reactive.
5. Reduce barriers of entry, no account creation, no extensions
6. 16 colors to keep things simple and minimize image size
7. Could add animations and sound effects to make it even better

Hackable

Chain services and APIs together with LN. Didn’t implement an image upload function to have others build it as a service. User pays 3rd party service for image upload, 3rd party pays Satoshis Place. You can have multiple layers of payment flows like this.

Architecture Outline

• Lightning node: access the LN
• Backend server: handle business logic and expose socket API
• Database: store settings, orders, and base64 png representation of canvas
• Web server: serve client applications

UX Flow

1. Visit satoshis.place draw and submit
2. Request is sent to backend server.
3. New order is received, validated, invoice is created and order is saved in database
4. Receive a payment request and node info
5. Route to node?
6. Yes Pay
7. No Open channel and wait
8. Get payment notification and see result
9. Payment received, fetch order with payment request, update canvas

Hosting

Zeit, mLab (database), odroid (LN node)

Lightning App Mental Model

Think of your LN node as just a payment processor with advanced capabilities. The application itself is still the most important thing.

Optimizations

• Encoding pixels to PNG
• Prune old orders, delete non-paid invoices that have expired
• Request limit API: rate limit by IP address

What worked

• Word of mouth
• SBC lightning node
• Eclair wallet was helpful
• Testnet to prototype
• Daily database backups
• Websockets to create a reactive experience
• Docker
• Community of developers helped

What didn’t work

• Telegram group devolved into trashtalking and altcoin shilling. Have to moderate communication channels.
• Saving image to database, would’ve been better to save canvas as an image in S3 to save bandwidth
• Missing message broker and payment watcher. A message broker is needed if the API is scaled to multiple instances so that client <-> backend messages can be routed properly. A payment watcher is needed to process the payment only once.

Takeaways

1. Iterate on your initial ideas
2. Keep it usable
3. Keep it simple
4. Deploy early
5. Collaborate
6. Be CRAEFUL
7. Have fun!