Conner Fromknecht - Stephan Livera
Speakers: Conner Fromknecht
Date: June 24, 2019
Transcript By: Michael Folkson
Category: Podcast
Stephan Livera podcast with Conner Fromknecht - June 24th 2019
https://twitter.com/kanzure/status/1143553150152060928
Podcast: https://stephanlivera.com/episode/83
Stephan: Conner I am a big fan of what you guys are doing at Lightning Labs and I’ve been trying to get you on for a while. I’m glad that we were finally able to make it happen. Welcome to the show.
Conner: Awesome, thank you Stephan. It has been a while since we first met in Australia. We’ve been busy putting stuff together for 0.7 recently but it is great to have that almost complete and great to be able to share some of the stuff we’ve been doing on watchtowers and such.
Stephan: There’s a lot of excitement. There’s a lot of feedback coming out of the Bitcoin community. What I was keen to discuss with you today was this watchtowers concept and try to deep dive on that exact idea. Before we go into watchtowers it might be good to go through a quick example of a channel set up and a breach in a non-watchtower world. For example’s sake, let’s say I’m running lnd, you’re running lnd. I want to set up a channel with you and I want to put 4 million sats into that channel or 0.04 BTC. Now we’ve got that channel open and that’s the initial state. Then if I want to make a payment to you what is going on underneath the surface is my lnd is saying “Let’s change the state of this channel” such that if I pay you 1 million, it’s saying “Conner has 1 million in the balance and now I have only 3 million sats in my balance.” Now let’s say we’ve been paying attention to Michael Goldstein’s “Everyone is a scammer” article and I’m actually a scammer. Let’s say I try to scam you. Suppose I’m a massive scammer and this has all been a long con and I now maliciously try to broadcast the incorrect state to the blockchain. So what’s happening there, just to explain that, is instead of broadcasting a Bitcoin transaction with the balance of 3 million for me and 1 million for you, I might attempt to broadcast a transaction that gives me 4 million sats and zero for you. Could you explain for us without watchtowers how would lnd deal with that?
Conner: That’s a great explanation. That state with 4 million sats on your side was the initial state. Once you sent that over to me we entered this agreement where that state has been revoked. I have this key to spend this special path on your commitment transaction for that prior historical state. Now what is going to happen is lnd is always monitoring the chain for spends of the funding output. When your commitment transaction hits it causes a spend of that output and that triggers a series of processes in lnd that will basically assess the situation and say “Was this a breach or not?” by looking at is it the latest state or not. After determining that it will also understand that it can spend this breach path. There may be several outputs on any given commitment transaction that can be revoked. It will take all those outputs, sign, produce witnesses and signatures that spend from them and sweep all those funds immediately back into the wallet. The reason I’m able to do that is because those outputs have what is called a CSV delay or relative timelock. When you broadcast you are holding yourself to let’s say a 144 block window where you can’t move the funds and you’ve given me this opportunity to revoke them. That’s what makes the channel secure. If you try to cheat I know that I’ll have this window to try and rectify the situation. If I’m online I’ll spend those coins back to my wallet. It is really no harm, no foul because I actually took more money than I was supposed to have. You would only send me 1 million sats and I just took all 4 million minus fees.
Stephan: You’ve scammed the scammer. You’ve won Conner.
Conner: That’s what you get from cheating.
Stephan: What it’s getting at there is let’s say I’m the scammer in this scenario and I’m trying to cheat you and steal the balance. What it is doing is at the time that I try to breach you I don’t actually have access to those funds yet and that’s the CheckSequenceVerify which is the relative timelock, that’s the CSV code. How does lnd detect the breach? Is it monitoring the blockchain for any movement that touches this particular output? How does it do that?
Conner: That’s correct. The way it detects this depends on the backend. If you’re using a RPC backend like btcd or bitcoind the RPC interface has a way to notify when this particular output or UTXO gets spent. With those RPC interfaces you subscribe to notifications on those outputs. With Neutrino it is a little different. Because it is a light client you have to do all the filtering locally but more or less when a new block comes in, it downloads the filter for it and checks if any of the channels that are mine…. I don’t really need to monitor all of the other ones for breaches, I only really care about my own. I’ll look in that filter for my own scripts being spent. If it matches, we’ll fetch the block and double-check whether or not it was actually spent. That produces a notification internally to lnd, it does the same effect as if you’d subscribed to the RPC.
Stephan: I think that is the example of the justice transaction. That’s saying “Someone is trying to scam me, let me do the justice transaction.” What’s the correct term? I know that’s the colloquial term, what’s the actual term?
Conner: There’s the breach remedy transaction, I think that is what they call it in the spec and the white paper. The breach remedy, justice transaction, they’re the same thing. That’s the transaction that takes all the funds on a commitment transaction that was revoked and sweeps it back to my wallet and takes all the funds for myself because the other person inherently either tried to cheat or accidentally restored an old state from a backup or something.
Stephan: Excellent. We should point out for the listeners that the behavior of that is proposed to change with eltoo. If we get eltoo the actual punishment will change where instead of the scenario where your lnd node takes the full balance, your lnd node would just publish the correct state. Is that correct?
Conner: Yes that’s correct. So you would publish…. it is always a game of I publish the more correct state. I could publish a newer one. If I also don’t know the latest one you could publish one later than that. Basically it ratchets all the way up to whoever knows the latest one. More or less the resulting balance is then played out onchain, whatever HTLCs are there at the time would be played out. Right now if you play an old state, even if there are HTLCs on that you ignore that and you just sweep it all because it is cheating.
Stephan: I’m also interested to discuss the difference between a collaborative close and an uncooperative close. Can you help articulate what is going on there?
Conner: Sure. A channel is identified by its funding outpoint which is a 2-of-2 multisig output in the blockchain. Because it is a 2-of-2 anything that has a signature from both participants is allowed to move those coins. When you do offchain transactions you have these commitment transactions that are signed by both signatures, by me and you. I guess one thing to also clarify is that commitment transactions are asymmetric. I will have a whole chain of commitment transactions from myself and you will have their mirror image on your node. That produces this whole asymmetric history. All those are signed with these two signatures. Any of them at any point in time could be broadcast and the blockchain doesn’t necessarily know whether it is an old state or a new state. And whether it is the most recent or who’s cheating. It does see that it has a multisig and it will accept it and let it confirm and spend that. A unilateral close is when either party publishes one of those pre-signed states. A cooperative close says “We’re going to sign a new transaction ignoring all the history and we’re just going to spend those in a way that splits up the amount however we agree.” If we ended up with 1 million, 3 million satoshis in the channel we could just sign a cooperative close which looks basically like a regular payment in Bitcoin other than the fact that it is spending this 2-of-2 multisig. It splits up the amount between you and I. The benefits here are that you’re not encumbered by any timelocks, it is a lot cheaper because there are a couple of transactions that can play out onchain depending on how many HTLCs are there or things like that. It is an advantage to always try to cooperative close but that’s not always the case. If one of us were to lose state then our only option is for one of us to publish this unilateral transaction because maybe you lost the signing keys and you can’t actually derive that cooperative close transaction now.
Stephan: Fascinating. Just to quickly paraphrase for the listeners, make sure everyone can follow along. It is essentially saying in the cooperative close case there is no timelock placed on the Bitcoins that have spent out of that because we’ve agreed this is the correct state. It is 3 million and 1 million. In the unilateral case, that’s the example where going back to what we were saying about the CSV, the CheckSequenceVerify, meaning that there’s an additional time, depending on that window. I think you said 144 blocks, I presume that can be changed as well, the length of that.
Conner: lnd scales that based on how much is in the channel currently. If more is at stake it gets longer.
Stephan: Fascinating. The other thing with that is also that over time if we are right. We are both bullish on Bitcoin and we think a lot more people are going to use Bitcoin. The blockchain will become more and more full. Part of Lightning’s security model is if somebody is trying to cheat you, you need to be able to do you justice tx back. Tuning that window in terms of how many blocks that CSV is, is essentially how you can allow people the chance to do a remedy.
Conner: That is correct. There are some other interesting dynamics at play though too. In the case where you send me 1 million and you have 3. Now you’re trying to steal 4 back or try to get back to a state where you have 4. Because I’m only promised 1 million at that point in time, I have 3 million sats available to spend on fees. So you publishing a breach trying to earn back 1 million, you may pay more in fees to outbid me to send that transaction than you may actually gain from doing it. There’s also those things to be aware of. Because I can have a scorched earth policy and say “I was only promised a million, I just want to get a million”. You have a lot more bidding to do to get in there. It is definitely possible but how much do you really want to scam someone out of their 1 million Bitcoin when you are going to have to spend more than 3 to get 1 back?
Stephan: Fascinating, I didn’t think about that. Let’s now go onto watchtowers. Enter watchtowers. Why might we need a watchtower?
Conner: That’s a great question. In the case when a node is offline it can’t really do this monitoring where it is watching for the funding output to be spent, getting the notifications over RPC or Neutrino. So the issue becomes if my node is offline, say I have a mobile phone or my node crashed for some reason. I’m out on a hike for the weekend. There could be a period where you see me go down because we’re not connected or we haven’t talked to each other in a week. You try to cheat me because you’re saying “This seems like the best time to do it. No better time then now.” You publish this old state trying to take this and if I don’t come back online within a reasonable amount time, before that CSV time has expired and has allowed you to move the coins then I lose those funds that I was initially promised. The solution here is basically to outsource the ability to do this monitoring on behalf of nodes that may have intermittent connectivity like a mobile phone. Or a backup measure for a large institution or company that doesn’t want to have any service outage preventing them from keeping their funds safe. That’s where watchtowers come in. I think an interesting analogy is to think of a watchtower as a form of channel insurance. You basically are having this other person help you to make sure your funds are safe. They can do that without your assistance. They’ll go to court for you. They’ll attest to the fact that this was actually a breach transaction and that you’re entitled to those funds and help you get that back.
Stephan: How does the watchtower detect a breach?
Conner: That’s a good question. There could be a number of different ways. The design space here is really, really large. In the current protocol, one of the basic goals of the watchtower protocol that we’re working on is that we don’t want the watchtower to necessarily know which channels it is monitoring. It doesn’t have the option to hook up to a bitcoind RPC interface and get a notification for UTXOs. That is one way to build a watchtower but it comes at the expense of all the users’ privacy because you now know exactly which channels you are watching. You can also say “If that node is public I can connect to them and test if they’re online.” Now the watchtower has the ability to see if these people are online. We want to remove the ability for the watchtower to actively know which channels it’s monitoring. That dictates this other approach where we use the transaction identifiers, the txids, of the breach commitment transactions and we generate what is called a hint. The hint is generated by hashing the breach transaction using no additional secret data. More or less the appearance of the breach transaction ID on the blockchain allows the watchtower to decrypt some state and broadcast the justice transaction for the user without them being online. To clarify, what you do as a tower is you take a block and you look at all the transactions in it. You take the transaction IDs, hash them again and now you get these hints and you check your database for any of the hints that match. You might be looking through a database of 2000 different hints for any given block and if any of them match you can turn that into a decryption key and create the justice transaction.
Stephan: That’s a bit complicated, I’m struggling to follow. How is it that the watchtower can’t steal your funds? If I’m a noobie and I don’t really understand what this is, why is that the watchtower can’t steal my money?
Conner: What allows the user to spend the breach transaction normally when it is online is that it knows the secret key to all these revocation clauses on the transaction. There’s a specific clause that has this revocation public key and I as the user am online. I know the secret key and I just craft the transaction and sign it and that spends back to my wallet. What prevents the watchtower from doing that is that we never give the watchtower that key. The watchtower doesn’t have unilateral control over where those funds end up. Instead what we do is when a phone for example wants to send this data to the tower so it can do its job, I pre-sign a transaction that spends from it. More or less what this looks like is every time you pay someone on lnd, you pretend as if the other person already tried to breach you. You sign transactions that say “If this had happened, this is what I would do with it.” I take that transaction, encrypt pieces of it and send it to the tower. All the tower is doing is basically decrypting this information, putting it back together and then broadcasting a transaction that I pre-authorized.
Stephan: And so it is already preset which address it is going to?
Conner: Yes it would be pre-set to an address that I control. All they can do is broadcast and it would end up in my wallet.
Stephan: Another thing that is interesting is the funding and operational model of these watchtowers. Is the idea that I’m going to be an entrepreneur and I’m going to run a watchtower service or is it more like we’re relying on people being an altruistic watchtower guardian?
Conner: I think this is very interesting. Initially we imagined that most watchtowers would be operating for a reward. I think that’s still a very possible likelihood in the future. If someone is doing a good job you want to incentivize them to continue to do a good job. Also if you want your tower to stay in business and keep watching for breaches for you, you’re going to want to pay them something probably. It was later on that we also realized that there was a niche that could be served by having these altruistic towers. It is funny that you call it that because that is what we called it in the codebase. The only distinction here is that an altruistic tower, when it creates a justice transaction, there is one output and all the funds except for fees go back to you. For a reward tower, the other name, there are two outputs. One is a reward paying to the tower, paying it for doing a good job and actually responding while I wasn’t online. The rest of it goes back to myself. The only difference is how the money gets split up between the two users. I think you’ll probably see use cases for both. It doesn’t really make sense for a company to run their own watchtower and then give it a reward. It is all their money and that’s just more fees and stuff like that. Also if you have a service or an app and you want to provide a watchtower for your users, you might not choose to charge them because there are other ways of monetizing their services. There are a lot of use cases where an altruistic tower is going to be useful. The reward towers are mostly built out in terms of lnd. There are a couple of things to fine-tune there as far as testing and stuff like that. For the most part that is also completed as well. That will allow people to also offer services where they get paid to do that. It will be up to users on whether or not they want to have a tower. Or a company that might want to offer an altruistic tower. Or if they choose to they might choose to reward a tower that offers a really good, reliable service. Maybe they know it is a respectable watchtower. Users will have control over that and it will be interesting to see which one plays out. I suspect we’ll be seeing both in the wild.
Stephan: Fascinating stuff. We’re basically going to see what happens and see what people try to start up and see if they want to run it as a business. And then the question is how much would the fees be? I’m also thinking now does it work from a UTXO point of view? If the amount of the fee is so small that the UTXO is like dust. Do you know what I’m getting at?
Conner: Yeah if I have a 1 cent channel and I’m giving you 0.01% that doesn’t make any sense. With things like that the client won’t even try to back those up. If a tower detects you trying to back up dumb amounts like that it will reject you and say “go away”. The reward rates are negotiated upfront so I can see if I’m getting 1% or 0% or 99%. It is interesting because you don’t know the channels you’re watching so you don’t know the full balance. At the very least a reward tower would have the option to filter out “Am I getting a reasonable amount?’. The other thing you can do is there is a proportional and a base reward. I can set it so I get 4x a dust UTXO plus some proportion of the channel size. There are ways you can fine-tune it. That will mostly be configurable. It will be interesting to see how the fee market develops. Ideally the interesting part about watchtowers is that hopefully none of this code ever executes. If everything was perfect none of this code would ever run. It is unclear how much you’d want to base your business model around people getting breached. I think a more likely scenario is that people pay a small amount to say “I’ll pay the tower a couple of cents and back up 10,000 of these states and that will hold me as a mobile phone for months.” Basically you’re charging for someone to store a bunch of data blobs the size of tweets so it is really not that bad from a storage perspective. I think the cost will remain pretty minimal. It is also a more stable source of income for a tower whereas a breach might be not at all. There might be one day where a bunch of breaches happen this year and other than that you have no idea what to feed your business model with. But it is still possible to do that.
Stephan: Who knows? Maybe we see Lightning service providers and they give you incoming liquidity and they give you a watchtower as well. They try to bundle it together with other things. I don’t know, I’m just spitballing.
Conner: I have some suspicion if you want to use a reward tower you probably have a bigger channel that you want to make sure the watchtower is incentivized to broadcast for you. That’s my underlying assumption but we’ll see how that plays out in practice.
Stephan: Another one that is interesting to me is in that example, 3 million, 1 million sats example, that is an example where you and I are making direct payments to each other. The other component of Lightning is the multihop nature of it. Let’s say you’ve got another connection with Laolu and he’s routing a payment through you to me. How do watchtowers protect in that case for the routed amount as opposed to you and I having the direct 3 million, 1 million sats? Is there a difference there?
Conner: From the perspective of the watchtower, no. From how lnd will protect against that or really any watchtower implementation will protect against that, no. There’s not really any difference there. One interesting thing there is that if I’m on a mobile phone for example I’m probably not going to be routing. That case occurs less on a mobile phone. If you’re a routing node you have this assumption that you’re online and available all the time. Your dependence on watchtowers is reduced. If you’re running a watchtower it will treat them all identically. I suspect that people will do that as sort of like a failover mode and safety net.
Stephan: Let’s talk about privacy. What are the things that we have to think about from a privacy point of view with watchtowers?
Conner: There are a number of different things that come into play. We’ve touched on some of them. One is making sure that the watchtower doesn’t know what channels it is watching. That is important for knowing if someone tries to come to the watchtower and say “Help me collude against this person.” It has no idea whether it is actually watching. It is hard for it to even try to target users in that sense. There’s other notions of traffic analysis for example. If I can watch your packets go by and detect by their size whether you’re doing a reward tower, an altruistic tower. That would allow some fingerprinting. If you can detect based on the size of the messages you’re sending. All of these things can be revealing in terms of what channel it is and what user. There are other instances of timing attacks where if I always make payments every day at 2 o clock the watchtower can try to sift through that data and determine these are the same user based on those habits. There are a number of different aspects that are similar to that. You want to mitigate the amount of deanonymization that the tower has access to. Especially in this setting of once it has all the data it now has a database of records to search through. How can you minimize the amount of information that the watchtower can learn from that? There’s ten or eleven different things that you can enumerate in terms of privacy. I’d say those are some of the most important ones.
Stephan: Of those what are you thinking in terms of lnd watchtowers out of the privacy considerations that you’ve named?
Conner: For us these privacy metrics were a pretty big motivating factor in terms of the final design that we have in lnd. For example, one of the ways you mitigate against people knowing which channel you’re watching is this idea of accession. Accession works and we touched on this earlier as well. You might pay for like 10,000 updates upfront or the ability to send 10,000 updates and that is agnostic of any channel in particular. You’re basically saying “I want 10,000 slots’ and you take whatever I give you. Then slowly you can fill those up but they’re not tied to any particular channel or user. Say you have five different towers up at once I could back some of them up to this one, some to another one and rotate across them. There’s really no need to have them all watch all histories of all channels. That allows the client a lot of freedom in how it picks and uses different heuristics to cover its traffic from the towers themselves. I think that was one of the big design decisions that was talked about on the mailing list for a bit. We ended up deciding that the privacy benefits were worth the extra complexity. Other things like the traffic analysis. The current protocol has fixed size payloads for every aspect of the protocol at this point. You can’t tell during the handshake when the client communicates with the tower. You can’t tell whether it is doing a reward or altruistic session. All the blobs that get sent to the tower, these encrypted blobs with the pieces of the justice transaction. These are all fixed size as well. Just by looking through the database, we tried to limit the amount of observable data that would help pin back to someone. All these things combined, there are a number of things where we’ve from a protocol and data level tried to remove the ability to fingerprint. The other things that remain are timing analysis things. When I send these to the tower how do I and who do I send these to? All these ideas of tower rotation and things like that. Those are yet to be implemented but all the building blocks are there and in the protocol to allow them to develop. As we continue to build on watchtowers we will have more of those features brought into lnd.
Stephan: Just to clarify, if you were to have five different channels you could use different watchtowers for each one? Is that the idea? You’re not putting all your trust into one watchtower to look out for you for every single Lightning channel that you have?
Conner: Correct. You can also back them all up to five different towers and then you have redundancy. I think my goal would be to have five but I want to back up the three and it auto-rotates between them and scrambles the pattern between which ones it is actually using. I think something like that would end up being pretty cool. Your entire history of use of that channel is not stored on one tower.
Stephan: That definitely makes it a little bit harder for somebody to try to attack you from a privacy point of view. It is a good point around this whole altruistic thing as well. The speculation is even now in terms of Bitcoin transactions and Electrum that some of these chain analysis companies are the ones who run the public Electrum servers. In doing so they want to cluster you and figure out this guy has x, y and z UTXOs and we saw him access it from this IP, that is already pretty bad from a privacy point of view. What you’re talking about here is to design the system in such a way that is harder for that outcome to happen.
Conner: Precisely. The way we attacked this was from the perspective of let’s assume that the entire watchtower infrastructure was concentrated by a single entity. What is the most privacy that we could get out of a protocol between all of the clients and that entity? How do we protect users in that case? In the worst case that it devolves to it. Maybe not one, you kind of need a couple to be able to do these ideas of verification and redundancy. If there are large entities we want to protect against the case where a service makes it so easy that a lot of people to connect to it. We want to make sure that we have the utmost protections in place for those worst case setups. Hopefully in doing so it isn’t so bad as if we didn’t have all these anonymity protections in place.
Stephan: Another thing is the cost of running a watchtower. Again going back to that example of who would run a Electrum public server, who has an incentive and the resources to do that? Chain analysis companies do. It is a similar idea of trying to keep the costs of running a full node cheap. Let’s talk about what it does cost to run a watchtower and then also if you could touch on if there is any impact on watchtowers if you are using Neutrino or a pruned node.
Conner: I think the overall cost of running a tower is minimal obviously depending on scale. On a per-user basis it is not too bad. Fundamentally as a watchtower you do one thing every ten minutes in terms of with the chain. You fetch one block, you hash two thousand transactions, you look up in a database for two thousand things. That’s all you do from that perspective. The other side of things is having a server that just accepts clients and writes their updates into this database. That happens in an ongoing process. The fundamental scalability limitations would be can you handle a bunch of connections? Is your box capable of handling clients and how does that look when you scale it out to thousands of users or millions? The actual performance requirements on the chain and on that side are fairly minimal. You have the scale that only impacts the server side, accepting of states. The only real limitation besides that is how much disk space you have. As I said earlier this initial version has these sort of blobs that are about 300 bytes per update, every time the commitment transactions are updated. That’s basically you storing a tweet every time someone makes a payment which probably isn’t too bad from a perspective of someone with a reasonable amount of infrastructure. As far as whether you can do this with Neutrino or a pruned node, yes you can do it with a pruned node. In fact it is probably better because it gives you more space for handling all these updates. You can also do Neutrino which also saves space. You also in theory run the risk of not being on the right chain because it is not fully validating. That can be protected by pointing it to a gateway full node that is validating. Then all your watchtower services can be very light and independent of that. That is a pretty good way of doing it I think.
Stephan: The other thing then is the hard drive capacity requirements. Is it one of those things where if you’re running a watchtower and you have a lot of users who say “Hey Conner I want to give you my five channels to watch for me” and then there are a lot of payments passing through those channels, does that start to become more troublesome from a computational hard drive cost or sync times?
Conner: It depends on your set up and depends on your implementation. Basically this problem of taking the transactions in a block and finding which hints and encrypted justice transactions match that is perfectly parallelizable. In theory you can try this out over multiple machines if you need to. You can even sort the database like that. Syncing in theory if you have a spinning disk it might be a fair amount of effort to do that but there are other ways of storing data on disk that would be more performant if you had the proper indexes in place. It could be if you don’t have the proper optimizations in place but once you have those optimizations you should be well on your way. In my opinion that shouldn’t be an issue in terms of actually ramping that up across multiple machines if you need to.
Stephan: What about the limitations of watchtowers? Just thinking out loud the main risk really is that if I were to have only one watchtower and I’m trusting that watchtower and that watchtower fails me, if it doesn’t broadcast justice transaction for me, I guess that’s one of the main risks. Are there any risks or limitations that you see there?
Conner: That’s a good one. For example at the moment in lnd this initial roll out will let you configure a single watchtower. We have some PRs down the road that are still in progress that would allow you to do multiple. Obviously there is still a risk that all n of your towers go down. That is obviously a risk. Hopefully the redundancy allows you to mitigate those. By picking mutually disinterested parties, something like that. I think another fundamental limitation of watchtowers is that because the watchtower doesn’t have access to your keys you have to sign your transactions with a predetermined fee rate. Depending on the fee market at any given time your transaction may or may not have the wait to get into a block. That can be mitigated in a couple of ways. You can more or less sign transactions at different fee rates and send those to the different towers. Maybe the different towers have different fee rate transactions and other things like that. At the end of the day if I sign a transaction for 200 satoshis a byte and the fee market is at 250 there isn’t a whole lot you can do without being online. I think that’s probably the more fundamental limitation and something without giving up your keys it is hard to extend the capability to do that without extra authorization. That being said there are ways to get around this. If you also include a reward for the tower or the tower is also trying to make sure it does its job it can attach fees itself to those transactions and bring them down on your behalf. It is not quite the same. I think it would be interesting to work out the dynamics of the incentives for that edge case. It is also possible for the watchtower to bump fees for you. Whether or not that is agreed by some service agreement or whatever is undecided at this point. I think that is a current limitation for how watchtowers work at the moment. Hopefully we will have more mitigations against down the road.
Stephan: Fascinating. Just to paraphrase again for the listeners, part of this is owing to the difficulty right now of fee estimation. Currently the way Lightning Network works is when you open that channel with somebody there is a fee reserve that is being allotted and theoretically part of the problem is if the fee market moves dramatically in that time then when you want to do the close or justice transaction then it might be not adequate in terms of the amount leftover for a fee. When you’re bringing in the watchtower aspect of it, that’s another level of complexity that you’re bringing because it means now the watchtower might not have enough fee to get the transaction inside that CSV 144 blocks, in that example 144 blocks. As you mentioned that is something that could get tuned over time. Imagine five or ten years down the line if we start to have full blocks would lnd developers at that point decide to expand the CSV window to give more time?
Conner: That’s definitely possible. Right now lnd will scale that CSV based on the amount in the channel. Right now the max channel size is somewhere around $1500 or so. If you don’t want to have your $1500 on the line for one day, maybe you want two weeks. lnd already scales that up in some sort of linear fashion. Down the road we can adjust those. Another interesting thing is say you back up your channel states today and you back them up with 50 sats a byte but then a year down the road you see that maybe this fee market is permanent and it is stuck at around 100 satoshis a byte you can go back and back up all of your states again with 100 satoshis a byte. You can retroactively upload these states as well and you can bump fee rates in that sort of sense. The case you brought up with the commitment transactions having this set fee rate and that it maybe not being enough in the future, there is some ongoing work in the BOLT 1.1 spec that will allow you to have a minimal fee on the commitment transaction and then you will attach fees later at the time you want to broadcast. This is similar to the idea with the watchtowers in that it has some sort of pre-committed fee but the watchtower can also attach as many fees as necessary on your behalf if it also doesn’t happen to be enough. Those ideas are related and this is something we’re coming to grips with on public blockchain protocols. There are some interesting directions there.
Stephan: Fascinating stuff. Let’s talk a little bit about how watchtowers might evolve in future. Obviously there are many different Lightning teams. There’s you guys, Lightning Labs, there’s Blockstream’s team, there’s also ACINQ, there’s some other teams working on Lightning. Is there a plan to try to have watchtowers become part of the actual spec? Or would everyone do their own thing?
Conner: Ideally I think they would be incorporated into the spec. Given the design space at this point I think it is a little premature to have people commit to something before we’ve given it a sufficient amount of testing. At least that’s been our opinion. In the process we’ve been doing some tests, seeing what works. I think we’d like to run this for a bit in the wild and get an idea for whether there are any pain points we would like to address before moving forward for a final spec version that then people have to commit to. The important thing with a lot of these things is if we were to rush into it and encourage the wider community to commit to this protocol that may have these defects is that the defaults tend to end up being pretty sticky. This is pretty bad from a privacy perspective. Once people go implement watchtower v1 there’s not very much incentive to implement watchtower v2. Ideally what we’re hoping for is to learn from this experience and take any new information that we get from this release and package this up into a more cohesive spec proposal. Then hopefully the spec version 1 would be more complete and have those kinds of kinks worked out. We can move from there. That’s the initial idea for how we get to spec land.
Stephan: How does lnd watchtowers compare with, I know there is a BLW, Bitcoin Lightning Wallet by, I’m not sure if I’ll pronounce the guy’s name correctly, I think it is Anton Kumaigorodskiy. I think he has some kind of watchtower baked into that wallet product. How does lnd compare with that?
Conner: That’s a good question. From my understanding the towers in lnd are designed from the ground-up to be a little more privacy preserving. I think that’s probably one big distinction. I think also he uses this cool thing for blinded tokens where you pay and you redeem those tokens later to be able to update states, stuff like that. I’m not totally sure but I don’t know if the protocol is accessible outside the app, it might be but I’m not totally familiar with the specifics. One other difference would be whether or not it does a reward or whether it is purely altruistic. There are a couple of distinctions there. I think the biggest one is probably the tower sort of knows which channels it is watching. With lnd you would delineate that.
Stephan: Another one I was meaning to ask is interoperability. If you are a c-lightning user you can’t use a lnd watchtower? Would they have to use their own?
Conner: There would be no fundamental restriction for using the tower or not. It is a matter of implementing the client side protocol. Or the server side. If there were already lnd watchtowers running, to integrate that into another implementation is the matter of making a client that does all the steps that the lnd ones do. That becomes easier to do once there is a formal spec. Apart from reading other people’s code. That’s definitely possible. You can definitely take advantage and do cross-pollination in that sense.
Stephan: We touched on this one earlier around eltoo and how that is a new proposed model. Hopefully once we get Schnorr and so on we would move to eltoo but eltoo has a different punishment model which we were touching on earlier. Can you explain if there are any impacts there from a watchtower point of view?
Conner: I think there are. There are some new ways you can formulate a watchtower with eltoo mainly with this idea that you only need the latest state. Right now with the watchtower designed for revocation based channels the watchtower ends up keeping every state because it needs to be able to revoke directly off of any of these possible states that the counterparty can produce and broadcast onchain. With eltoo you can keep the last one and that automatically skips all the way to the end. That works as a justice transaction for all the prior states. There is the ability to implement this constant space watchtower with eltoo. You can also do an eltoo tower in the same sort of style as the one that exists today. Basically you would back up and keep every state. The advantage of that over constant space design would be the level of privacy that it offers. When I’m updating the last state and telling the tower the new latest state, you have to tell it the previous one which it needs to delete. That inherently gives you this chain of actions that link all the updates to each other. Whereas the one where you keep everything you don’t necessarily know what’s being replaced, what’s being updated. There is a trade-off in terms of privacy and the amount of storage for a tower. Personally I think an eltoo tower if its my own personal private company for example and I want to backup my node this is undoubtedly a good way to do it because you basically keep a constant amount of space for all your channels. You don’t care about the privacy. I am the person, I already know everything about my company. There definitely becomes a realistic tradeoff when you are interfacing with a service that is running this. How much do you trust that service to protect your privacy or give it away? How much do you care as a user? Maybe the eltoo constant space won’t cost more but then you have this privacy tradeoff. It is a little unclear how you’d modify it. The current protocol can be used in a way that is compatible with eltoo with the same sort of privacy model. With eltoo you also get these extra possibilities in terms of how you construct the watchtower. I think it will be interesting to see how that all plays out.
Stephan: I know you guys are working very hard on lnd 0.7. Can you tell us a little about what is coming with that?
Conner: There’s a lot of cool stuff. I think our last release was around 3 months ago. This will be a more targeted release with a handful of features that I think are really cool. One of them being this initial release of altruistic towers. Another cool feature is that we now do routing. We’ve introduced this probability based scoring system into the routing heuristics. We now consider what is the likelihood that this node will fail when we pathfind. As we attempt payments and learn this route succeeded or this hop failed, we’ll feed that back into our mental model of how the network is looking and behaving at any given time. That will inform which path to find next based off the historical patterns of the network that we’ve observed. It is a really powerful primitive because as the model continues to improve and we continue to improve the data fed into it you can really start to learn the network in a way for each user that is independent because they are all in different positions in the network. The model should be able to learn “In my position within the network these are the best routes to take and they always are able to handle my payments and these always fail.” This is the beginning of a more sentient routing capability within the network.
Stephan: It is coming alive. I guess that is helping encourage uptime and maybe it is encouraging nodes to keep the channels well balanced and not lopsided.
Conner: That’s correct. It will start to pick out those nodes that are indirectly doing this and favor those in the routing preferences. I think that’s really cool and that will become a major factor. Right now we’re discarding a lot of really valuable data about “I’ve attempted this payment. This hop failed every time. Why do I keep trying this?”. This will feed all of that back in to make the whole system more reliable. One of the features I’m really excited about is we’ve improved the initial sync times for all backends by using a feature that we built into the seed format, almost a year and a half ago. For those who are not familiar lnd uses a specific seed format that we call aezeed which is a class of cipher seeds where your seed data is encrypted. Inside that the seed has a birthday that we measure in terms of days since Bitcoin’s genesis which is some random thing we made up. Basically there’s a birthday in there which measures roughly how far from the genesis block that this seed was created. What that allows you to do is when you create a new wallet, the wallet has a birthday now. Instead of rescanning the entire chain for your funds it knows that no funds could be created before the wallet’s birthday, before the seed was created. What we do now is when you come up and you have a fully synced chain we now binary search for your birthday and then start from there. That has some really good effects. If you’re doing a rescan on half a million blocks but you just created the wallet, you now get to skip straight to the tip. Also when you’re doing a recovery this will also work because it will see that it is only a third of the way back, not all the way back to genesis. It should have a pretty good performance improvement for all backends. I think bitcoind sync is down by 50x or something like that. It is a pretty massive improvement. I’m excited for people to give that a try.
Stephan: I find the birthday thing hilarious. How old is your lnd node? How many birthdays has it had Conner?
Conner: It is probably going on 450 days. Oh no days since Bitcoin genesis.
Stephan: 450 days back from now. I suppose just for listeners it helps search when your software is trying to figure out what’s my balance, it is helping you search more quickly and not waste computational power trying to search really far back in the chain when the birthday is telling you I only started this wallet 30 days ago so don’t search back two years on the blockchain.
Conner: Correct. For anyone wondering why we chose days since Bitcoin genesis as the metric, I think this kind of interesting too, I don’t think it has been explained anywhere. You could put a timestamp in there but that’s more like 8 bytes but we were short on space in the seed format. We wanted as much space available for other things. We wanted to squeeze it down into 2 bytes so we needed something with coarser granularity. We ended up realizing that days is a pretty sufficient granularity in terms of a wallet. Bitcoin has been around for ten years, that’s like 3000 days. You should be able to get a pretty good estimate for the birthday with that. There’s also the case that you can set your birthday without even syncing the chain. If you need to know a particular block hash or height that the chain is at you can’t make the seed until you sync the chain. This allows you to create this seed without any data but the current time. The other reason that it is from Bitcoin is because we know that there are no funds on testnet or anything before the Bitcoin genesis block. That’s the metric. That will allow us to express birthdays through 2189 or something so we should be good.
Stephan: We’ll be long dead before people in the year 2189 have to deal with it.
Conner: I’ll let someone else figure that out.
Stephan: Anything else you want to touch on with lnd 0.7?
Conner: I think that covers the majority of the changes. There is one more. We refactored a lot of the internal router state for the payments and being able to reliably display them over the RPC. You can now make a payment, kill lnd, come back and all that information is properly displayed over the RPC and accounted for. Before you didn’t lose any money but the RPC data may not have been all in the correct places. There’s been some pretty big refactoring there to make all that possible. That’s one of the major other things that is in lnd now. That’s been a lot of work by Johan who spearheaded that. That will enable a lot more services to really rely on lnd as a pure data metric without having to jump through hoops to do the accounting. I think that will be a big feature that people are looking for and we’ve been having a lot of requests for.
Stephan: Excellent. That’s just about all we’ve got time for. Conner before we let you go, tell everyone where you can find you and where they can keep up with what you guys are doing at Lightning Labs.
Conner: I’m on Twitter, bitconner. You can find me there. I’m also on the lnd Slack, lnd developer community. Hit me up there. Thanks Stephan.